basic_bot.services.vision

Provide image feed and object recognition based on open-cv for the video capture input. This service will provide a list of objects and their bounding boxes in the image feed via central hub.

MJPEG Video Feed

A Motion JPEG (MJPEG) video feed is provided via http://:/video_feed that can be used as the src attribute to an HTML 'img' element. The image feed is a multipart jpeg stream (for now; TODO: reassess this). Assuming that the vision service is running on the same host machine as the browser client location, you can do something like:

WebRTC video supported

The vision service when running will respond to WebRTC offers a allow streaming video from the robot camera.

It should not be neccessary to use both MJPEG and WebRTC streaming and WebRTC video should be preferred for its lower latency and less overhead needed to encode jpg frames for MJPEG streaming.

See, webrtc_test_client.js and associated .html for example of how to use WebRTC stream from browser.

Object Recognition

The following data is provided to central_hub as fast as image capture and recognition can be done:

The [x1, y1, x2, y2] bounding box above is actually sent as the numeric values of the bounding box in the image.

Video Recording

To use the video recording feature, you must have the ffmpeg command and the libx264-dev library installed on the host machine:

The video feed can be recorded using the record_video REST API. The video is recorded in the BB_VIDEO_PATH directory. Example:

where - localhost is replaced by the IP address of the host running the vision service. - 5801 is the port the vision service is running on (default). See BB_VISION_PORT in the configuration docs.

Filename of the saved file is the current date and time in the format YYYYMMDD-HHMMSS.mp4.

Origin

Some of this code was originally pilfered from https://github.com/adeept/Adeept_RaspTank/blob/a6c45e8cc7df620ad8977845eda2b839647d5a83/server/app.py

Which looks like it was in turn pilfered from https://blog.miguelgrinberg.com/post/flask-video-streaming-revisited

Thank you, @adeept and @miguelgrinberg!

<img src="http://localhost:5001/video_feed" />

{
    "recognition": [
        {
            "bounding_box": [x1, y1, x2, y2],
            "classification": "person",
            "confidence": 0.99
        },
        {
            "bounding_box": [x1, y1, x2, y2],
            "classification": "dog",
            "confidence": 0.75
        }
    ]
}

sudo apt install -y ffmpeg libx264-dev

curl http://localhost:5801/record_video

send_stats

async def send_stats(_request: Request) -> Response

Return the FPS and other stats of the vision service.

pause_recognition

async def pause_recognition(_request: Request) -> Response

Use a GET request to pause the recognition provider.

resume_recognition

async def resume_recognition(_request: Request) -> Response

Use a GET request to resume the recognition provider.

record_video

async def record_video(request: Request) -> Response

Record the video feed to a file.

recorded_video

async def recorded_video(_request: Request) -> Response

Returns json array of string filenames (without extension) of all recorded video files. The filenames can be used to download the using a url like http://<ip>:<port>/recorded_video/<filename>.mp4 or http://<ip>:<port>/recorded_video/<filename>.jpg for the thumbnail image.

get_recorded_video_file

async def get_recorded_video_file(request: Request) -> Response

Sends a recorded video file.

video_feed

async def video_feed(request: Request) -> StreamResponse

MJPEG video streaming route. This is used as the src attribute of an html tag.