Python OpenCV streaming from camera - multithreading, timestamps

Question

I ran simple python script on Raspberry Pi 3. This script is responsible to open video device and stream data (800x600) to HTTP endpoint using MJPEG. When I receive this stream one of my Raspberry Pi cores works on 100%. It possible to run OpenCV with multi threading?

This is my code

import cv2
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import time
import argparse
import socket as Socket    
camera = None  

def setUpCameraCV():
    global camera
    camera = cv2.VideoCapture(0)

class mjpgServer(BaseHTTPRequestHandler):

    ip = None
    hostname = None

    def do_GET(self):

        print('connection from:', self.address_string())

        if self.ip is None or self.hostname is None:
            self.ip, _ = 0.0.0.0
            self.hostname = Socket.gethostname()

        if self.path == '/mjpg':

            self.send_response(200)
            self.send_header('Cache-Control', 'no-cache')
            self.send_header('Pragma', 'no-cache')
            self.send_header('Connection', 'close')
            self.send_header(
                'Content-type',
                'multipart/x-mixed-replace; boundary=mjpegstream'
            )
            self.end_headers()

            while True:
                if camera:
                    ret, img = camera.read()

                else:
                    raise Exception('Error, camera not setup')

                if not ret:
                    print('no image from camera')
                    time.sleep(1)
                    continue

                ret, jpg = cv2.imencode('.jpg', img)
                
                self.end_headers()
                self.wfile.write('--mjpegstream')
                self.end_headers()

                self.send_header('Content-type', 'image/jpeg')
                self.send_header('Content-length', str(jpg.size))
                self.end_headers()
                self.wfile.write(jpg.tostring())    

def main():
    try:
        setUpCameraCV()         
        mjpgServer.ip = 0.0.0.0
        mjpgServer.hostname = Socket.gethostname()
        server = HTTPServer((ipv4, args['port']), mjpgServer)
        print("server started on {}:{}".format(Socket.gethostname(), args['port']))
        server.serve_forever()

    except KeyboardInterrupt:
        print('KeyboardInterrupt')

    server.socket.close()


if __name__ == '__main__':
    main()

Another question, how to get timestamp of each frame on the client side (receiver) it possible?

Take a look [here](https://www.pyimagesearch.com/2015/12/28/increasing-raspberry-pi-fps-with-python-and-opencv/) — GPPK, Mar 11 '19 at 10:15

nathancy · Accepted Answer · 2022-05-06T03:33:53.900

Using threading to handle I/O heavy operations (such as reading frames from a webcam) is a classic programming model. Since accessing the webcam/camera using cv2.VideoCapture().read() is a blocking operation, our main program is stalled until the frame is read from the camera device and returned to our script. Essentially the idea is to spawn another thread to handle grabbing the frames in parallel instead of relying on a single thread (our 'main' thread) to grab the frames in sequential order. This will allow frames to be continuously read from the I/O thread, while our root thread processes the current frame. Once the root thread finishes processing its frame, it simply needs to grab the current frame from the I/O thread without having to wait for blocking I/O operations.

Thus we can improve performance by creating a new thread that does nothing but poll for new frames while our main thread handles processing the current frame. For an implementation to handle multiple camera streams, take a look at capture multiple camera streams with OpenCV

from threading import Thread
import cv2, time
 
class VideoStreamWidget(object):
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        # Start the thread to read frames from the video stream
        self.thread = Thread(target=self.update, args=())
        self.thread.daemon = True
        self.thread.start()

    def update(self):
        # Read the next frame from the stream in a different thread
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(.01)
    
    def show_frame(self):
        # Display frames in main program
        cv2.imshow('frame', self.frame)
        key = cv2.waitKey(1)
        if key == ord('q'):
            self.capture.release()
            cv2.destroyAllWindows()
            exit(1)

if __name__ == '__main__':
    video_stream_widget = VideoStreamWidget()
    while True:
        try:
            video_stream_widget.show_frame()
        except AttributeError:
            pass

Related camera/IP/RTSP, FPS, video, threading, and multiprocessing posts

Curious, i) Why is the time.sleep(.01) used? ii) Is it needed when reading from real IP cameras? Because the solution I'm working on needs real time, but this 100ms addition is a lot of delay when looked at, from a bigger picture. — Aakash Basu, Nov 09 '20 at 10:06
@AakashBasu time.sleep is used because the camera can only poll so fast, there's no need to poll at a ridiculous rate if there's no new frames in the buffer. I highly recommend it, you would have to look at your IP camera specs. Most IP cameras are in the 30-60 FPS range so polling any faster will have no benefit. You can reduce the delay or take it out entirely, I found out that the delay was useful for my camera — nathancy, Nov 09 '20 at 22:12
So a good point of reference would be 1 second / fps? e.g. on a 30 fps camera you should poll at 1/30 = 0.033 seconds or less? — Josh, Feb 09 '21 at 16:12
@Josh Sounds about right, my cameras are only capable of 30 FPS. A good idea is to poll a bit faster then your camera's capacity to ensure that you always get the latest frame. Every situation is different so just experiment to determine what value produces smooth frames — nathancy, Feb 17 '21 at 00:48
this code never checks `self.status`. further, the check for `capture.isOpened()` is wrong. it's supposed to be checked ONCE, BEFORE any loops are entered. this code simply causes an infinite loop instead. that is BAD. — Christoph Rackwitz, Aug 22 '21 at 11:03

Python OpenCV streaming from camera - multithreading, timestamps

1 Answers1

Linked

Related