14

I have this situation where I use OpenCV to detect faces in front of the camera and do some ML on those faces. The issue that I have is that once I do all the processing, and go to grab the next frame I get the past, not the present. Meaning, I'll read whats inside the buffer, and not what is actually in front of the camera. Since I don't care which faces came in front of the camera while processing, I care what is now in front of the camera.

I did try to set the buffer size to 1, and that did help quite a lot, but I still will get at least 3 buffer reads. Setting the FPS to 1, also dose not help remove this situation 100%. Bellow is the flow that I have.

let cv = require('opencv4nodejs');

let camera = new cv.VideoCapture(camera_port);

camera.set(cv.CAP_PROP_BUFFERSIZE, 1);
camera.set(cv.CAP_PROP_FPS, 2);
camera.set(cv.CAP_PROP_POS_FRAMES , 1);

function loop()
{
    //
    //  <>> Grab one frame from the Camera buffer.
    //
    let rgb_mat = camera.read();

    //  Do to gray scale

    //  Do face detection

    //  Crop the image

    //  Do some ML stuff

    //  Do whats needs to be done after the results are in.

    //
    //  <>> Release data from memory
    //
    rgb_mat.release();

    //
    //  <>> Restart the loop
    //
    loop();
}

My question is:

Is it possible to remove the buffer all-together? And if so, how. If not, a why would be much appreciated.

Jeru Luke
  • 20,118
  • 13
  • 80
  • 87
David Gatti
  • 3,576
  • 3
  • 33
  • 64

3 Answers3

20

Whether CAP_PROP_BUFFERSIZE is supported appears quite operating system and backend-specific. E.g., the 2.4 docs state it is "only supported by DC1394 [Firewire] v 2.x backend currently," and for backend V4L, according to the code, support was added only on 9 Mar 2018.

The easiest non-brittle way to disable the buffer is using a separate thread; for details, see my comments under Piotr Kurowski's answer. Here Python code that uses a separate thread to implement a bufferless VideoCapture: (I did not have a opencv4nodejs environment.)

import cv2, Queue, threading, time

# bufferless VideoCapture
class VideoCapture:

  def __init__(self, name):
    self.cap = cv2.VideoCapture(name)
    self.q = Queue.Queue()
    t = threading.Thread(target=self._reader)
    t.daemon = True
    t.start()

  # read frames as soon as they are available, keeping only most recent one
  def _reader(self):
    while True:
      ret, frame = self.cap.read()
      if not ret:
        break
      if not self.q.empty():
        try:
          self.q.get_nowait()   # discard previous (unprocessed) frame
        except Queue.Empty:
          pass
      self.q.put(frame)

  def read(self):
    return self.q.get()

cap = VideoCapture(0)
while True:
  frame = cap.read()
  time.sleep(.5)   # simulate long processing
  cv2.imshow("frame", frame)
  if chr(cv2.waitKey(1)&255) == 'q':
    break

The frame reader thread is encapsulated inside the custom VideoCapture class, and communication with the main thread is via a queue.

This answer suggests using cap.grab() in a reader thread, but the docs do not guarantee that grab() clears the buffer, so this may work in some cases but not in others.

Ulrich Stern
  • 10,761
  • 5
  • 55
  • 76
1

I set cap value after reading each frame to None and my problem solved in this way:

import cv2
from PyQt5.QtCore import QThread

if __name__ == '__main__':
while True:
    cap = cv2.VideoCapture(0)
    ret, frame = cap.read()
    cv2.imshow('A', frame)
    cv2.waitKey(0)

    print('sleep!')
    QThread.sleep(5)
    print('wake up!')
    cap = None
Misam Mehmannavaz
  • 146
  • 1
  • 2
  • 10
0

I have the same problem but in C++. I didn't find proper solution in OpenCV but I found a workaround. This buffer accumulates a constant number of images, say n frames. So You can read n frames without analysis and read frame once more. That last frame will be live image from camera. Something like:

buffer_size = n;

for n+1
{
  // read frames to mat variable
}
// Do something on Mat with live image
Piotr Kurowski
  • 346
  • 1
  • 3
  • So, the idea would be: find out the last frame in the buffer, use if if there is any, ask the camera for more if no frame was returned? Yes, No, Maybe? :) – David Gatti Jan 31 '19 at 14:11
  • The catch is you typically do not know how many frames to read at a certain point in time. To fix this, one could, e.g., time how long the `cap.read()` takes, and assume it was from the buffer if it was fast. But this does not work well, e.g., if the frame processing takes just about the time between frames, in which case one would assume the frame read right after processing was from the buffer and end up dropping every other frame. – Ulrich Stern Feb 07 '19 at 23:21
  • You have to check how many frames are in buffer. In my case it was always 5, but it isn't an const number. In my example You should read 5+1 frames to be sure that last frame will be actual. – Piotr Kurowski Feb 08 '19 at 12:51
  • Assuming there are always a certain number of frames in the buffer leads to brittle code: the constant needs to be adjusted, e.g., if the analysis code is changed, if the time the analysis takes changes for other reasons (library change, machine change, etc.), or if the frame rate changes (due to lighting change, different camera, etc.). – Ulrich Stern Feb 08 '19 at 15:18
  • So, how dose one check how many frames are in the buffer with OpenCV? Is there a function that will tell you that? – David Gatti Feb 11 '19 at 20:10
  • @DavidGatti, there is no such function. – Ulrich Stern Feb 18 '19 at 21:43