OpenCV display and processing webcam stream

Question

I was looking for process a stream with opencv coming from webcam. My goal is to detect rectangles for each frames. I'm able to display the result but the framerate is really slow, ~1 fps.

Here is an overview of my source code:

int main( int argc, char** argv ) {

CvCapture* cap=cvCreateCameraCapture(0);
cvNamedWindow("LiveFeed",CV_WINDOW_AUTOSIZE);

    while(true) {
    frame=cvQueryFrame(cap);
    if(!frame) 
        printf("\n no");
    else {
        Mat mat_img(frame);
        std::vector<std::vector<cv::Point>> rectangle = findSquaresInImage(mat_img);
        rectangle=filterRectangleSimilaire(rectangle,20.0);
        Mat res=debugSquares(rectangle,mat_img);

        cvShowImage("LiveFeed",new IplImage(res));//new IplImage(res));
    }
    char c=cvWaitKey(33);
    if(c==27)
        break;
}

cvReleaseCapture(&cap);
cvDestroyAllWindows();
return 0;
}

I would like to know if it's possible to make it threaded to gain at least 30fps ?

I'm working on windows 8.1, Visual 2010, c++ & OpenCV 2.4.10

score 3 · Answer 1 · answered Jan 21 '15 at 18:42

You could use threads before while(true) loop. Of course, reading frames from the camera and displaying them would have to be serialized.

I see you're using C++ std::vector, so I would suggest you use C++11 threads and mutex. The most performatic way I can think of is openning all threads at startup (before the while(true) loop) and make then receive the frames captured by the main thread. All threads would be locked on the same mutex for reading the frame queue and on a second mutex for displaying it. The frame queue may be implemented as:

struct frameQueue{
    std::vector<Mat> frames;
    std::mutex mtx;
};

Depending on the size of your project and side effects, it may even be declared global.

I assume you experiement this solution and it should worked. I asked about thread solution but i optimize/find some different code for image processing from stacoverflow and now i have at least 30 fps. Anyway, thanks for your solution — Erwan Douaille, Jan 27 '15 at 15:37

score 2 · Accepted Answer · edited May 23 '17 at 10:27

I fixed this issue using another solution found on stackoverflow. Here is my complete code:

Mat mat_img(frame);
Mat gray_mat_img, threshold_mat_img, detected_edges;
std::vector<std::vector<cv::Point>> contours;
vector<Vec4i> hierarchy;

cvtColor(mat_img, gray_mat_img, CV_BGR2GRAY);
blur(gray_mat_img, gray_mat_img, Size(10,10));
cv::threshold(gray_mat_img, threshold_mat_img, 140, 255, cv::THRESH_BINARY | cv::THRESH_OTSU);
Canny(threshold_mat_img, detected_edges, 100, 100, 3);
cv::findContours(detected_edges, contours, hierarchy, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE, Point(0, 0));

RNG rng(12345);
//contours = filterRectangleSimilaire(contours, 10);
//if (contours.size() < 5) {
vector<Rect> boundRect( contours.size() );
vector<Point> approx;


for (size_t i = 0; i < contours.size(); i++) {
    approxPolyDP(Mat(contours[i]), approx, arcLength(Mat(contours[i]), true)*0.02, true);
    if (approx.size() == 4 && fabs(contourArea(Mat(approx))) > 100 && isContourConvex(Mat(approx))) {
        double maxCosine = 0;
        for (int j = 2; j < 5; j++) {
            double cosine = fabs(angle(approx[j%4], approx[j-2], approx[j-1]));
            maxCosine = MAX(maxCosine, cosine);
        }
        if (maxCosine < 0.3) {
            Scalar color = Scalar(rng.uniform(0, 255), rng.uniform(0,255), rng.uniform(0,255));
            boundRect[i] = boundingRect( Mat(approx) );
            rectangle( mat_img, boundRect[i].tl(), boundRect[i].br(), color, 2, 8, 0 );
            drawContours(mat_img, contours, i, color, 2, 8, hierarchy, 0, Point());
        }
    }
}

Here is my reference: OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection

OpenCV display and processing webcam stream

2 Answers2