Tracking Objects by Shape from a live camera feed rather than an Image

Question

I have the below C++ code, which aims to detect shapes from a pre-designated image and draw around the shapes' perimeters. However, I wish to take this to the next step, and track the shapes from a camera feed rather than just an image. However, I am not to familiar with how I could make this transition.

#include <opencv2\opencv.hpp>
#include <opencv2\highgui\highgui.hpp>

int main()
{

    IplImage* img = cvLoadImage("C:/Users/Ayush/Desktop/FindingContours.png");

    //show the original image
    cvNamedWindow("Raw");
    cvShowImage("Raw", img);

    //converting the original image into grayscale
    IplImage* imgGrayScale = cvCreateImage(cvGetSize(img), 8, 1);
    cvCvtColor(img, imgGrayScale, CV_BGR2GRAY);

    //thresholding the grayscale image to get better results
    cvThreshold(imgGrayScale, imgGrayScale, 128, 255, CV_THRESH_BINARY);

    CvSeq* contours;  //hold the pointer to a contour in the memory block
    CvSeq* result;   //hold sequence of points of a contour
    CvMemStorage *storage = cvCreateMemStorage(0); //storage area for all contours

    //finding all contours in the image
    cvFindContours(imgGrayScale, storage, &contours, sizeof(CvContour), CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE, cvPoint(0, 0));

    //iterating through each contour
    while (contours) {
        //obtain a sequence of points of contour, pointed by the variable 'contour'
        result = cvApproxPoly(contours, sizeof(CvContour), storage, CV_POLY_APPROX_DP, cvContourPerimeter(contours)*0.02, 0);

        //if there are 3  vertices  in the contour(It should be a triangle)
        if (result->total == 3) {
            //iterating through each point
            CvPoint *pt[3];
            for (int i = 0; i < 3; i++) {
                pt[i] = (CvPoint*)cvGetSeqElem(result, i);
            }

            //drawing lines around the triangle
            cvLine(img, *pt[0], *pt[1], cvScalar(255, 0, 0), 4);
            cvLine(img, *pt[1], *pt[2], cvScalar(255, 0, 0), 4);
            cvLine(img, *pt[2], *pt[0], cvScalar(255, 0, 0), 4);

        }

        //if there are 4 vertices in the contour(It should be a quadrilateral)
        else if (result->total == 4) {
            //iterating through each point
            CvPoint *pt[4];
            for (int i = 0; i < 4; i++) {
                pt[i] = (CvPoint*)cvGetSeqElem(result, i);
            }

            //drawing lines around the quadrilateral
            cvLine(img, *pt[0], *pt[1], cvScalar(0, 255, 0), 4);
            cvLine(img, *pt[1], *pt[2], cvScalar(0, 255, 0), 4);
            cvLine(img, *pt[2], *pt[3], cvScalar(0, 255, 0), 4);
            cvLine(img, *pt[3], *pt[0], cvScalar(0, 255, 0), 4);
        }

        //if there are 7  vertices  in the contour(It should be a heptagon)
        else if (result->total == 7) {
            //iterating through each point
            CvPoint *pt[7];
            for (int i = 0; i < 7; i++) {
                pt[i] = (CvPoint*)cvGetSeqElem(result, i);
            }

            //drawing lines around the heptagon
            cvLine(img, *pt[0], *pt[1], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[1], *pt[2], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[2], *pt[3], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[3], *pt[4], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[4], *pt[5], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[5], *pt[6], cvScalar(0, 0, 255), 4);
            cvLine(img, *pt[6], *pt[0], cvScalar(0, 0, 255), 4);
        }

        //obtain the next contour
        contours = contours->h_next;
    }

    //show the image in which identified shapes are marked   
    cvNamedWindow("Tracked");
    cvShowImage("Tracked", img);

    cvWaitKey(0); //wait for a key press

    //cleaning up
    cvDestroyAllWindows();
    cvReleaseMemStorage(&storage);
    cvReleaseImage(&img);
    cvReleaseImage(&imgGrayScale);

    return 0;
}

Any help in this matter is more than appreciated. Thank you!

Answer 1

If you're using C++, I'd start by rewriting this to use the C++ OpenCV API, instead of this old C one -- IMHO it's easier to use. In the process, refactor the code into smaller functions and decouple processing from I/O. Finally, consider that a video is just a sequence of images. If you can process one image, then you can process a video, one frame at a time.

---

So, how to achieve this. Let's start from the top, and write a main() function.

To read a video stream, we'll use cv::VideoCapture. We'll begin by initializing (and making sure that worked), and preparing some named windows to display the input and output frames.

Then we will start handling the individual frames in an infinite loop, quitting only when frame acquisition fails or the user hits an escape key. In each iteration we will:

• Read the frame from the video stream (and make sure this succeeded)
• Process the frame (we'll write a function for this later)
• Display the raw and processed frames in our named windows
• Wait a bit, and check if user pressed escape key, handling that appropriately

Code:

int main()
{
    cv::VideoCapture cap(0); // open the video camera no. 0

    if (!cap.isOpened())  // if not success, exit program
    {
        std::cout << "Cannot open the video cam\n";
        return -1;
    }

    cv::namedWindow("Original", CV_WINDOW_AUTOSIZE);
    cv::namedWindow("Tracked", CV_WINDOW_AUTOSIZE);

    // Process frames from the video stream...
    for(;;) {
        cv::Mat frame, result_frame;

        // read a new frame from video
        if (!cap.read(frame)) {
            std::cout << "Cannot read a frame from video stream\n";
            break;
        }

        process_frame(frame, result_frame);

        cv::imshow("Original", frame);
        cv::imshow("Tracked", result_frame);
        if (cv::waitKey(20) == 27) { // Quit on ESC
            break;
        }
    }

    return 0;
}

NB: The use of cv::waitKey at an appropriate time is essential for the GUI to work. Read the documentation carefully.

---

With that done, it's time to implement our process_frame function, but first, let's make some useful global typedefs.

In the C++ API, a contour is a std::vector of cv::Point objects, and since more than one contour can be detected, we also need a std::vector of contours. Similarly, hierarchy is represented as a std::vector of cv::Vec4i objects. (the "is" is a lie-to-children, as it could be other data types too, but this is not important right now).

typedef std::vector<cv::Point> contour_t;
typedef std::vector<contour_t> contour_vector_t;
typedef std::vector<cv::Vec4i> hierarchy_t;

Let's work on the function -- it needs to take two parameters:

• Input frame (cv::Mat) which we don't want to modify, we just analyze it.
• Output frame, into which we draw the result of processing.

We need to:

• Copy the original frame into result, so that we can later draw over it.
• Make a grayscale version using cv::cvtColor, so that we can
• cv::threshold it, binarizing the image
• cv::findContours on the binary image
• Finally, process each detected contour (possibly drawing into the result frame).

Code:

void process_frame(cv::Mat const& frame, cv::Mat& result_frame)
{
    frame.copyTo(result_frame);

    cv::Mat feedGrayScale;
    cv::cvtColor(frame, feedGrayScale, cv::COLOR_BGR2GRAY);

    //thresholding the grayscale image to get better results
    cv::threshold(feedGrayScale, feedGrayScale, 128, 255, cv::THRESH_BINARY);

    contour_vector_t contours;
    hierarchy_t hierarchy;
    cv::findContours(feedGrayScale, contours, hierarchy, cv::RETR_LIST, cv::CHAIN_APPROX_SIMPLE);
    for (size_t k(0); k < contours.size(); ++k) {
        process_contour(result_frame, contours[k]);
    }
}

---

Last step, function to process a single contour. It needs:

• An image (cv::Mat) to draw on
• A contour to work with

First, we want to approximate a polygon, using fraction of the perimeter length (we can use cv::arcLength to calculate that) as a parameter. We will continue by processing this approximated contour.

Next, we want to handle 3 specific cases: triangles, quadrilaterals and heptagons. We want to draw the contour of each of those using a different colour, otherwise we don't do anything. To draw the sequence of lines making up the contour, we can use cv::polylines.

Code:

void process_contour(cv::Mat& frame, contour_t const& contour)
{
    contour_t approx_contour;
    cv::approxPolyDP(contour, approx_contour, cv::arcLength(contour, true) * 0.02, true);

    cv::Scalar TRIANGLE_COLOR(255, 0, 0);
    cv::Scalar QUADRILATERAL_COLOR(0, 255, 0);
    cv::Scalar HEPTAGON_COLOR(0, 0, 255);

    cv::Scalar colour;
    if (approx_contour.size() == 3) {
        colour = TRIANGLE_COLOR;
    } else if (approx_contour.size() == 4) {
        colour = QUADRILATERAL_COLOR;
    } else if (approx_contour.size() == 7) {
        colour = HEPTAGON_COLOR;
    } else {
        return;
    }

    cv::Point const* points(&approx_contour[0]);
    int n_points(static_cast<int>(approx_contour.size()));

    polylines(frame, &points, &n_points, 1, true, colour, 4);
}

NB: std::vector is guaranteed to be continuous. That's why we can safely take a pointer by getting the address of the first element (&approx_contour[0]).

---

NB: Avoid using

using namespace std;
using namespace cv;

For more info, see Why is “using namespace std” considered bad practice?