OpenCV image matching - form photo vs form template

Question

I'm trying to detect wether a photo represents a predefined formular template filled with data.

I'm new to image processing and OpenCV but my first attempt is to use FlannBasedMatcher and compare the count of keypoints detected.

Is there a better way to do this?

filled-form.jpg

form-template.jpg

import numpy as np
import cv2
from matplotlib import pyplot as plt
MIN_MATCH_COUNT = 10
img1 = cv2.imread('filled-form.jpg',0)          # queryImage
img2 = cv2.imread('template-form.jpg',0) # trainImage
# Initiate SIFT detector
sift = cv2.xfeatures2d.SIFT_create()
# find the keypoints and descriptors with SIFT
kp1, des1 = sift.detectAndCompute(img1,None)
kp2, des2 = sift.detectAndCompute(img2,None)
FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(des1,des2,k=2)
# store all the good matches as per Lowe's ratio test.
good = []
for m,n in matches:
  if m.distance < 0.7*n.distance:
    good.append(m)
if len(good)>MIN_MATCH_COUNT:
  print "ALL GOOD!" 
else:
  print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
  matchesMask = None

Answer 1

I think that using SIFT and a keypoints matcher is the most robust approach to this problem. It should work fine with many different form templates. However, SIFT algorithm being patented, here is another approach that should work well too:

Step 1: Binarize

• Threshold your photo and the template form using THRESH_OTSU tag.
• Invert the two binary result Mats with the bitwise_notfunction.

Step 2: Find the forms' bounding rect

For the two binary Mats from Step 1:

• Find the largest contour.
• Use approxPolyDPto approximate the found contour to a quadrilateral (see picture above).

In my code, this is done inside getQuadrilateral().

Step 3: Homography and Warping

• Find the transformation between the two forms' bounding rect with findHomography
• Warp the photo's binary Mat using warpPerspective (and the homography Mat computed previously).

Step 4: Comparison between template and photo

• Dilate the template form's binary Mat.
• Subtract the warped binary Mat and the dilated template form's binary Mat.

This allows to extract the filled informations. But you can also do it the other way around:

Template form - Dilated Warped Mat

In this case, the result of the subtraction should be totally black. I would then use mean to get the average pixel's value. Finally, if that value is smaller than (let's say) 2, I would assume the form on the photo is matching the template form.

---

Here is the C++ code, it shouldn't be too hard to translate to Python :)

vector<Point> getQuadrilateral(Mat & grayscale)
{
    vector<vector<Point>> contours;
    findContours(grayscale, contours, RETR_EXTERNAL, CHAIN_APPROX_NONE);

    vector<int> indices(contours.size());
    iota(indices.begin(), indices.end(), 0);

    sort(indices.begin(), indices.end(), [&contours](int lhs, int rhs) {
        return contours[lhs].size() > contours[rhs].size();
    });

    vector<vector<Point>> polygon(1);
    approxPolyDP(contours[indices[0]], polygon[0], 5, true);
    if (polygon[0].size() == 4) // we have found a quadrilateral
    {
        return(polygon[0]);
    }
    return(vector<Point>());
}

int main(int argc, char** argv)
{
    Mat templateImg, sampleImg;
    templateImg = imread("template-form.jpg", 0);
    sampleImg = imread("sample-form.jpg", 0);
    Mat templateThresh, sampleTresh;
    threshold(templateImg, templateThresh, 0, 255, THRESH_OTSU);
    threshold(sampleImg, sampleTresh, 0, 255, THRESH_OTSU);

    bitwise_not(templateThresh, templateThresh);
    bitwise_not(sampleTresh, sampleTresh);

    vector<Point> corners_template = getQuadrilateral(templateThresh);
    vector<Point> corners_sample = getQuadrilateral(sampleTresh);

    Mat homography = findHomography(corners_sample, corners_template);

    Mat warpSample;
    warpPerspective(sampleTresh, warpSample, homography, Size(templateThresh.cols, templateThresh.rows));

    Mat element_dilate = getStructuringElement(MORPH_ELLIPSE, Size(8, 8));
    dilate(templateThresh, templateThresh, element_dilate);

    Mat diff = warpSample - templateThresh;

    imshow("diff", diff);

    waitKey(0);

    return 0;
}

I Hope it is clear enough! ;)

P.S. This great answer helped me to retrieve the largest contour.