How to identify a group of pixels as approximately rectangular?

Question

What is the mathematical logic to consider the group of pixels on left as rectangular?

Answer 1

Here's the implementation of the idea in the comments. The approach is:

1. Find the rotated rectangular bounding box
2. Calculate the area of this rotated bounding box and the area of the contour
3. Compare the two. If the area of the contour is say at least 80% (or any arbitrary threshold value) of the rotated bounding box area then we consider it as rectangular

---

Here's a visualization of the image processing pipeline

Input image -> Threshold -> Detected rotated rectangular bounding box -> Mask

Contour area: 17719.0
Mask area: 20603.0
Compared area percentage: 86.002%
It is a rectangle!

For the other image

Input image -> Threshold -> Detected rotated rectangular bounding box -> Mask

Contour area: 13395.5
Mask area: 19274.5
Compared area percentage: 69.499%
It is not a rectangle!

You didn't specify a language, so here's a simple implementation using Python OpenCV

import cv2
import numpy as np

# Load image, convert to grayscale, Otsu's threshold for binary image
image = cv2.imread('1.png')
mask = np.zeros(image.shape, dtype=np.uint8)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

# Find contours, find rotated rectangle, find contour area
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
contour_area = cv2.contourArea(cnts[0])
print('Contour area: {}'.format(contour_area))
rect = cv2.minAreaRect(cnts[0])
box = np.int0(cv2.boxPoints(rect))
cv2.drawContours(image, [box], 0, (36,255,12), 3)

# Find area of rotated bounding box and draw onto mask image
mask_area = cv2.contourArea(box)
cv2.drawContours(mask, [box], 0, (255,255,255), -1)
print('Mask area: {}'.format(mask_area))

# Compare areas and calculate percentage
rectangular_threshold = 80
percentage = (contour_area / mask_area) * 100
print('Compared area percentage: {:.3f}%'.format(percentage))
if percentage > rectangular_threshold:
    print('It is a rectangle!')
else:
    print('It is not a rectangle!')

# Display
cv2.imshow('image', image)
cv2.imshow('thresh', thresh)
cv2.imshow('mask', mask)
cv2.waitKey()

Answer 2

This is pretty similar to one of the comments (bounding box and check the internal area). I suggest you use the equations for area and perimeter to make a more robust and configurable shape tester.

-Obtain a fitted bounding box around the shape of interest.

-Check area: blob pixel count should be approximately equal to A(blob):L*W(rotated bounding box)

-Check perimeter: Run a edge transform (like Canny or equivalent) and compare the number of edge pixels to rectangular perimeter equation: P(blob):2W+2H(rotated bounding box)

---

-Running a hough line on the edge image and checking orthagonality could be another fitness estimator to be considered if it that is an important parameter to you.

Basically any time you are checking for shape fitness, you often want to check various estimators of fitness based around what things about the shapes you care about (ie clean edges, perpendicular corners, internal fill, straight edges, etc), and you weight or bound each one of those estimators to create the appropriate combined specificity that you are after.

Edit: (Code and additional explanation). So to be clear for your example I agree that area is the strongest fitness estimate. But, in shape identification in general (circles, squares, rectangles, complex shapes like leaves, etc), it is often useful to track several estimators of fitness so that you can really dial in what you care about and what you don't. I added an additional (completely contrived) shape to your sample to demonstrate where area alone may be less reliable (in-cuts in an otherwise clean rectangle). As you can see, area is very sensitive to external shape errors like protrusions, but perimeter is more sensitive to internal shape errors. (again this may or may not matter to you, but imo its interesting and could be helpful). I added one additional parameter (perimeter to area ratio), this is a neat metric that is used in estimating shape complexity (specifically in leaves).

Code:

#include <stdio.h>
#include <opencv2/opencv.hpp>
#include <Windows.h>
#include <string>

using namespace std;
using namespace cv;

void drawOrientedBoundingBox(Mat img, RotatedRect obb, Scalar color)
{
    Point2f vertices[4];
    obb.points(vertices);
    for (int i = 0; i < 4; i++)
        line(img, vertices[i], vertices[(i + 1) % 4], color, 2);
}

int main(int argc, char** argv)
{
    //play with these to dial in what you care about (across multiple sample images)
    double areaFitHigh = 1.2;
    double areaFitLow = 0.8;
    double perimFitHigh = 1.05;
    double perimFitLow = 0.85;

    std::string fileName = "C:/Local Software/voyDICOM/resources/images/RectTester.jpg";
    Mat tempImage = imread(fileName, cv::IMREAD_GRAYSCALE);

    Mat bwImg;
    cv::threshold(tempImage, bwImg, 0, 255, cv::THRESH_OTSU);

    Mat contImg = Mat(bwImg.rows, bwImg.cols, CV_8UC3, Scalar(0, 0, 0));

    vector<vector<Point> > contours;
    findContours(bwImg, contours, RETR_LIST, CHAIN_APPROX_NONE);

    bool areaFitnessFlag = false;
    bool perimFitnessFlag = false;

    for (size_t i = 0; i < contours.size(); i++)
    {
        std::string contourName = "S_" + std::to_string(i);
        std::cout << "-------------Contour Detected------------" << std::endl;
        std::cout << contourName << std::endl;

        if (contours[i].size() >= 2 * bwImg.cols + 2 * bwImg.rows - 4)
        {
            std::cout << "image boundary... don't know how to make findContours not detect this up front" << std::endl;
            continue;
        }

        RotatedRect obb = minAreaRect(contours[i]);

        //draw graphics for debug purposes
        drawOrientedBoundingBox(contImg, obb, Scalar(255, 0, 0));
        drawContours(contImg, contours, static_cast<int>(i), Scalar(0, 0, 255), 2);
        putText(contImg, contourName, obb.center, cv::FONT_HERSHEY_DUPLEX, 1, cv::Scalar(0, 255, 0), 1, false);

        //perform fitness checks
        double areaBlob = contourArea(contours[i]);
        double areaOBB = obb.size.area();

        double perimeterBlob = contours[i].size();
        double perimeterOBB = 2 * obb.size.width + 2 * obb.size.height;

        double perimToArea = 0;
        if (areaBlob > 0) { perimToArea = perimeterBlob / areaBlob; }

        std::cout << "area: " << areaBlob << " , " << areaOBB << std::endl;
        std::cout << "perimeter: " << perimeterBlob << " , " << perimeterOBB << std::endl;
        std::cout << "Perimeter to Area Ratio: " << perimToArea << std::endl;

        double areaFitness = 0;
        if (areaOBB > 0) { areaFitness = areaBlob / areaOBB; }
        std::cout << "Area Fitness: " << areaFitness << std::endl;

        double perimeterFitness = 0;
        if (perimeterOBB > 0) { perimeterFitness = perimeterBlob / perimeterOBB; }
        std::cout << "Perimeter Fitness: " << perimeterFitness << std::endl;


        if (areaFitness > areaFitHigh || areaFitness < areaFitLow)
        { areaFitnessFlag = false; }
        else
        { areaFitnessFlag = true; }

        if (perimeterFitness > perimFitHigh || perimeterFitness < perimFitLow)
        { perimFitnessFlag = false; }
        else
        { perimFitnessFlag = true; }

        if (areaFitnessFlag && perimFitnessFlag)
        { std::cout << "This is a rectangle!" << std::endl; }
        else
        { std::cout << "This is not a rectangle..." << std::endl; }
    }

    namedWindow("Original", WINDOW_AUTOSIZE);
    imshow("Original", tempImage);

    namedWindow("Thresh", WINDOW_AUTOSIZE);
    imshow("Thresh", bwImg);

    namedWindow("Contours", WINDOW_AUTOSIZE);
    imshow("Contours", contImg);

    waitKey(0);
    system("pause");
    return 0;
}

Picture of results:

Useful links:

https://docs.opencv.org/master/df/dee/samples_2cpp_2minarea_8cpp-example.html

https://docs.opencv.org/master/db/dd6/classcv_1_1RotatedRect.html

https://learnopencv.com/contour-detection-using-opencv-python-c/

Draw rotated rectangle in opencv c++