How to crop and resize the section of an Image within a Rect bounding box?

Question

After detecting a face with CameraX and MLKit I need to pass the image to a custom TFLite model (I'm using this one), which detects a facemask. The model accepts images of 224x224 pixels, so I need to take out the part of ImageProxy#getImage() corresponding to Face#getBoundingBox() and resize it accordingly.

I've seen this answer which could have been fine but ThumbnailUtils.extractThumbnail() can't work with a Rect of 4 coordinates and it's relative to the center of the image, while the face's bounding box might be elsewhere.

The TFLite model accepts inputs like this:

val inputFeature0 = TensorBuffer
    .createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
    .loadBuffer(/* the resized image as ByteBuffer */)

Note that the ByteBuffer will have a size of 224 * 224 * 3 * 4 bytes (where 4 is DataType.FLOAT32.byteSize()).

Edit: I've cleaned up some of the old text because it was getting overwhelming. The code suggested below actually works: I just forgot to delete a piece of my own code which was already converting the same ImageProxy to Bitmap and it must have caused some internal buffer to be read until the end, so it was either necessary to rewind it manually or to delete that useless code altogether.

However, even if the cropRect is applied to the ImageProxy and the underlying Image, the resulting bitmap is still full size so there must be something else to do. The model is still returning NaN values, so I'm going to experiment with the raw output for a while.

fun hasMask(imageProxy: ImageProxy, boundingBox: Rect): Boolean {
    val model = MaskDetector.newInstance(context)
    val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)

    // now the cropRect is set correctly but the image itself isn't
    // cropped before being converted to Bitmap
    imageProxy.setCropRect(box)
    imageProxy.image?.cropRect = box

    val bitmap = BitmapUtils.getBitmap(imageProxy) ?: return false
    val resized = Bitmap.createScaledBitmap(bitmap, 224, 224, false)

    // input for the model
    val buffer = ByteBuffer.allocate(224 * 224 * 3 * DataType.FLOAT32.byteSize())
    resized.copyPixelsToBuffer(buffer)

    // use the model and get the result as 2 Floats
    val outputFeature0 = model.process(inputFeature0).outputFeature0AsTensorBuffer
    val maskProbability = outputFeature0.floatArray[0]
    val noMaskProbability = outputFeature0.floatArray[1]

    model.close()
    return maskProbability > noMaskProbability
}

Hi, you can look at my solution that I gave here https://stackoverflow.com/a/67348548/13300615. It should help you to get cropped image without converting it to bitmap. — Alex F., May 02 '21 at 10:03

score 1 · Accepted Answer · answered Feb 05 '21 at 19:37

1

We will provide a better way to handle the image processing when working with ML Kit.

For now, you could try this method: https://github.com/googlesamples/mlkit/blob/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/BitmapUtils.java#L74

It will convert the ImageProxy to Bitmap, and rotate it to upright. The bounding box from the face detection should be applied to the bitmap directly, which means you should be able to crop the bitmap with the Rect bounding box.

answered Feb 05 '21 at 19:37

Shiyu

875
4
5

Thanks, I can't wait for CameraX's future improvements. By saying that "the bounding box should be applied to the bitmap directly", you mean that after calling `detector.process()` with an `InputImage.fromBitmap()` I will have to do `imageProxy.cropRect = boundingBox` and convert `imageProxy` again into a bitmap? I'm sure I'm not getting something because it seems a bit convoluted. – rdxdkr Feb 06 '21 at 09:31
After fixing some stuff your suggested code finally worked, however the image is still not cropped. I've updated the question with the recent changes. – rdxdkr Feb 08 '21 at 22:27
1

`imageProxy.setCropRect(box)` would not crop the image for you. After resizing your image, you could use https://developer.android.com/reference/android/graphics/Bitmap#createBitmap(android.graphics.Bitmap,%20int,%20int,%20int,%20int) this method to crop the image with the detected boundingbox. – Shiyu Feb 11 '21 at 18:47
1

Yes, I had already found out that the best (and probably the only) thing to do was to create a new bitmap from a subarea of the other one, like `val cropped = Bitmap.createBitmap(bitmap, boundingBox.left, boundingBox.top, boundingBox.width(), boundingBox.height())`. – rdxdkr Feb 15 '21 at 09:24

How to crop and resize the section of an Image within a Rect bounding box?

1 Answers1

Linked