Converting YUV_420_888 to JPEG and saving file results distorted image

Question

I've used the ImageUtil class provided in https://stackoverflow.com/a/40152147/2949966 within my git repo: https://github.com/ahasbini/cameraview/tree/camera_preview_imp (note the implementation is in camera_preview_imp branch) to implement a frame preview callback. An ImageReader is set to preview frames in the ImageFormat.YUV_420_888 format which will be converted into ImageFormat.JPEG using the ImageUtil class and send it to the frame callback. The demo app saves a frame from the callback to a file every 50 frames. All of the saved frame images are coming out distorted similar to below:

If I've changed the ImageReader to use ImageFormat.JPEG instead by doing the following changes in Camera2:

mPreviewImageReader = ImageReader.newInstance(previewSize.getWidth(),
    previewSize.getHeight(), ImageFormat.JPEG, /* maxImages */ 2);
mCamera.createCaptureSession(Arrays.asList(surface, mPreviewImageReader.getSurface()),
    mSessionCallback, null);

the image is coming properly without any distortions however the frame rate drops significantly and the view starts to lag. Hence I believe the ImageUtil class is not converting properly.

where I can see `onImageAvailable(ImageReader reader)` (ImageReader.OnImageAvailableListener) method? — Volodymyr Kulyk, May 17 '17 at 10:33
in ```Camera2``` class within the ```mOnPreviewAvailableListener``` variable. — ahasbini, May 17 '17 at 10:34
there you go: https://github.com/ahasbini/cameraview/blob/camera_preview_imp/library/src/main/api21/com/google/android/cameraview/Camera2.java#L188 — ahasbini, May 17 '17 at 10:36
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/144450/discussion-between-ahasbini-and-volodymyr-kulyk). — ahasbini, May 17 '17 at 10:37

score 45 · Accepted Answer · answered Aug 28 '17 at 20:24

Solution provided by @volodymyr-kulyk does not take into consideration the row stride of the planes within the image. Below code does the trick (image is of android.media.Image type):

data = NV21toJPEG(YUV420toNV21(image), image.getWidth(), image.getHeight(), 100);

And the implementations:

private static byte[] NV21toJPEG(byte[] nv21, int width, int height, int quality) {
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    YuvImage yuv = new YuvImage(nv21, ImageFormat.NV21, width, height, null);
    yuv.compressToJpeg(new Rect(0, 0, width, height), quality, out);
    return out.toByteArray();
}

private static byte[] YUV420toNV21(Image image) {
    Rect crop = image.getCropRect();
    int format = image.getFormat();
    int width = crop.width();
    int height = crop.height();
    Image.Plane[] planes = image.getPlanes();
    byte[] data = new byte[width * height * ImageFormat.getBitsPerPixel(format) / 8];
    byte[] rowData = new byte[planes[0].getRowStride()];

    int channelOffset = 0;
    int outputStride = 1;
    for (int i = 0; i < planes.length; i++) {
        switch (i) {
            case 0:
                channelOffset = 0;
                outputStride = 1;
                break;
            case 1:
                channelOffset = width * height + 1;
                outputStride = 2;
                break;
            case 2:
                channelOffset = width * height;
                outputStride = 2;
                break;
        }

        ByteBuffer buffer = planes[i].getBuffer();
        int rowStride = planes[i].getRowStride();
        int pixelStride = planes[i].getPixelStride();

        int shift = (i == 0) ? 0 : 1;
        int w = width >> shift;
        int h = height >> shift;
        buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
        for (int row = 0; row < h; row++) {
            int length;
            if (pixelStride == 1 && outputStride == 1) {
                length = w;
                buffer.get(data, channelOffset, length);
                channelOffset += length;
            } else {
                length = (w - 1) * pixelStride + 1;
                buffer.get(rowData, 0, length);
                for (int col = 0; col < w; col++) {
                    data[channelOffset] = rowData[col * pixelStride];
                    channelOffset += outputStride;
                }
            }
            if (row < h - 1) {
                buffer.position(buffer.position() + rowStride - length);
            }
        }
    }
    return data;
}

Method was gotten from the following link.

I struggled for a whole afternoon for those strides until I reach this post. I wish to give you 100 upvotes!!!! — Sira Lam, May 02 '18 at 09:31
By the way, the conversion is a bit slow, it takes in my device from 40ms~140ms for each frame. — Sira Lam, May 02 '18 at 09:38
@SiraLam thanks for the feedback. You could open a bounty on the question and award it the answer if you'd like to ;P. With regards to the processing yes ur right it is a slow process, image conversation tends to be like that by nature. To be able to process more frames faster that would require some level of multithreading with some synchronization between the threads to achieve a pipelining effect. Since frame processing is around 100 ms, user wouldn't really feel a lag if viewing the camera stream — ahasbini, May 02 '18 at 09:48
I would consider that :P Well, user won't feel any lag on the preview since my preview is using another surface different from what I use for these frames. Once I use another thread to do the conversion work, the preview is still very smooth. In fact I am using these frames to do face detections and overlay something on those faces... And sad that using Camera 1 API this is super easy and fast at the same time :( — Sira Lam, May 02 '18 at 09:58
Xiaomi Mi A1, app was crashing using JPEG format for image reader, converted to YUV_420_888 then used your method. Super thanks. — Jumpa, Jun 22 '18 at 21:39
Great answer! But for some reason, the converted jpeg for me is rotated 90 deg clockwise. Anyone else experienced same thing? — Kashif, Nov 12 '18 at 05:46
This works perfectly on Xiaomi POCO M3 which otherwise would provide a corrupt NV21 byte array to the face detector. Thank you! — Matt from vision.app, Feb 28 '22 at 22:12
I've searched the whole internet, found every possible solution, native, renderscript, you name it. This is the only code which creates proper image on every device (so far) and is also the fastest. — Milan Markovic, Nov 19 '22 at 16:55

score 3 · Answer 2 · answered May 17 '17 at 10:54

Updated ImageUtil:

public final class ImageUtil {

    public static byte[] NV21toJPEG(byte[] nv21, int width, int height, int quality) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        YuvImage yuv = new YuvImage(nv21, ImageFormat.NV21, width, height, null);
        yuv.compressToJpeg(new Rect(0, 0, width, height), quality, out);
        return out.toByteArray();
    }

    // nv12: true = NV12, false = NV21
    public static byte[] YUV_420_888toNV(ByteBuffer yBuffer, ByteBuffer uBuffer, ByteBuffer vBuffer, boolean nv12) {
        byte[] nv;

        int ySize = yBuffer.remaining();
        int uSize = uBuffer.remaining();
        int vSize = vBuffer.remaining();

        nv = new byte[ySize + uSize + vSize];

        yBuffer.get(nv, 0, ySize);
        if (nv12) {//U and V are swapped
            vBuffer.get(nv, ySize, vSize);
            uBuffer.get(nv, ySize + vSize, uSize);
        } else {
            uBuffer.get(nv, ySize , uSize);
            vBuffer.get(nv, ySize + uSize, vSize);
        }
        return nv;
    }

    public static byte[] YUV_420_888toI420SemiPlanar(ByteBuffer yBuffer, ByteBuffer uBuffer, ByteBuffer vBuffer,
                                                     int width, int height, boolean deInterleaveUV) {
        byte[] data = YUV_420_888toNV(yBuffer, uBuffer, vBuffer, deInterleaveUV);
        int size = width * height;
        if (deInterleaveUV) {
            byte[] buffer = new byte[3 * width * height / 2];

            // De-interleave U and V
            for (int i = 0; i < size / 4; i += 1) {
                buffer[i] = data[size + 2 * i + 1];
                buffer[size / 4 + i] = data[size + 2 * i];
            }
            System.arraycopy(buffer, 0, data, size, size / 2);
        } else {
            for (int i = size; i < data.length; i += 2) {
                byte b1 = data[i];
                data[i] = data[i + 1];
                data[i + 1] = b1;
            }
        }
        return data;
    }
}

Operations to write in file byte[] data as JPEG:

//image.getPlanes()[0].getBuffer(), image.getPlanes()[1].getBuffer()
//image.getPlanes()[2].getBuffer(), image.getWidth(), image.getHeight()
byte[] nv21 = ImageUtil.YUV_420_888toI420SemiPlanar(yBuffer, uBuffer, vBuffer, width, height, false);
byte[] data = ImageUtil.NV21toJPEG(nv21, width, height, 100);
//now write `data` to file

!!! do not forget to close image after processing !!!

image.close();

updates posted in chat: http://chat.stackoverflow.com/rooms/144450/discussion-between-ahasbini-and-volodymyr-kulyk — ahasbini, May 17 '17 at 11:35

score 0 · Answer 3 · answered Oct 04 '18 at 12:51

Camera2 YUV_420_888 to Jpeg in Java(Android):

@Override
public void onImageAvailable(ImageReader reader){
    Image image = null;

    try {
        image = reader.acquireLatestImage();
        if (image != null) {

            byte[] nv21;
            ByteBuffer yBuffer = mImage.getPlanes()[0].getBuffer();
            ByteBuffer uBuffer = mImage.getPlanes()[1].getBuffer();
            ByteBuffer vBuffer = mImage.getPlanes()[2].getBuffer();

            int ySize = yBuffer.remaining();
            int uSize = uBuffer.remaining();
            int vSize = vBuffer.remaining();

            nv21 = new byte[ySize + uSize + vSize];

            //U and V are swapped
            yBuffer.get(nv21, 0, ySize);
            vBuffer.get(nv21, ySize, vSize);
            uBuffer.get(nv21, ySize + vSize, uSize);

            String savingFilepath = getYUV2jpg(nv21);



        }
    } catch (Exception e) {
        Log.w(TAG, e.getMessage());
    }finally{
        image.close();// don't forget to close
    }
}

  public String getYUV2jpg(byte[] data) {
    File imageFile = new File("your parent directory", "picture.jpeg");//no i18n
    BufferedOutputStream bos = null;
    try {
        bos = new BufferedOutputStream(new FileOutputStream(imageFile));
        bos.write(data);
        bos.flush();
        bos.close();
    } catch (IOException e) {

        return e.getMessage();
    } finally {
        try {
            if (bos != null) {
                bos.close();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

    }
    return imageFile.getAbsolutePath();
}

Note: Handle the image rotation issue.

This doesnt work :( the format is still not jpeg after this. — Jorge Barraza Z, Sep 12 '19 at 00:49

score 0 · Answer 4 · answered Aug 05 '21 at 03:19

0

I think there is some confusion here with NV and YV formats of YUV. NV (semi-planar) have interleaved U/V. YV (planar) do not. So the conversions being done here are YV12/21 not NV12/21.

answered Aug 05 '21 at 03:19

programmer

1
1

Converting YUV_420_888 to JPEG and saving file results distorted image

4 Answers4

Linked

Related