Understanding and speeding up addCallbackBuffer

Question

I want to understand how the preview buffer passed to addCallbackBuffer relates to the byte[] array passed through onPreviewFrame, which prompts the following related questions.

Q1. I am guessing that the buffer passed in addCallbackBuffer is used to store a new camera frame and that before onPreviewFrame is called, that buffer is copied into the data buffer passed through onPreviewFrame. If it is the case, that would mean that I can reuse my preview frame buffer by calling addCallbackBuffer as soon as I enter onPreviewFrame and note at the end of the function when I am done processing the buffer returned by onPreviewFrame. Is that correct?

Q2. I am also not clear about the mechanism for using two preview frame buffers. Say I have two private byte[] preview buffers added as follows during initialization:

addCallbackBuffer(mPreviewBuffer1);
addCallbackBuffer(mPreviewBuffer2);

How do I know which preview buffer was used when I am in onPreviewFrame so that I can re-add the correct preview frame buffer with addCallbackBuffer again?

private byte[] mPreviewBuffer1;
private byte[] mPreviewBuffer1;
...
public void onPreviewFrame(byte[] camera, Camera c) {
  ...
  // how do I decide which buffer to re-add?
  //c.addCallbackBuffer(mPreviewBuffer1);
  //c.addCallbackBuffer(mPreviewBuffer2);
  ...
}

Q3. Am I understanding correctly that another thread is responsible for acquiring the frame buffer, i.e. that as long as a preview buffer is in the queue, we will be capturing a frame while onPreviewFrame is executing? If that's not the case, having two call back buffers would not help with speed, would it?

score 3 · Accepted Answer · edited May 23 '17 at 10:28

Q1 yes, you can return the buffer to camera early if you don't care about its contents. You may not be able to read from this buffer after you call addCallbackBuffer() for it, or maybe you can read, but the pixel data will be wrong.

Q2 you can simply return to camera the buffer that you receive on the callback, i.e.

@Override public void onPreviewFrame(byte[] data, Camera camera) {
  …
  camera.addCallbackBuffer(data);
}

In this case you don't care if data == mPreviewBuffer1 or data == mPreviewBuffer2. But the following code is also OK:

private byte[][] mPrevieBuffers = new byte[4][];
@Override public void onPreviewFrame(byte[] data, Camera camera) {
  for (int i=0; i<mPreviewBuffers.length; i++) {
    if (data == mPreviewBuffers[i]) {
       processData(i);
    }
  }
}

Q3 correct, onPreviewFrame() executes in parallel with filling the other buffer, but all onPreviewFrame() callbacks are called on the same thread. If you want to process as many preview frames as possible, you should a) delegate processing to a worker thread (especially, on a multi-core device). Note that you can call safely addCallbackBuffer() from that other thread; b) start the camera on a separate Looper to have onPreviewFrame() decoupled from the UI thread.

Awesome. You clarified everything. In particular, realizing that there isn't any copy occurring and that the data pointer we get from onPreviewFrame is precisely one of the preview buffers added with the addCallbackBuffer was key! — Lolo, Sep 29 '15 at 18:09
There is a memcpy, but it is hidden from us: this is when a frame arrives from the camera HAL (system process) through Binder interface to your (user) process. The frame that is copied to buffer(s) we supply in `addCallbackBuffer()` is also used to display live preview. — Alex Cohn, Sep 29 '15 at 19:54

score 2 · Answer 2 · answered Sep 29 '15 at 04:20

This works for me:

public void surfaceChanged(SurfaceHolder holder, int format, int ww, int hh)
{
    if (mCamera == null) return;
    mCamera.stopPreview();
    mCamera.setPreviewCallback(null);
    //get camera parameters
    prepareSizeScreen();

    mCamera.setParameters(mMyCameraParameters.makeParameters(mCamera.getParameters(), mSizeScreen));
    try
    {
        mCamera.setPreviewDisplay(mHolder);
    } catch (Exception e)
    {
    }
    Camera.Size setSize = mCamera.getParameters().getPreviewSize();
    int bufferSize = setSize.width * setSize.height
            * ImageFormat.getBitsPerPixel(mCamera.getParameters().getPreviewFormat()) / 8;

    setupCallback(bufferSize);//this is what you are looking for


    mCamera.startPreview();
}

You can choose the amount of callbacks more callbacks more often you will get onPreviewFrame.

static final private int NUM_BUFFERS = 5;
private void setupCallback(int bufferSize)
{
    mCamera.setPreviewCallbackWithBuffer(this);
    for (int i = 0; i <= NUM_BUFFERS; ++i)
    {
        byte[] cameraBuffer = new byte[bufferSize];
        mCamera.addCallbackBuffer(cameraBuffer);
    }
}

When you get your call back you should use the very same data that you get to set another callback.

@Override
public void onPreviewFrame(byte[] data, Camera camera)
{//data has NV21 format        
    processData(data);
    camera.addCallbackBuffer(data);//same data is sent for another callback. So you will be managing NUM_BUFFERS at all times.
}

And dont forget onDestroy: public void surfaceDestroyed(SurfaceHolder holder) { if (mCamera != null) { mCamera.setPreviewCallback(null); mCamera.stopPreview(); mCamera.release(); } mCamera = null; }

this is too slow if you're using an older device (let's say android 4.4 common devices)... — Rafael Sanches, Mar 30 '18 at 06:50

Understanding and speeding up addCallbackBuffer

2 Answers2