10

I'm currently writing an android app where I need to cache video-frames so that I can easily go back and forth with little to no delay.

Right now I'm letting android decode the video frame by providing a Surface to the Configure call of the MediaCodec object and calling releaseOutputBuffer with the render flag set to true.

The only way I found to access the decoded surface data (besides decoding the returned bytebuffer whose format appears to be device-dependent) is to call updateTeximage on the SurfaceTexture linked to the Surface, attaching this to the GL_TEXTURE_EXTERNAL_OES target and rendering it to a GL_TEXTURE2D target texture I created myself in order to cache it.

I would like to optimize this caching process and be able to decode the frames on a different thread. Using my current method, this means that I would have to create another EGL context for the video decoder, share the context etc...

My question is: Is it possible to access the EGL-image or native buffer data associated with the Surface without calling updateTexImage?

That way I could cache the egl image (which does not require EGL context according to EGL_ANDROID_image_native_buffer). This would also cache in YUV format which would be much more storage-efficient than the raw RGB textures I'm caching now.

PaulG
  • 13,871
  • 9
  • 56
  • 78
Nico Cornelis
  • 103
  • 1
  • 4

1 Answers1

6

Short answer: no.

Longer answer: the Surface encapsulates a queue of buffers. (Edit: the system is now explained in some detail here.) When you call updateTexImage(), if a new frame of data is available, the buffer at the head is dropped and the next one in the queue becomes current. Calling updateTexImage() is necessary to see successive frames; there is no mechanism for examining buffers not at the head.

A SurfaceTexture wraps an instance of GLConsumer. This consumer requires the producer (the video decoder) to generate data in a format that can be used as a "hardware texture", i.e. something the device's GL implementation can understand. It may or may not be YUV. More to the point, the consumer doesn't require that the buffer be available to "software", which means you can't assume that you can access the data directly -- you need to use GLES. (See the gralloc header for the full list of flags.)

What would be nice here is the ability to copy the buffer from the head of the BufferQueue to a separate data structure (BufferArrayList?) without doing a format conversion, but there isn't a mechanism like that at present (Android 4.3). I don't know of a better way to go about it than what you describe (shared EGL contexts, etc).

Update: My office-mate had a suggestion: use a shader to render the buffer into two textures, one for the Y and for CbCr (in GLES 3 you can use an RG texture). That keeps all the manipulation in GLES without expanding into full RGB. Internally it'll convert the MediaCodec output to RGB and grind through it twice, but that's likely cheaper than copying it out to userspace and doing it yourself on the CPU.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • First of all, thank you very much for this detailed answer. I've done some more experimenting and tried accessing the buffers directly. I found the video output format to be 0x7fa30c03 on my device which isn't a standard format in AOSP. After some manual retiling I was able to store a YUV420 cache-frame for each decoded frame. Maybe there's a device-independent function that converts the bytebuffer to regular yuv? I tried the YuvImage class but it looks like the 0x7fa30c03 format is not implemented for this class. Thanks again for the reply. – Nico Cornelis Sep 12 '13 at 21:03
  • You're firmly in non-portable device-specific territory. 0x7fa30c03 is the proprietary OMX_QCOM_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka (declared in native/include/media/openmax/OMX_IVCommon.h); the closest thing to a decoder library is the vendor-specific `/system/lib/libI420colorconvert.so`, used by the Video Editor lib (https://android.googlesource.com/platform/frameworks/av/+/jb-mr2-release/libvideoeditor/lvpp/I420ColorConverter.cpp). Most of the YUV stuff in the framework is for use with the camera, which is better behaved than the video decoder when it comes to buffer formats. – fadden Sep 12 '13 at 21:43
  • Thanks again for the prompt response. I guess I'll stick with the GLES path for now. I went through the AOSP code once more and I found that the EGL image is created using the EGL_ANDROID_image_native_buffer extension. I'm not very familiar with native buffers. Maybe it's possible to cache the bytebuffers returned by the mediacodec and wrap it in a struct that can be passed as EGLClientBuffer to the eglCreateImage when I want to retreive it from cache? – Nico Cornelis Sep 12 '13 at 22:05
  • 1
    (Answer slightly updated.) One quirk of the media / graphics drivers is that they're not guaranteed symmetric: just because it copied OMX 0x7fa30c03 out into your `ByteBuffer` doesn't mean there's a matching gralloc buffer format. Maybe it did a straight memcpy, maybe it tweaked alignment or stride. Filling out a buffer might work for a given device and software release, but it'll be fragile at best. This may be of interest: http://snorp.net/2011/12/16/android-direct-texture.html (again, not using public APIs). If you want to dig in deeper, you could try replacing `GLConsumer`. – fadden Sep 12 '13 at 23:57