8

It's been demonstrated how to feed MediaCodec with Surface input like the CameraPreview, but are there practical ways of buffering this input before submission to MediaCodec?

In my experiments, a Galaxy Nexus experiences unacceptable hiccups in producing audio / video streams using the direct, synchronous encoding method in CameraToMpegTest.java

When using MediaCodec with byte[] or ByteBuffer input, we can submit unencoded data to a ExecutorService or similar queue for processing to ensure no frames are dropped, even if the device experiences spikes in CPU usage out of our application's control. However, due to the requirement of performing color format conversion between Android's Camera and MediaCodec this method is unrealistic for high resolution, live video.

Thoughts:

  1. Is there a way to feed the NativePixmapType created with EGL14.eglCopyBuffers(EGLDisplay d, EGLSurface s, NativePixmapType p) to MediaCodec?

  2. Can anyone from Android comment on whether harmonizing ByteBuffer formats between the Camera and MediaCodec is on the roadmap?

Community
  • 1
  • 1
dbro
  • 1,718
  • 1
  • 20
  • 34
  • So, did you find a way to solve the hiccups when passing frames to MediaCodec's input? In my case, I am buffering the output of the MediaCodec when feeding it to MediaMuxer. But I am experiencing hiccups when calling swapBuffers() to feed MediaMuxer's input. – Petrakeas Oct 27 '15 at 16:33

1 Answers1

5

You really don't want to copy the data at all. Allocating storage for and copying a large chunk of data can take long enough to kill your frame rate. This generally rules out byte[] and ByteBuffer[] solutions, even if you didn't have to do a U/V plane swap.

The most efficient way to move data through the system is with a Surface. The trick is that a Surface isn't a buffer, it's an interface to a queue of buffers. The buffers are passed around by reference; when you unlockCanvasAndPost() you're actually placing the current buffer onto a queue for the consumer, which is often in a different process.

There is no public mechanism for creating a new buffer and adding it to the set used by the queue, or for extracting buffers from the queue, so you can't implement a DIY buffering scheme on the side. There's no public interface to change the number of buffers in the pool.

It'd be useful to know what it is that's causing the hiccups. The Android tool for analyzing such issues is systrace, available in Android 4.1+ (docs, example, bigflake example). If you can identify the source of the CPU load, or determine that it's not CPU but rather some bit of code getting tangled up, you'll likely have a solution that's much easier than adding more buffers to Surface.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • Great advice! Systrace really helped out. Is it possible to nest `Trace` segments? I can't seem to get segments that begin and end within a parent to appear. – dbro Oct 11 '13 at 23:23
  • It should work. I tried a simple example (two `beginSection()` calls at different points in `onCreate()`, two `endSection()` calls right at the end) and got what I expected. – fadden Oct 12 '13 at 00:13
  • I'm seeing "hiccups" on tablets while trying to stream video, either from a h264 TS file on device or over UDP (live camera feed or streaming H264 TS from a server) A galaxy note 10.1 with 4.1.2 (i think, its not here in front of me) hiccups and also always returns BufferInfo.size of 0 (we've since upped our minimum sdk to 18 (4.3) as thats when CTS for codec went in)... however Lenovo Yoga 10 @4.3 has solid stuttering during playback...it appears due to the fact that input buffers are rarely available... – PapaWhiskey Jun 16 '14 at 21:38
  • I am using a similar pipeline to the Grafika "Show and capture video" example. Calling swapBuffers() to deliver the buffer to MediaCodec can get quite slow. Performing Android alpha animations at the same time can degrade the swapBuffers() performance even more. Perhaps GPU's fill rate is the bottleneck. What is the solution proposed by this answer? – Petrakeas Oct 27 '15 at 16:25
  • @Petrakeas: you should create a new question, rather than comment on an existing answer. Use tags for [android] [mediacodec] [grafika], and describe in detail what you're trying to do and how you're measuring performance. – fadden Oct 27 '15 at 17:20
  • @fadden OK. I'm in the process of creating a simple demo that reproduces the problem before posting the question, so I thought of commenting here first. – Petrakeas Oct 27 '15 at 17:26