7

I have a pretty good understanding of decoding with Android MediaCodec and feeding YUV through a Surface into an OpenGL texture. I would like to do something similar with Vulkan. However I have not been successful in finding any documentation or sample code.

My question is: how would I wire up the following pipeline?

MediaCodec Video Decoder ⇨ Surface ⇨ texture ⇨ Vulkan

Details

OpenGL Comparison

For comparison, in OpenGL case an Android Surface is constructed and used like so

 textureId = glGenTextures( &textureId )
 surface = new Surface( new SurfaceTexture( textureId ) )
 mediaCodec.configure( surface )
Peter Tran
  • 1,626
  • 1
  • 17
  • 26

2 Answers2

8

This is currently not possible, as there is no way to import memory objects from outside Vulkan, or any SDK Vulkan object that can export a Surface. Take a look at VK_KHX_external_memory and related extensions for how parts of this might work in the future.

EDIT 2018-05-23: This is now possible using the VK_ANDROID_external_memory_android_hardware_buffer extension and the extensions it depends on. You can use AImageReader_newWithUsage() to create an AImageReader compatible with GPU sampling. Get the ANativeWindow from that AImageReader and use it as the AMediaCodec's output surface. Then for each image you receive, get the AHardwareBuffer and import that into a VkDeviceMemory/VkImage pair using the extension.

fadden
  • 51,356
  • 5
  • 116
  • 166
Jesse Hall
  • 6,441
  • 23
  • 29
2

Just FYI: I don't know Android well.

Creating Vulkan "texture" is done with vkCreateImage. Unlike your OpenGL ES example there is some extra work managing the memory explicitly (with vkAllocateMemory and vkBindImageMemory).

Next step would be the hard one. There is apparently no SurfaceTexture for Vulkan (yet).

The work making that efficient from Vulkan side was released quite recently. I.e. VK_KHX_external_memory and associated extensions of Vulkan. So hopefully the official Android SurfaceTexture for Vulkan is comming too.

That being said, you could implement a new SurfaceTexture yourself. Or at least "import" Vulkan Image into OpenGL ES. The problem with this is:

  • the extensions are considered experimental UPDATE: not anymore
  • the drivers probably do not support the extensions anyway (yet)
  • it is probably not trivial to program, and also may require messing with some low-level linux API on Android.

So, best way for now would be to create some shim, which copies the data.
I would:

  1. It seems the MediaCodec is able to work with a ByteBuffer instead of Surface. And ByteBuffer can wrap raw bytes[].

  2. Create VkImages and VkBuffer. The Image on the Device memory (the resulting object we want). And the Buffer on the Host side (to facilitate copying).

  3. Map (vkMapMemory), and wrap the Host side Buffer with the ByteBuffer used by MediaCodec.

  4. Each time there's new data copy it with submitting vkCmdCopyBufferToImage in Vulkan.

I omitted a lot of boilerplate (especially synchronization), but hopefully you get the idea.

Alexander Ushakov
  • 5,139
  • 3
  • 27
  • 50
krOoze
  • 12,301
  • 1
  • 20
  • 34
  • Thanks for this answer. Would you actually expect that to be faster than using GL for large videos? It seems to me the extra copy will overwhelm any overhead reduction you get from using Vulkan instead of GL. – griffin2000 Jun 24 '17 at 19:31
  • @griffin2000 I would not expect raw performance gains for ordinary video playback in Vulkan, if that's what you meant by "large videos". – krOoze Jun 29 '17 at 21:05
  • But would it be significantly slower? My suspicion is doing what's described here would be slower than the GL equivalent. So that the overhead win in doing faster 3D calls in the rest of the app would be completely counteracted by copying all the video frames (unless you had a very low res video, or lots of 3D draw calls). Does that seem correct? – griffin2000 Jun 29 '17 at 21:08
  • @griffin2000 I don't feel confident answering that. Probably depends on the device, size and frequency of the copy, and what is the "3D calls in the rest of the app". But well..., modern RAM can push ungodly amounts of data quickly. I was offering the above approach only as a desperate or lazy workaround though. – krOoze Jun 29 '17 at 23:13
  • I need to process 4K video at 30 fps. This 'shim' solution was too slow due to copying; it resulted in less than 10fps. – Peter Tran Aug 25 '17 at 22:18