Enabling OMX support in GStreamer for Android

Question

I'm currently trying to hook onto an HW H.264 decoder on Oculus Quest 2 (Adreno 650).

The platform supports just one HW decoder OMX.qcom.video.decoder.avc

The video decoding pipeline is created using GStreamer and currently looking like this: rtspsrc location=rtsp://192.168.1.239:8554/left ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw,width=1920,height=1080,format=RGBA ! appsink emit-signals=true name=wallesink sync=false

The decodebin bin should automatically use the available OMX decoder, but GStreamer library isn't built with it being supported.

I'm currently using this release of GStreamer.

So my question is simple: What is the easiest way to add support for said OMX decoder?

I've tried building from source with Meson, but was unable to do it with OMX enabled.

Are there any other builds for Android that have this decoder bundled?

Mikael Lepistö · Accepted Answer · 2022-11-17T14:25:06.187

Those codecs are prebuilt in the android binaries, distributed from gstreamer home page https://gstreamer.freedesktop.org/data/pkg/android/

OMX decoder is available at least in gstreamer 1.18.6 and when using decodebin3 node it knew how to choose the HW decoder on quest1 and quest2.

However with 1.20.4 automatic selecting didn't work out-of-box and one has to set higher ranking for omx decoders to make decodebin to select them.

https://gstreamer.freedesktop.org/documentation/tutorials/playback/hardware-accelerated-video-decoding.html?gi-language=c

Here I did find one C example how to do it from github: https://github.com/github0925/test/blob/0a919ce6f278a8fc5c7d203aaa220b3e703bb6b1/source/vendor/vpu/gstreamer_middleware/player/gstreamer_middleware_decode.c#L7

int change_omx_decode_element_rank()
{
    GstRegistry *registry = gst_registry_get();

    if (!registry) {
        ERROR("Failed to get gstreamer registry");
        return -1;
    }

    char *omx_decode_element_list[] = {        
        "amcviddec-omxqcomvideodecoderh263",
        "amcviddec-omxqcomvideodecoderavc",
        "amcviddec-omxqcomvideodecoderhevc",
        "amcviddec-omxqcomvideodecodermpeg2",
        "amcviddec-omxqcomvideodecodermpeg4",
        "amcviddec-omxqcomvideodecodervp8",
        "amcviddec-omxqcomvideodecodervp9",
        "amcvidenc-omxqcomvideoencoderavc",
        "amcvidenc-omxqcomvideoencoderh263",
        "amcvidenc-omxqcomvideoencoderhevc",
        "amcvidenc-omxqcomvideoencodermpeg4",
        "amcvidenc-omxqcomvideoencodervp8"
    };

    for (int i = 0; i < ARRAYSIZE(omx_decode_element_list); i++) {
        GstPluginFeature *feature =
            gst_registry_lookup_feature(registry, omx_decode_element_list[i]);
        if (!feature) {
            WARN("Featuer does not exist: %s", omx_decode_element_list[i]);
            continue;
        }

        gst_plugin_feature_set_rank(feature, GST_RANK_PRIMARY + 1);
        gst_registry_add_feature(registry, feature);
        gst_object_unref(feature);
    }

    return 0;
}

Also I noticed that with 1.20.4 incoming HEVC stream should be generated with main-10 profile, since for some reason, main profile is not listed in amcviddec-omxqcomvideodecoderhevc capabilities anymore after gstreamer 1.18.6

Thanks for the valuable insight! I wasn't aware of the `registry` in GStreamer so thanks to this I was able to list all available features my GStreamer shared library has been bundled with. The decoder that the `decodebin` element has been using is `avdec_h264` which I thought wasn't right as Quest 2 only lists OMX support.. After some more debugging I've found out that the actual bottleneck of my pipeline wasn't the decoding part, but the subsequent transform made with `videoconvert` from YUV to RGB.. — Stanislav Svědiroh, Feb 27 '23 at 19:24
Couldn't figure out how to speed that up so I've just passed the intermediate format into the fragment shader directly and made my own conversion into RGB. — Stanislav Svědiroh, Feb 27 '23 at 19:27
@StanislavSvědiroh fast scaling + colorconvert can be done like this: `glupload ! glcolorconvert ! glcolorscale ! video/x-raw(memory:GLMemory),texture-target=2D,format=(string)RGBA,width=(int)5760,height=(int)3240` — Mikael Lepistö, Mar 08 '23 at 12:05

Enabling OMX support in GStreamer for Android

1 Answers1