For Exoplayer's AdaptiveTrackSelection, should I switch to a single track with multiple bitrates instead of four tracks with separate bitrates?

Question

Currently, I have a server that streams four RTMP MediaSources, one with 720p video source, one with 360p video source, one with 180p video source, and one audio-only source. If I wanted to switch resolutions, I have to stop the ExoPlayer instance, prepare the other track I wanted to switch to, then play.

The code I use to prepare the ExoPlayer instance:

    TrackSelection.Factory adaptiveTrackSelectionFactory = new AdaptiveTrackSelection.Factory(bandwidthMeter);
    TrackSelector trackSelector = new DefaultTrackSelector(adaptiveTrackSelectionFactory);

    RtmpDataSourceFactory rtmpDataSourceFactory = new RtmpDataSourceFactory(bandwidthMeter);
    ExtractorsFactory extractorsFactory = new DefaultExtractorsFactory();
    factory = new AVControlExtractorMediaSource.Factory(rtmpDataSourceFactory);
    factory.setExtractorsFactory(extractorsFactory);

    createSource();

    //noinspection deprecation
    mPlayer = ExoPlayerFactory.newSimpleInstance(mActivity, trackSelector, new DefaultLoadControl(
            new DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE),
            1000,  // min buffer
            2000, // max buffer
            1000, // playback
            1000,   //playback after rebuffer
            DefaultLoadControl.DEFAULT_TARGET_BUFFER_BYTES,
            true
    ));

    vwExoPlayer.setPlayer(mPlayer);

    mPlayer.addAnalyticsListener(mAnalyticsListener);

With createSource() being:

private void createSource() {
    factory.setTrackPlaybackFlag(AVControlExtractorMediaSource.PLAYBACK_BOTH_AV);
    mMediaSource180 = factory.createMediaSource(Uri.parse(API.GAME_VIDEO_STREAM_URL_180()));
    mMediaSource180.addEventListener(getHandler(), new MSourceDebuggerListener("GameMediaSource180"));

    mMediaSource360 = factory.createMediaSource(Uri.parse(API.GAME_VIDEO_STREAM_URL_360()));
    mMediaSource360.addEventListener(getHandler(), new MSourceDebuggerListener("GameMediaSource360"));

    mMediaSource720 = factory.createMediaSource(Uri.parse(API.GAME_VIDEO_STREAM_URL_720()));
    mMediaSource720.addEventListener(getHandler(), new MSourceDebuggerListener("GameMediaSource720"));

    factory.setTrackPlaybackFlag(AVControlExtractorMediaSource.PLAYBACK_AUDIO_ONLY);
    mMediaSourceAudio = factory.createMediaSource(Uri.parse(API.GAME_AUDIO_STREAM_URL()));
    mMediaSourceAudio.addEventListener(getHandler(), new MSourceDebuggerListener("GameMediaSourceAudio"));
}

private void releaseSource() {
    mMediaSource180.releaseSource(null);
    mMediaSource360.releaseSource(null);
    mMediaSource720.releaseSource(null);
    mMediaSourceAudio.releaseSource(null);
}

And the code I currently use to switch between these MediaSources is:

private void changeTrack(MediaSource source) {
    if (currentMediaSource == source) return;

    try {
        this.currentMediaSource = source;
        mPlayer.stop(true);
        mPlayer.prepare(source, true, true);
        mPlayer.setPlayWhenReady(true);

        if (source == mMediaSourceAudio) {
            if (!audioOnly) {
                try {
                    TransitionManager.beginDelayedTransition(rootView);
                } catch (Exception ignored) {

                }

                layAudioOnly.setVisibility(View.VISIBLE);
                vwExoPlayer.setVisibility(View.INVISIBLE);
                audioOnly = true;

                try {
                    GameQnAFragment fragment = findFragment(GameQnAFragment.class);
                    if (fragment != null) {
                        fragment.signAudioOnly();
                    }
                } catch (Exception e) {
                    Trace.e(e);
                }

                try {
                    GamePollingFragment fragment = findFragment(GamePollingFragment.class);
                    if (fragment != null) {
                        fragment.signAudioOnly();
                    }
                } catch (Exception e) {
                    Trace.e(e);
                }
            }
        } else {
            if (audioOnly) {
                TransitionManager.beginDelayedTransition(rootView);
                layAudioOnly.setVisibility(View.GONE);
                vwExoPlayer.setVisibility(View.VISIBLE);
                audioOnly = false;
            }
        }
    } catch (Exception ignore) {

    }
}

I wanted to implement a seamless switching between these MediaSources so that I don't need to stop and re-prepare, but it appears that this feature is not supported by ExoPlayer.

In addition, logging each MediaSource structure with the following code:

MappingTrackSelector.MappedTrackInfo info = ((DefaultTrackSelector)trackSelector).getCurrentMappedTrackInfo();
    if(info != null) {
        for (int i = 0; i < info.getRendererCount(); i++) {
            TrackGroupArray trackGroups = info.getTrackGroups(i);
            if (trackGroups.length != 0) {
                for(int j = 0; j < trackGroups.length; j++) {
                    TrackGroup tg = trackGroups.get(j);
                    for(int k = 0; k < tg.length; k++) {
                        Log.i("track_info_"+i+"-"+j+"-"+k, tg.getFormat(k)+"");
                    }
                }
            }
        }
    }

Just nets me 1 video format and 1 audio format each.

My current workaround is to prepare another ExoPlayer instance in the background, replace the currently running instance with that upon preparations being complete, and release the old instance. That reduces the lag between the MediaSources somewhat, but doesn't come close to achieving seamless resolution changes like Youtube.

Should I implement my own TrackSelector and jam-pack all the 4 sources into that, should I implement another MediaSource that handles all 4 sources, or should I just tell the colleague who maintains the streams to switch to just one RTMP MediaSource with a sort of manifest that lists all the resolutions available for the AdaptiveTrackSelection to switch between them?

Answer 1

Adaptive Bit Rate Streaming is designed to allow easy switching between different bit rate streams, but it requires the streams to be segmented and the player to download the video segment by segment.

In this way the player can decide which bit rate to choose for the next segment depending on the current network conditions (and the device display size and t type). The player is able to seamlessly, apart from the different bitrate and quality, move from one bit rate to another this way.

See here for some more info: https://stackoverflow.com/a/42365034/334402

All the above relies on a delivery protocol which supports this segmentation and different bit rate streams. The most common ones today are HLS and MPEG-DASH.

The easiest way to support what I think you are looking for would be for you colleague who is supplying the stream to supply it using HLS and/or DASH.

Note that at the moment, both HLS and DASH are required as apple devices require HLS while other devices tend to default to DASH. Traditionally HLS used TS as the container for the video in the segments and DASH used fragmented MP4, but there is now a move for both to use CMAF, which is essentially fragmented MP4.

So in theory a single set of bit rate videos can be used for HLS and DASH now - in practice this will depend on whether your content is encrypted or not, as HLS and apple used one encryption mode and everyone else another in the past. This is changing now also but will take time before all devices support the new approach, where all devices can support the same encryption mode, so if your streams are encrypted this is an added complication at the moment.