Does the AC3 audio format support variable bitrate encoding?
If so, is it possible to use FFmpeg's -vbr option to achieve that?
I'd like to avoid converting from 5.1 to a 2-channel stereo because I've read that FFmpeg doesn't produce good results when mixing the channels.
Oh, that awful, misleading thread. The ATSC A/52A (AC-3) standard (refer to sections 6.1.12 & 7.8, and also this post by pandy) provides detailed info about how proper downmixing should occur for AC-3.
Note that the Low Frequency Effects (LFE) channel may be omitted when downmixing, and I believe that's what ffmpeg does.
If you want to change the downmix method then try the suggestion by pandy in the same thread:
-af aformat=sample_fmts=fltp:channel_layouts=downmix
I did not test this, so I'm not sure it will do what you want.
Also, investigate the -dmix_mode private option. See ffmpeg -h encoder=ac3.
The -vbr option is used as a "private" encoding option for some encoders, not as a global option:
Private options are specific to the given container, device or codec. These options are listed under their corresponding containers/devices/codecs.
...meaning the use of this option and what it does may vary depending on the encoder; if the encoder even supports this option (refer to ffmpeg -h encoder=<encoder name>). Other audio encoders may use -qscale:a (or -q:a) for VBR.
The encoder ac3 ignores both -vbr and -qscale:a/-q:a.
