0

I am trying to use libavcodec and libavformat to write an mp4 video file in realtime using h264. I am using an approach heavily inspired by this answer here This works well as a non-realtime solution however, avcodec_receive_packet() starts running much slower after 20 frames or so (this is usually around the first time it returns success and thus av_interleaved_write_frame() is called for the first time). This is so slow that my writing cannot work in realtime.

Solutions I have tried:

  • Enabling multithreading on my codec context
  • Running avcodec_receive_packet() and av_interleaved_write_frame() on a separate thread to my capture from the realtime video source
  • Changing the gop_size in the video context
  • Lowering my bitrate in the video context

Is there anything I'm missing? Possibly some fundamental rules to capturing video in realtime. I am not very experienced with programming with video.

WalleyM
  • 172
  • 7
  • 1
    _"Is there anything I'm missing?"_ MP4 is not suitable for live recording (the H.264 video codec can do it but the MP4 container itself needs a complete video stream to exist for updating its own header details). If possible try to find out how to create a **fragmented MP4** (of type: ISO BMFF) in FFmpeg. A fragmented MP4 contains each frame (or group of frames) in a single playable MP4 chunk. I only use command line so cannot advise about code. – VC.One Oct 31 '22 at 14:37
  • 1
    I believe this isn't the problem I'm encountering since I think ffmpeg is dealing with this problem silently in the backend. When I finish my video writing I call [av_write_trailer](https://ffmpeg.org/doxygen/3.4/group__lavf__encoding.html#ga7f14007e7dc8f481f054b21614dfec13) which creates the final output file. Since this occurs after the writing has finished, a complete video stream is available. Using my current code an .mp4 file is written, the problem is writing is too slow so frames are dropped. – WalleyM Oct 31 '22 at 16:15
  • 1
    H.264 encoding can be very computationally expensive so it really shouldn't come as a surprise that it can't always be done in realtime. Try basic [encoding using `ffmpeg`](https://trac.ffmpeg.org/wiki/Encode/H.264) with various parameter settings (presets etc.) to get an idea of what your CPU can manage. – G.M. Oct 31 '22 at 16:31
  • 1
    So are you saying it's more realtime if you write, for example, a smaller resolution? if yes then check how you might **enable GPU acceleration** on FFmpeg. Also check on using/enabling **threads**. Here is [an article about tests for live recording with FFmpeg](https://streaminglearningcenter.com/blogs/ffmpeg-command-threads-how-it-affects-quality-and-performance.html). – VC.One Nov 01 '22 at 02:54

1 Answers1

0

I have solved this by not using h.264 encoding and instead using libavcodec's mpeg2video encoder. This leads to a much larger file size however the frame by frame encoding has a much more consistent processing time. Thanks to @G.M.'s comment for that. I have not yet tested any other encoders so possibly those could be helpful too.

Another possible solution was using GPU acceleration as mentioned by @VC.One, however for my use case, this was not feasible due to the target hardware for this to run not being known. A possible hybrid method is to determine the target hardware in the code and enable H.264 with GPU processing if a powerful enough GPU is available and to use mpeg2video encoding if not.

WalleyM
  • 172
  • 7