4

Issue

I'm using AVFoundation to implement a Camera that is able to record videos while running special AI processing.

Having an AVCaptureMovieFileOutput (for video recording) and a AVCaptureVideoDataOutput (for processing AI) running at the same time is not supported (see Can use AVCaptureVideoDataOutput and AVCaptureMovieFileOutput at the same time?), so I have decided to use a single AVCaptureVideoDataOutput which is able to record videos to a file while running the AI processing in the same captureOutput(...) callback.

To my surprise, doing that drastically increases RAM usage from 58 MB to 187 MB (!!!), and CPU from 3-5% to 7-12% while idle. While actually recording, the RAM goes up even more (260 MB!).

I am wondering what I did wrong here, since I disabled all the AI processing and just compared the differences between AVCaptureMovieFileOutput and AVCaptureVideoDataOutput.

My code:

AVCaptureMovieFileOutput

Setup

    if let movieOutput = self.movieOutput {
      captureSession.removeOutput(movieOutput)
    }
    movieOutput = AVCaptureMovieFileOutput()
    captureSession.addOutput(movieOutput!)

Delegate

(well there is none, AVCaptureMovieFileOutput handles all that internally)

Benchmark

When idle, so not recording at all:

  • RAM: 56 MB
  • CPU: 3-5%

When recording using AVCaptureMovieFileOutput.startRecording:

  • RAM: 56 MB (how???)
  • CPU: 20-30%

AVCaptureVideoDataOutput

Setup

    // Video
    if let videoOutput = self.videoOutput {
      captureSession.removeOutput(videoOutput)
      self.videoOutput = nil
    }
    videoOutput = AVCaptureVideoDataOutput()
    videoOutput!.setSampleBufferDelegate(self, queue: videoQueue)
    videoOutput!.alwaysDiscardsLateVideoFrames = true
    captureSession.addOutput(videoOutput!)

    // Audio
    if let audioOutput = self.audioOutput {
      captureSession.removeOutput(audioOutput)
      self.audioOutput = nil
    }
    audioOutput = AVCaptureAudioDataOutput()
    audioOutput!.setSampleBufferDelegate(self, queue: audioQueue)
    captureSession.addOutput(audioOutput!)

Delegate

extension CameraView: AVCaptureVideoDataOutputSampleBufferDelegate,
                      AVCaptureAudioDataOutputSampleBufferDelegate {

  public final func captureOutput(_ captureOutput: AVCaptureOutput,
                                  didOutput sampleBuffer: CMSampleBuffer,
                                  from _: AVCaptureConnection) {
    // empty
  }

  public final func captureOutput(_ captureOutput: AVCaptureOutput,
                                  didDrop buffer: CMSampleBuffer,
                                  from _: AVCaptureConnection) {
    // empty
  }
}

yes, they are literally empty methods. My RAM and CPU usage is still that high without doing any work here.

Benchmark

When idle, so not recording at all:

  • RAM: 151-187 MB
  • CPU: 7-12%

When recording using a custom AVAssetWriter:

  • RAM: 260 MB
  • CPU: 64%

Why is the AVCaptureMovieFileOutput so much more efficient than an empty AVCaptureVideoDataOutput? Also, why does it's RAM not go up at all when recording, compared to how my AVAssetWriter implementation alone consumes 80 MB?

Here's my custom AVAssetWriter implementation: RecordingSession.swift, and here's where I call it.

Any help appreciated!

mrousavy
  • 857
  • 8
  • 25
  • Maybe the `AVCaptureMovieFileOutput` resource usage is not happening in your process. Maybe it's happening in `mediaserverd`? – Rhythmic Fistman May 03 '21 at 15:52
  • @RhythmicFistman how can I check that? If it's a special process for that, I assume it's recycling that allocated RAM, no? Also, how would that explain the difference in CPU usage for my process? – mrousavy May 03 '21 at 15:59

1 Answers1

0

Based on this answer, I tried messing around with the sessionPreset in the AVCaptureSession object. The default is set to .high, so I tried changing it to .medium. While the RAM consumption and CPU usage are reduced quite significantly, the video quality is also greatly reduced as well. I ended up using the .hd1280x720 preset. The RAM consumption and CPU usage are not as high as using the default preset, and the quality is still pretty good.

let captureSession = AVCaptureSession()
captureSession.sessionPreset = .hd1280x720

Hope this helps

cleanrun
  • 549
  • 6
  • 20
  • Thanks but this does not answer the question. I don't get how the video data output uses **more** memory than the movie output. – mrousavy Jun 19 '23 at 13:40