iOS - Creating multiple time-delayed live camera preview views

Question

I have done a ton of research, and haven't yet been able to find a viable solution, for many reasons, which I will outline below.

---

Problem

In my iOS app, I want three views that indefinitely show a delayed-live preview of the device's camera.

For example, view 1 will show a camera view, delayed for 5s, view 2 will show the same camera view, delayed for 20s, and view 3 will show the same camera view, delayed 30s.

This would be used to record yourself performing some kind of activity, such as a workout exercise, and then watch yourself a few seconds later in order to perfect your form of a given exercise.

Solutions Tried

I have tried and researched a couple different solutions, but all have problems.

1. Using AVFoundation and AVCaptureMovieFileOutput:

• Use AVCaptureSession and AVCaptureMovieFileOutput to record short clips to device storage. Short clips are required because you cannot play video from a URL, and write to that same URL simultaneously.
• Have 3 AVPlayer and AVPlayerLayer instances, all playing the short recorded clips at their desired time-delays.
• Problems:
  1. When switching clips using AVPlayer.replaceCurrentItem(_:), there is a very noticeable delay between clips. This needs to be a smooth transition.
  2. Although old, a comment here suggests not to create multiple AVPlayer instances due to a device limit. I haven't been able to find information confirming or denying this statement. E: From Jake G's comment - 10 AVPlayer instances is okay for an iPhone 5 and newer.

2. Using AVFoundation and AVCaptureVideoDataOutput:

• Use AVCaptureSession and AVCaptureVideoDataOutput to stream and process each frame of the camera's feed using the didOutputSampleBuffer delegate method.
• Draw each frame on an OpenGL view (such as GLKViewWithBounds). This solves the problem of multiple AVPlayer instances from Solution 1..
• Problem: Storing each frame so they can be displayed later requires copious amounts of memory (which just isn't viable on an iOS device), or disk space. If I want to store a 2 minute video at 30 frames per second, that's 3600 frames, totalling over 12GB if copied directly from didOutputSampleBuffer. Maybe there is a way to compress each frame x1000 without losing quality that would allow me to keep this data in memory. If such a method exists, I haven't been able to find it.

Possible 3rd Solution

If there is a way to read and write to a file simultaneously, I believe the following solution would be ideal.

• Record video as a circular stream. For example, for a video buffer of 2 minutes, I would create a file output stream that will write frames for two minutes. Once the 2 minute mark is hit, the stream will restart from the beginning, overriding the original frames.
• With this file output stream constantly running, I would have 3 input streams on the same recorded video file. Each stream would point to a different frame in the stream (effectively X seconds behind the writing stream). Then each frame would be displayed on the input streams respective UIView.

Of course, this still has an issue of storage space. Event if frames were stored as compressed JPEG images, we're talking about multiple GBs of storage required for a lower quality, 2 minute video.

Question

1. Does anyone know of an efficient method to achieve what I want?
2. How can I fix some of the problems in the solutions I've already tried?

Answer 1

Things have changed since the accepted answer. There is now an alternative to a segmented AVCaptureMovieFileOutput that doesn't drop frames on iOS when you create new segments and that alternative is AVAssetWriter!

As of iOS 14, AVAssetWriter can create fragmented MPEG4 which are essentially mpeg 4 files in memory. Intended for HLS streaming applications, it is however also an incredibly convenient method for caching video and audio content.

This new capability is described by Takayuki Mizuno in the WWDC 2020 session Author fragmented MPEG-4 content with AVAssetWriter.

With a fragmented mp4 AVAssetWriter in hand, it is not too hard to create a solution to this problem by writing the mp4 segments to disk and playing them back with the desired time offsets using several AVQueuePlayers.

So this would be a fourth solution: capture the camera stream and write it to disk as fragmented mp4 using AVAssetWriter's .mpeg4AppleHLS output profile and play the video back with differing delays using AVQueuePlayers and AVPlayerLayers.

If you need to support iOS 13 and below you'll have to replace the segmented AVAssetWriter, which gets technical quickly, especially if you want to write audio too. Thanks, Takayuki Mizuno!

import UIKit
import AVFoundation
import UniformTypeIdentifiers

class ViewController: UIViewController {
    let playbackDelays:[Int] = [5, 20, 30]
    let segmentDuration = CMTime(value: 2, timescale: 1)

    var assetWriter: AVAssetWriter!
    var videoInput: AVAssetWriterInput!
    var startTime: CMTime!

    var writerStarted = false
    
    let session = AVCaptureSession()
    
    var segment = 0
    var outputDir: URL!
    var initializationData = Data()
    
    var layers: [AVPlayerLayer] = []
    var players: [AVQueuePlayer] = []

    override func viewDidLoad() {
        super.viewDidLoad()
        
        for _ in 0..<playbackDelays.count {
            let player = AVQueuePlayer()
            player.automaticallyWaitsToMinimizeStalling = false
            let layer = AVPlayerLayer(player: player)
            layer.videoGravity = .resizeAspectFill
            layers.append(layer)
            players.append(player)
            view.layer.addSublayer(layer)
        }
        
        outputDir = FileManager.default.urls(for: .documentDirectory, in:.userDomainMask).first!
    
        assetWriter = AVAssetWriter(contentType: UTType.mpeg4Movie)
        assetWriter.outputFileTypeProfile = .mpeg4AppleHLS // fragmented mp4 output!
        assetWriter.preferredOutputSegmentInterval = segmentDuration
        assetWriter.initialSegmentStartTime = .zero
        assetWriter.delegate = self
        
        let videoOutputSettings: [String : Any] = [
            AVVideoCodecKey: AVVideoCodecType.h264,
            AVVideoWidthKey: 1024,
            AVVideoHeightKey: 720
        ]
        videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: videoOutputSettings)
        videoInput.expectsMediaDataInRealTime = true

        assetWriter.add(videoInput)

        // capture session
        let videoDevice = AVCaptureDevice.default(for: .video)!
        let videoInput = try! AVCaptureDeviceInput(device: videoDevice)
        session.addInput(videoInput)
        
        let videoOutput = AVCaptureVideoDataOutput()
        videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue.main)
        session.addOutput(videoOutput)
        
        session.startRunning()
    }
    
    override func viewDidLayoutSubviews() {
        let size = view.bounds.size
        let layerWidth = size.width / CGFloat(layers.count)
        for i in 0..<layers.count {
            let layer = layers[i]
            layer.frame = CGRect(x: CGFloat(i)*layerWidth, y: 0, width: layerWidth, height: size.height)
        }
    }
    
    override var supportedInterfaceOrientations: UIInterfaceOrientationMask {
        return .landscape
    }
}

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        
        if startTime == nil {
            let success = assetWriter.startWriting()
            assert(success)
            startTime = sampleBuffer.presentationTimeStamp
            assetWriter.startSession(atSourceTime: startTime)
        }
        
        if videoInput.isReadyForMoreMediaData {
            videoInput.append(sampleBuffer)
        }
    }
}

extension ViewController: AVAssetWriterDelegate {
    func assetWriter(_ writer: AVAssetWriter, didOutputSegmentData segmentData: Data, segmentType: AVAssetSegmentType) {
        print("segmentType: \(segmentType.rawValue) - size: \(segmentData.count)")
        
        switch segmentType {
        case .initialization:
            initializationData = segmentData
        case .separable:
            let fileURL = outputDir.appendingPathComponent(String(format: "%.4i.mp4", segment))
            segment += 1

            let mp4Data = initializationData + segmentData
            try! mp4Data.write(to: fileURL)

            let asset = AVAsset(url: fileURL)

            for i in 0..<players.count {
                let player = players[i]
                let playerItem = AVPlayerItem(asset: asset)
                player.insert(playerItem, after: nil)
                
                if player.rate == 0 && player.status == .readyToPlay {
                    let hostStartTime: CMTime = startTime + CMTime(value: CMTimeValue(playbackDelays[i]), timescale: 1)

                    player.preroll(atRate: 1) { prerolled in
                        guard prerolled else { return }
                        player.setRate(1, time: .invalid, atHostTime: hostStartTime)
                    }
                }
            }
            
        @unknown default:
            break
        }
    }
}

The result looks like this

and the performance is reasonable: my 2019 iPod is sitting in 10-14% cpu and 38MB of memory.

Answer 2

1. on iOS AVCaptureMovieFileOutput drops frames when switching files. On osx this doesn't happen. There's a discussion around this in the header file, see captureOutputShouldProvideSampleAccurateRecordingStart.

A combination of your 2. and 3. should work. You need to write the video file in chunks using AVCaptureVideoDataOutput and AVAssetWriter instead of AVCaptureMovieFileOutput so you don't drop frames. Add 3 ring buffers with enough storage to keep up with playback, use GLES or metal to display your buffers (use YUV instead of RGBA use 4/1.5 times less memory).

I tried a more modest version of this back in the days of the mighty iPhone 4s and iPad 2. It showed (I think) now and 10s in the past. I guestimated that because you could encode 30fps at 3x realtime, that I should be able to encode the chunks and read the previous ones using only 2/3 of the hardware capacity. Sadly, either my idea was wrong or there was a non-linearity with the hardware, or the code was wrong and the encoder kept falling behind.