3

I record video (.mp4 file) using AVAssetWriter with CMSampleBuffer data (from video, audio inputs).

While recording I want to process frames, I'm converting CMSampleBuffer to CIImage and processing it.

but how to update CMSampleBuffer with my new processed image buffer from CIImage?

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if output == videoOutput {
       let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
       let ciimage: CIImage = CIImage(cvPixelBuffer: imageBuffer)
       ... // my code to process CIImage (for example add augmented reality)
       // but how to convert it back to CMSampleBuffer?
       // because AVAssetWriterInput to encode video/audio in file needs CMSampleBuffer
       ...
    } 
    ...
}
Joe
  • 8,868
  • 8
  • 37
  • 59
user924
  • 8,146
  • 7
  • 57
  • 139

1 Answers1

3

You need to render your CIImage into a CVPixelBuffer, using CIContext's render(_:to:bounds:colorSpace:) method.

Then you can create a CMSampleBuffer from the CVPixelBuffer using e.g. CMSampleBufferCreateReadyWithImageBuffer(_:_:_:_:_:)

You may need to use a pool of CVPixelBuffer for efficiency reasons, an example of this is shown in Apple's AVCamPhotoFilter sample code. In particular, see the RosyCIRenderer class.

Also see this answer which may help you Applying a CIFilter to a Video File and Saving it

patroclus974
  • 364
  • 1
  • 3
  • yes I already do it and it's very slow (`render(_:to:bounds:colorSpace:)`) - https://stackoverflow.com/questions/49066195/what-is-the-best-way-to-record-a-video-with-augmented-reality – user924 Mar 02 '18 at 12:35
  • and I need to process frames while recording – user924 Mar 02 '18 at 12:40
  • Are you initialising the CIContext for GPU- based rendering with OpenGL or metal ? Ie init(eaglContext:) or init(mtlDevice:) – patroclus974 Mar 02 '18 at 12:50
  • I don't have CIContext here https://stackoverflow.com/questions/49066195/what-is-the-best-way-to-record-a-video-with-augmented-reality – user924 Mar 02 '18 at 12:58
  • `CIContext(options: nil)`, without options, anyway I don't want to load CPU or GPU, there is something else, it just does some unnecessary converting which eats resources, I don't think that for such task I need to use GPU, I did similar task using FFmpeg Recorder/OpenCV on Android (all processing on CPU), adding text to frames – user924 Mar 02 '18 at 13:03
  • Your question is "how to update CMSampleBuffer with my new processed image buffer from CIImage?". I believed this has been answered here. If you have a question about performance, as another question or amend this one? – patroclus974 Mar 03 '18 at 00:18
  • yes I found answer myself already (I already gave a link), this question can be closed - https://stackoverflow.com/questions/49066195/what-is-the-best-way-to-record-a-video-with-augmented-reality and performance sucks that's why I asked different question about best way to add text/ better performance - https://stackoverflow.com/questions/49066195/what-is-the-best-way-to-record-a-video-with-augmented-reality – user924 Mar 03 '18 at 06:43
  • @user924 I did not find any solution in your link – famfamfam Jun 08 '23 at 05:12