2

Is there way to set up an observer / callback on an AVPlayer to get notified when the frame changes?

I am aware of both addBoundaryTimeObserver and addPeriodicTimeObserver however these are approximations that require me to estimate the frame rate, etc.

There is a note that:

General State Observations: You can use Key-value observing (KVO) to observe state changes to many of the player’s dynamic properties, such as its currentItem or its playback rate. You should register and unregister for KVO change notifications on the main thread. This avoids the possibility of receiving a partial notification if a change is being made on another thread. AV Foundation invokes observeValue(forKeyPath:of:change:context:) on the main thread, even if the change operation is made on another thread.

However currentTime on AVPlayerItem is a method, not a property so I cannot use KVO for that.

Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120

1 Answers1

1

You could add an AVPlayerItemVideoOutput to your AVPlayerItem and periodically poll the output with hasNewPixelBufferForItemTime which will tell you of the arrival of a new frame. However you then need to acquire the frame with copyPixelBufferForItemTime, so you should probably immediately release it. Here's an example of setting up AVPlayerItemVideoOutput. This is polling, so you could realise late or even miss a frame change.

You could also quickly preprocess the video file (if it is a file) without decompressing the frames, to determine the frame presentation time stamps. You could feed those timestamps one at a time to addBoundaryTimeObserver to decide when you'd crossed a frame boundary. Here's an example of parsing a video file.

AVSampleBufferDisplayLayer, which is a lower level AVPlayerLayer, that lets you feed it video frame CMSampleBuffers looks like a promising way to find out when a frame changes, but it doesn't seem to tell you when it has displayed one of the sample buffers you gave it. And I don't think AVSampleBufferDisplayLayer handles audio, either.

You could also reimplement the AVPlayer playback system - then you'd be painfully (and accurately) aware of frame changes (and audio changes, and opengl/metal). Surely that kind of effort is not required here. What kind of feature are are you trying to implement exactly?

Community
  • 1
  • 1
Rhythmic Fistman
  • 34,352
  • 5
  • 87
  • 159
  • Trying to paint an overlay op top of the screen corresponding to the given frame. Since the overlay corresponds to a given frame (as opposed to a given timestamp, like a scrubber), I want to update the overlay when the frame changes. – Alex Rothberg May 10 '17 at 22:10
  • This answer helped me find a solution. AVPlayerItemVideoOutput has a delegate that can get called when there's a new frame. Unfortunately you have request a callback each time, and the docs specifically mention don't call this request function repeatedly to get new frame callbacks for every frame. My use case was after seeking, do something once the frame is rendered, so this was fine. Just in the seek(to: ) callback, call this after setting a delegate on the output https://developer.apple.com/documentation/avfoundation/avplayeritemvideooutput/1386046-requestnotificationofmediadatach – ryan gordon Dec 03 '21 at 00:41