This one was absolutely killing me until I figured it out, picking up clues from this thread and a few others.  The biggest single problem in my code was that I was instantiating my video player controller every time I wanted to play a video.  Now, it gets instantiated once in the primary controller (in this case, my DetailViewContoller):
@interface DetailViewController () {
    VideoPlayerViewController *videoPlayerViewController;
}
- (void) viewDidLoad
{
    [super viewDidLoad];
    videoPlayerViewController = [[VideoPlayerViewController alloc] initWithNibName: nil bundle: nil];
}
When I want to show a video, I call my DetailViewController's startVideoPlayback method:
- (void) startVideoPlayback: (NSString *)videoUID
{
    videoPlayerViewController.videoUID = videoUID;
    [self presentModalViewController: videoPlayerViewController animated: YES];
}
(NOTE: I'm passing it 'videoUID' -- a unique identified that was used to create the video in another part of the app.)
In the VideoPlayerViewController (which is largely cribbed from Apple's AVPlayerDemo sample), the one-time screen setup (initializing the AVPlayer, setting up the toolbar, etc.) is done in viewDidLoad -- which now only get's called once, and all per-video setup gets done within viewWillAppear, which then calls prepareToPlay:
- (void) prepareToPlay
{
    [self initScrubberTimer];   
    [self syncPlayPauseButtons];
    [self syncScrubber];
    NSString *documentsDirectory = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
    //*** Retrieve and play video at associated with this videoUID
    NSString *destinationPath = [documentsDirectory stringByAppendingFormat: @"/%@.mov", videoUID];
    if ([self fileExists: destinationPath]) {
        //*** Show the activity indicator spinny thing
        [pleaseWait startAnimating];
        [self setURL: [NSURL fileURLWithPath: destinationPath]];
        //*** Get things going with the first video in this session
        if (isFirst) {
            isFirst = NO;
        //*** Subseqeunt videos replace the first one
        } else {
            [self.mPlayer replaceCurrentItemWithPlayerItem: [AVPlayerItem playerItemWithURL: [NSURL fileURLWithPath: destinationPath]]];
        }
    }
}