How to change offset of EIA-608 subtitles in an MP4?

Question

I have an MP4 file containing H.264 Video, AAC Audio, and EIA-608 Closed Captions:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.83.100
  Duration: 00:11:26.02, start: 0.000000, bitrate: 4952 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], Closed Captions, 4818 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : SoundHandler

However, the subtitles are displaying about 2 seconds too early.

The catch is, as you can see, ffmpeg doesn't recognize eia_608 as a "stream" that can be manipulated with the -map commands (as in this question). So inputting the file multiple times to remux the subtitles back in with a delay gives the following error:

Stream map '1:s' matches no streams.
To ignore this, add a trailing '?' to the map.

And obviously -itsoffset affects the video exactly as much as the subtitles in every case, anyway.

So how do I add a slight delay to eia_608 subtitles in an MP4 file without recoding?