Let's say I start an FFmpeg job that takes hours and hours:
$ ffmpeg -i large.mp4 out.mp4
Is there any way I can preview the finished part of out.mp4 while the rest is yet to be encoded?
Trying this complains "moov atom not found". If I am to speculate why, it would be because playback requires updated and correct header information to be present at the beginning of the file, and FFmpeg doesn't bother to do this while encoding files; it waits until it's completely finished.
Simply put, no. MP4 files cannot be viewed unless they're done encoding. As you correctly identified, in order to start playback, a player needs to parse the moov atom, which contains important info about what's to be played. For a description, you can have a look at the QuickTime File Format specification, which became the basis for MPEG-4 Part 14, or short, the MP4 format.
This atom can only be written once the media tracks are completely encoded. It's not that ffmpeg “doesn't bother" to write it—you just have to wait until all data is there to have all the information that is required.
The atom then gets written at the end of the container. For faster playback, you can move this atom to the beginning of the file (e.g., using qt-faststart or with the ffmpeg option -movflags +faststart). This, of course, can also only be done once encoding has finished.
That said, fragmented MP4s can be previewed up until a certain point, as Gyan suggests, by adding the following option:
-movflags +frag_keyframe+separate_moof+omit_tfhd_offset+empty_moov
What you could also do is switch over to a segmented encoding method, e.g. using the segment muxer. These create partial clips up to a predefined length, which means that you can start previewing the existing segments. You could then later join the segments back into one file with stream-copying.
