FFMPEG mosaicing stutters

Question

EDIT: Suddenly, with the same scripts on the same hardware, the problem is gone. I'm still curious about the cause, but until I can reproduce it again there are few options for diagnostics. My apologies.

I'm using FFMPEG to get live feeds from several webcams. When the feeds are independent (in other words, each v4l2 input running through its own pipeline inside the filter_complex to its own xv output), everything works very nicely: there's about a half-second delay, which I can live with, and all the videos run smoothly.

I would prefer to have these videos mosaiced together, both for watching live and for the copy it saves to disk. But when I try this, either with a series of overlays or with hstack+vstack, the video begins stuttering: one feed will run smoothly for a few seconds while the others are still, then another one will run for a few seconds while the first one freezes, and so on. The video is still coming in quasi-real-time, so it's not terrible for security purposes, but it makes following movement very difficult.

Is there anything I can do to fix this, or at least reduce the stuttering? I'd be fine with a lower framerate, as long as it was consistent.

Script which works:

ffmpeg \
    -f lavfi -thread_queue_size 1024 -i color=s=${size}:c=000000 \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_ne \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_sw \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_se \
    -f alsa -thread_queue_size 1024 -i $audio1 \
    -f alsa -thread_queue_size 1024 -i $audio2 \
    -filter_complex "
        [0:v] fifo, setpts=PTS-STARTPTS [v0];
        [1:v] fifo, setpts=PTS-STARTPTS [v1];
        [2:v] fifo, setpts=PTS-STARTPTS [v2];
        [3:v] fifo, setpts=PTS-STARTPTS [v3]" \
    -map "[v0]" -f xv - \
    -map "[v1]" -f xv - \
    -map "[v2]" -f xv - \
    -map "[v3]" -f xv - 

Script which doesn't:

ffmpeg \
    -f lavfi -thread_queue_size 1024 -i color=s=${size}:c=000000 \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_ne \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_sw \
    -f v4l2 -thread_queue_size 1024 -video_size $size -i $video_se \
    -f alsa -thread_queue_size 1024 -i $audio1 \
    -f alsa -thread_queue_size 1024 -i $audio2 \
    -filter_complex "
        [0:v] fifo, setpts=PTS-STARTPTS [v0];
        [1:v] fifo, setpts=PTS-STARTPTS [v1];
        [2:v] fifo, setpts=PTS-STARTPTS [v2];
        [3:v] fifo, setpts=PTS-STARTPTS [v3];
    [v0][v1] hstack [tmp1];
    [v2][v3] hstack [tmp2];
    [tmp1][tmp2] vstack [vout]"
-map "[vout]" -f sdl2 -

(I'm using only three videos right now, since I have four CPU cores.)