Correctly Allocate And Fill Frame In FFmpeg

Question

I am filling a Frame with a BGR image for encoding, and I am getting a memory leak. I think I got to the source of the problem but it appears to be a library issue instead. Since FFmpeg is such a mature library, I think I am misusing it and I would like to be instructed on how to do it correctly.

I am allocating a Frame using:

AVFrame *bgrFrame = av_frame_alloc();

And later I allocate the image in the Frame using:

av_image_alloc(bgrFrame->data, bgrFrame->linesize, bgrFrame->width, bgrFrame->height, AV_PIX_FMT_BGR24, 32);

Then I fill the image allocated using:

av_image_fill_pointers(bgrFrame->data, AV_PIX_FMT_BGR24, bgrFrame->height, originalBGRImage.data, bgrFrame->linesize);

Where originalBGRImage is an OpenCV Mat.

And this has a memory leak, apparently, av_image_alloc() allocates memory, and av_image_fill_pointers() also allocates memory, on the same pointers (I can see bgrFrame->data[0] changing between calls).

If I call

av_freep(&bgrFrame->data[0]);

After av_image_alloc(), it's fine, but if I call it after av_image_fill_pointers(), the program crashes, even though bgrFrame->data[0] is not NULL, which I find very curious.

Looking FFmpeg's av_image_alloc() source code, I see it calls av_image_fill_pointers() twice inside it, once allocating a buffer buff....and later in av_image_fill_pointers() source code, data[0] is substituted by the image pointer, which is (I think) the source of the memory leak, since data[0] was holding buf from the previous av_image_alloc() call.

So this brings the final question: What's the correct way of filling a frame with an image?.

Since the data is already there, you could use `avpicture_fill`, which does not allocate. In any case, you should only allocate one frame and reuse that. Also, have a look if you really need `av_freep` or `av_free`. — SpamBot, Feb 23 '16 at 12:03
But as far as I know, I need to give a frame for encoding, how I convert the AVPicture into a AVFrame? — Michel Feinstein, Feb 23 '16 at 15:13

score 4 · Answer 1 · edited May 23 '17 at 11:50

You should allocate your frame once.

AVFrame* alloc_picture(enum PixelFormat pix_fmt, int width, int height)
{
AVFrame *f = avcodec_alloc_frame();
if (!f)
    return NULL;
int size = avpicture_get_size(pix_fmt, width, height);
uint8_t *buffer = (uint8_t *) av_malloc(size);
if (!buffer) {
    av_free(f);
    return NULL;
}
avpicture_fill((AVPicture *)f, buffer, pix_fmt, width, height);
return f;
}

Yes, the cast (AVPicture*) is allowed https://stackoverflow.com/a/20498359/2079934 . In subsequent frames, you can write into the this frame. Since your OpenCV raw data is BGR and you need RGB or YUV, you can use sws_scale. In my application, I mirror vertically:

struct SwsContext* convertCtx = sws_getContext(width, height, PIX_FMT_RGB24, c->width, c->height, c->pix_fmt, SWS_FAST_BILINEAR, NULL, NULL, NULL);
avpicture_fill(&pic_raw, (uint8_t*)pixelBuffer, PIX_FMT_RGB24, width, height);
// flip
pic_raw.data[0] += (height - 1) * pic_raw.linesize[0]; 
pic_raw.linesize[0] *= -1;
sws_scale(convertCtx, pic_raw.data, pic_raw.linesize, 0, height, f->data, f->linesize);
out_size = avcodec_encode_video(c, outputBuffer, outputBufferSize, f);

(You can adapt PIX_FMT_RGB24 to your needs and read from cv::Mat without copying data.)

I got it to work using `av_fill_arrays()` and I pass my BGR image pointer and allocated frame (which is only being allocated once). I am also using `sws_scale()` for conversion, the only piece missing was the `av_fill_arrays()` actually. — Michel Feinstein, Feb 24 '16 at 16:03

score 0 · Answer 2 · answered Feb 24 '16 at 16:05

0

av_fill_arrays() does the job. It will fill the frame's data[] and linesizes[] but not reallocating any memory.

answered Feb 24 '16 at 16:05

Michel Feinstein

13,416
16
91
173

I was looking "for av_fill_arrays" but couldn't find it. I couldn't find the source code or the docu. Is it depreciated? – Darkwonder Nov 16 '19 at 13:43
I don't know, last time I used it was in 2016, but I can find `av_image_fill_arrays` on Google, maybe it's the same? – Michel Feinstein Nov 16 '19 at 22:01

Trương Quốc Khánh · Answer 3 · 2022-04-14T11:42:08.510

Too late for answer. But after take many hours, i want to share.
In document

 /**
    * AVBuffer references backing the data for this frame. All the pointers in
    * data and extended_data must point inside one of the buffers in buf or
    * extended_buf. This array must be filled contiguously -- if buf[i] is
    * non-NULL then buf[j] must also be non-NULL for all j < i.
    *
    * There may be at most one AVBuffer per data plane, so for video this array
    * always contains all the references. For planar audio with more than
    * AV_NUM_DATA_POINTERS channels, there may be more buffers than can fit in
    * this array. Then the extra AVBufferRef pointers are stored in the
    * extended_buf array.
    */
AVBufferRef *buf[AV_NUM_DATA_POINTERS];

Then buf is "smart pointer" for data (extended_buf for extended_data)
for example, i using image one linesize only

int size = av_image_get_buffer_size(AVPixelFormat::AV_PIX_FMT_BGRA, width, height, 1);
AVBufferRef* dataref = av_buffer_alloc(size);//that for av_frame_unref

memcpy(dataref->data, your_buffer, size );

AVFrame* frame = av_frame_alloc();
av_image_fill_arrays(frame->data, frame->linesize, dataref->data, AVPixelFormat::AV_PIX_FMT_BGRA, source->width, source->height, 1
frame->buf[0] = dataref;

av_frame_unref will unref frame->buf and free pointer if ref count to zero

Correctly Allocate And Fill Frame In FFmpeg

3 Answers3

Linked