Is there a way to read data from CGImage without internal caching?

Question

I am fighting with an internal caching (about 90 MB for 15 mp image ) in CGContextDrawImage/CGDataProviderCopyData functions.
Here is the stack-trace in profiler:

In all cases, IOSurface is created as a "cache", and isn't cleaned after @autoreleasepool is drained.
This leaves a very few chances for an app to survive.
Caching doesn't depend on image size: I tried to render 512x512, as well as 4500x512 and 4500x2500 (full-size) image chunks.

I use @autoreleasepool, CFGetRetainCount returns 1 for all CG-objects before cleaning them.

The code which manipulates the data:

+ (void)render11:(CIImage*)ciImage fromRect:(CGRect)roi toBitmap:(unsigned char*)bitmap {
    @autoreleasepool
    {
        int w = CGRectGetWidth(roi), h = CGRectGetHeight(roi);

        CIContext* ciContext = [CIContext contextWithOptions:nil];
        CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

        CGContextRef cgContext = CGBitmapContextCreate(bitmap, w, h,
                                                   8, w*4, colorSpace,
                                                   kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);


        CGImageRef cgImage = [ciContext createCGImage:ciImage
                                         fromRect:roi
                                           format:kCIFormatRGBA8
                                       colorSpace:colorSpace
                                         deferred:YES];


        CGContextDrawImage(cgContext, CGRectMake(0, 0, w, h), cgImage);

        assert( CFGetRetainCount(cgImage) == 1 );

        CGColorSpaceRelease(colorSpace);
        CGContextRelease(cgContext);
        CGImageRelease(cgImage);
    }
}

What I know about IOSurface: it's from the previously private framework IOSurface.
CIContext has a function render: ... toIOSurface:.
I've created my IOSurfaceRef and passed it to this function, and the internal implementation still creates its own surface, and doesn't clean it.

So, do you know (or assume):
1. Are there other ways to read CGImage's data buffer except CGContextDrawImage/CGDataProviderCopyData ?
2. Is there a way to disable caching at render?
3. Why does the caching happen?
4. Can I use some lower-level (while non-private) API to manually clean up system memory?

Any suggestions are welcome.

Two thoughts: Have you tried calling `clearCaches` on the `ciContext` after you created the `cgImage`? Also you could try to init the `ciContext` with `contextWithCGContext:options:` after creating the `cgContext`, passing it as an argument. That should tell Core Image to render directly into that context (instead of an intermediate buffer) and you don't need to call `CGContextDrawImage`. I haven't tried it, though. — Frank Rupprecht, Nov 20 '18 at 19:25
@FrankSchlegel Thanks! 1) Yes, I tried, it does nothing. 2) No (only `[CIContext contextWithOptions:nil]`), I'll try. — olha, Nov 21 '18 at 13:37
@FrankSchlegel Well, based on your advice I tried `[CIContext contextWithOptions:]` and passed `kCIContextCacheIntermediates:NO`. The function `CreateCachedSurface ` is still called inside `[CIContext render: toBitmap:]`. — olha, Dec 03 '18 at 09:39
CoreImage caching happens because Lanczos requires intermediate because it is implemented as a vertical and horizontal passes. These surfaces will remain after the render is complete but they are marked as volatile so they will have no impact on jetsam pressure. Also note that [cictx clearCaches] will empty (but not delete) the surfaces associated with the cictx instance. — David Hayward, Apr 08 '21 at 01:55

score 6 · Accepted Answer · answered Dec 10 '18 at 03:11

6

To answer your second question,

Is there a way to disable caching at render?

setting the environment variable CI_SURFACE_CACHE_CAPACITY to 0 will more-or-less disable the CIContext surface cache. Moreover, you can specify a custom (approximate) cache limit by setting that variable to a given value in bytes. For example, setting CI_SURFACE_CACHE_CAPACITY to 2147483648 specifies a 2 GiB surface cache limit.

Note it appears that all of a process's CIContext instances share a single surface cache. It does not appear to be possible to use separate caches per CIContext.

answered Dec 10 '18 at 03:11

Michael Allman

116
5

Thanks, your advice is fantastic!!! I set CI_SURFACE_CACHE_CAPACITY=0, and that "more-or-less" is 200 KB cache instead of 90 MB. While I'm still testing for other photos. Do you know why isn't this variable documented? Google search returns 2 hits for "CI_SURFACE_CACHE_CAPACITY". – olha Dec 10 '18 at 20:29
1

Apple probably considers surface caching in CIContext an implementation detail subject to change at any time. They don't want to document something unless they are going to support it long term. That's my speculation. Don't be surprised if this fix stops working in a future version of macOS. Hopefully if it does stop working there will be another solution to this problem, or the problem will go away altogether. – Michael Allman Dec 11 '18 at 10:10

E.Coms · Answer 2 · 2018-11-10T22:35:49.777

1

If you just need to manipulate CIImage data, may consider to use CIImageProcessorKernel to put data into CPU or GPU calculation without extracting them.

I notice that

[ciContext render:image toBitmap:bitmap rowBytes: w*4 bounds:image.extent format:kCIFormatRGBA8 colorSpace:colorSpace];

There is no such 90M cache. Maybe it's what you want.

edited Nov 10 '18 at 22:35

answered Nov 09 '18 at 20:47

E.Coms

11,065
2
23
35

Thanks! I do need to read the data into `unsigned char*` buffer. Is it possible with `CIImageProcessorKernel ` ? – olha Nov 10 '18 at 06:41
It depends on knowledge level. Baseaddress has the data you need. If you cannot get it out, Simply way is to render to somewhere like iosurface or screen then capture screen. – E.Coms Nov 10 '18 at 13:07
cgimage can render directly to data. If i call createCGImage, it's 260M. call this it's only 170M during peak. – E.Coms Nov 10 '18 at 17:50
"Baseaddress has the data you need" - thanks, I'll try. "cgimage can render directly to data. If i call createCGImage" - yeah, but CGContextDrawImage does that unwanted caching inside – olha Nov 10 '18 at 21:41
sorry, not CGImage but CIImage. In my answer, I upgrade with CIcontext to render a CIImage to data without cgimage. – E.Coms Nov 10 '18 at 21:43
Sorry, `[ciContext render:image toBitmap:bitmap` - this was the intial code which does has a leak. All `CIContext render*` methods have it. – olha Nov 10 '18 at 21:47
Could you clarify: "Baseaddress has the data you need" - is it in `CIImageProcessorKernel` approach? – olha Nov 10 '18 at 21:47
1

"Baseaddress has the data you need." Yes, but you may not get it out of the block. – E.Coms Nov 10 '18 at 22:20
I did not see get leaked portion. [ciContext render:image toBitmap:bitmap]. – E.Coms Nov 10 '18 at 22:36

Is there a way to read data from CGImage without internal caching?

2 Answers2