Queue of AFHTTPRequestOperations creating Memory Buildup

Question

I just updated to AFNetworking 2.0 and I am re-writing my code to download data & insert it into Core Data.

I download JSON data files (anywhere from 10-200mb files), write them to disk, then pass them off to background threads to process the data. Below is the code that downloads the JSON & write it to disk. If I just let this run (without even processing the data), the app uses up memory until it is killed.

I assume as the data is coming in, it is being stored in memory, but once I save to disk why would it stay in memory? Shouldn't the autorelease pool take care of this? I also set the responseData, and downloadData to nil. Is there something blatantly obvious that I am doing wrong here?

@autoreleasepool
{
    for(int i = 1; i <= totalPages; i++)
    {
        NSString *path = ....
        NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:path]];
        AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
        op.responseSerializer =[AFJSONResponseSerializer serializer];

        [op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) 
        {
            //convert dictionary to data 
            NSData *downloadData = [NSKeyedArchiver archivedDataWithRootObject:responseObject];

            //save to disk
            NSError *saveError = nil;
            if (![fileManager fileExistsAtPath:targetPath isDirectory:false])
            {
                [downloadData writeToFile:targetPath options:NSDataWritingAtomic error:&saveError];
                if (saveError != nil) 
                {
                    NSLog(@"Download save failed! Error: %@", [saveError description]);
                }
            }

            responseObject = nil;
            downloadData = nil;

        } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
            DLog(@"Error: %@", error);
        }];
    }
    [mutableOperations addObject:op];
}

NSArray *operations = [AFURLConnectionOperation batchOfRequestOperations:mutableOperations progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
    DLog(@"%lu of %lu complete", (unsigned long)numberOfFinishedOperations, (unsigned long)totalNumberOfOperations);
} completionBlock:^(NSArray *operations) {
    DLog(@"All operations in batch complete");
}];

mutableOperations = nil;
[manager.operationQueue addOperations:operations waitUntilFinished:NO];

Thanks!

EDIT #1 Adding an @autoreleasepool within my complete block seemed to slow the memory usage a bit, but it still builds up and eventually crashes the app.

Answer 1

If your JSON files are really 10-200mb each, this would definitely cause memory problems, because this sort of request is going to load the responses in memory (rather than streaming them to persistent storage). Worse, because your using JSON, I think the problem is twice as bad, because you're going to be loading this into a dictionary/array, which also takes up memory. So, if you have four 100mb downloads going on, your peak memory usage could be of the order of magnitude of 800mb (100mb for the NSData plus ~100mb for the array/dictionary (possibly much larger), times four for the four concurrent requests). You could quickly run out of memory.

So, a couple of reactions:

1. When dealing with this volume of data, you'd want to pursue a streaming interface (a NSURLConnection or NSURLSessionDataTask where you write the data as it comes in, rather than holding it in memory; or use NSURLSessionDownloadTask which does this for you), one that writes the data directly to persistent storage (rather than trying to hold it in a NSData in RAM as it's being downloaded).

   If you use NSURLSessionDownloadTask, this is really simple. If you need to support iOS versions prior to 7.0, I'm not sure if AFNetworking supports streaming of the responses directly to persistent storage. I'd wager you could write your own response serializer that does that, but I haven't tried it. I've always written my own NSURLConnectionDataDelegate methods that download directly to persistent storage (e.g. something like this).

2. You might not want to use JSON for this (because NSJSONSerialization will load the whole resource into memory, and then parse it to a NSArray/NSDictionary, also in memory), but rather use a format that lends itself to streamed parsing of the response (e.g. XML) and write a parser that stores the data to your data store (Core Data or SQLite) as it's being parsed, rather than trying to load the whole thing in RAM.

   Note, even NSXMLParser is surprisingly memory inefficient (see this question). In the XMLPerformance sample, Apple demonstrates how you can use the more cumbersome LibXML2 to minimize the memory footprint of your XML parser.

3. By the way, I don't know if your JSON includes any binary data that you have encoded (e.g. base 64 or the like), but if so, you might want to consider a binary transfer format that doesn't have to do this conversion. Using base-64 or uuencode or whatever can increase your bandwidth and memory requirements. (If you're not dealing with binary data that has been encoded, then ignore this point.)

4. As an aside, you might want to use Reachability to confirm the user's connection type (Wifi vs cellular), because it is considered bad form to download that much data over cellular (at least not without the user's permission), not only because of speed issues, but also the risk of using up an excessive portion of their carrier's monthly data plan. I've even heard that Apple historically rejected apps that tried to download too much data over cellular.