OutOfMemoryException in C#

Question

This code causes some kind of memory leak. I assume it's caused by the new byte[]. But shouldn't the GC avoiding this? If the program runs long enough, the code will cause a OutOfMemoryException

using (var file = new FileStream(fileLoc, FileMode.Open))
{
    int chunkSize = 1024 * 100;
    while (file.Position < file.Length)
    {
        if (file.Length - file.Position < chunkSize)
        {
            chunkSize = (int)(file.Length - file.Position);
        }
        byte[] chunk = new byte[chunkSize];
        file.Read(chunk, 0, chunkSize);
        context.Response.BinaryWrite(chunk);
    }
}

Did you use a profiler to identify this code as the culprit? — Daniel Hilgarth, Feb 21 '12 at 15:15
What is the value of chunkSize at the time that you get the exception? — JMarsch, Feb 21 '12 at 15:39
Ok, I've run it for a while with the BinaryWrite commented out and it doesn't leak in full-fat .net 4. Not sure if that helps much. — , Feb 21 '12 at 16:24
I usually use debuggers in similar cases (deleaker, valgrind or VLD)... prefer first) — John Smith, Feb 23 '12 at 17:34

score 6 · Accepted Answer · answered Feb 21 '12 at 15:23

The problem is almost certainly that you're repeatedly allocating new arrays and in memory they're allocated as contiguous blocks, so I can understand how it's chewing through it.

How about rejigging things slightly so that you only create the buffer once and then reuse it unless you get into the if where the chunksize required is less than the standard chunk size.

using (var file = new FileStream(fileLoc, FileMode.Open)) {
    int chunkSize = 1024 * 100;
    byte[] chunk = new byte[chunkSize];

    while (file.Position < file.Length) {
        if (file.Length - file.Position < chunkSize) {
            chunkSize = (int)(file.Length - file.Position);
            chunk = new byte[chunkSize];
        }
        file.Read(chunk, 0, chunkSize);
        context.Response.BinaryWrite(chunk);
    } 
}

Sounds like what I said earlier about newing something up in a While loop or for loop... good catch Nanhydrin I will +1 you now — MethodMan, Feb 21 '12 at 15:46

score 2 · Answer 2 · 2012-02-22T00:14:41.523

May I suggest that you try with a smaller buffer size?

The problem could be that you're repeatedly allocating a memory block that is larger than 85000 bytes and that goes to a special heap (Large Object Heap) that unfortunately is never compacted!

See here for a detailed explanation of how the Large Object Heap works. This unfortunately can lead to severe heap fragmentation and ultimately cause an out of memory error like the one you're describing (see here: loh fragmentation causes OutOfMemory exception)

If you allocate smaller chunks (smaller than 85,000 bytes) then they will be allocated on the regular heap, then the GC will be able to perform compaction and almost certainly, your problem will be gone. I would also strongly recommend you modify your code as suggested by @Nanhydrin, since this avoids repeated allocations and should perform slightly better

score 1 · Answer 3 · edited May 23 '17 at 10:26

I'm not entirely sure what you mean by "runs long enough" however that code allocates an array that is at least 100 KB (possibly larger if the file is larger). In itself this probably won't cause a failure however in an environment with only 32 MB of virtual address space this is a reasonably large allocation of memory. If there are many of these running in parallel this could easily multiply that up to a relatively high memory usage in which case you may see an OutOfMemoryException.

With the assumption that context.Response is a HttpResponse it looks like you are just trying to write the contents of a file to a HTTP response in which case you can do this far more efficiently using something like the following:

using (var file = new FileStream(fileLoc, FileMode.Open))
{
    CopyStream(file, context.Response.OutputStream);
}

See Best way to copy between two Stream instances - C# for an implementation of CopyStream that copies data bit-by-bit in smaller chunks rather than attempting to read the entire file in one go.

Justin how big is your code is it possible to post the code so that we can clarify if there are any memory leaks really going on.. there are a lot of great opinions and views being posted but I think that we are all truly flying blind so to speak without seeing more code.. I am sure that this is not the only place in your code where you are creating instances of Objects, arrays, etc... thanks — MethodMan, Feb 21 '12 at 15:27

OutOfMemoryException in C#

3 Answers3