Why does my program's memory usage rise quickly and then go flat?

Question

Hi everyone I'm making a C# WinForms application that searches for duplicate images in a directory. It starts by calling a constructor with every image in the directory.

There is a lot of files in the directory and memory was quickly rising to 2gb and then the program would throw an out of memory exception.

Now I've added a check in my for loop to check if the memory has exceeded 800 Megabits and I force a garbage collection. But I've noticed after the first forced collection the memory no longer rises. (The forced garbage collection occurs at loop ~180 out of ~800 then it never occurs again)

(It looks a little like a shark fin swimming through the water leaving waves in its wake.)

I'm stumped as to why this is happening and have come here in search of help.

private void GeneratePrints()
{
    for (int i = 0; i < files.Count; i++)
    {
        if (imageFileExtensions.Contains(Path.GetExtension(files[i])))
            prints.Add(new FilePrint(directory + "/" + files[i]));

        //800 Megabits
        const long MAX_GARBAGE = 800 * 125000;

        if (GC.GetTotalMemory(false) > MAX_GARBAGE)
        {
            GC.Collect();
            GC.WaitForPendingFinalizers();
        }
    }

    Console.WriteLine("File Prints Complete.");
}

GeneratePrints() is called 1 time, once a directory is selected.

I will also show you the constructor for the FilePrint class. I'm pretty sure this all has something to do with the MemoryStream object.

public FilePrint(string filename)
{
    Bitmap img;

    using (var fs = new FileStream(filename, FileMode.Open))
    {
        using (var ms = new MemoryStream())
        {
            fs.CopyTo(ms);
            ms.Position = 0;
            img = (Bitmap)Bitmap.FromStream(ms);

            ms.Close();
        }

        fs.Close();
    }

    this.size = img.Size;
    img = ResizeImage(img, 8, 8);
    img = MakeGrayscale(img);

    //I do some basic for-loop arithmetic here 
    //calculating the average color of the image, not worth posting.

    img.Dispose();
}

So basically I'm wondering how can I make it so that the 'shark-fin-like' memory usage spike at the start never happens so that I do not have to force a garbage collection.

Here is a memory snapshot I have taken when the forced garbage collection occurs (Am I not disposing of the MemoryStreams properly?):

Thank you for your replies, ideas and answers in advance!

Answer 1

You don't show the methods ResizeImage(img, 8, 8) and MakeGrayscale(img), but most likely they simply create and return a new image based on the old. If that's true, your code constructs two Bitmap objects that it never explicitly disposes, so try disposing them e.g. as follows:

using (var old = img)
    img = ResizeImage(old, 8, 8);
using (var old = img)
    img = MakeGrayscale(old);

You might also want to guarantee that the final img is disposed using a try/finally:

Bitmap img = null;
try
{
    img = new Bitmap(filename);  // Here I simplified the code, but this will leave the file locked until `img` is disposed after resizing.

    this.size = img.Size;
    using (var old = img)
        img = ResizeImage(old, 8, 8);
    using (var old = img)
        img = MakeGrayscale(old);

    //I do some basic for-loop arithmetic here 
    //calculating the average color of the image, not worth posting.
}
finally
{
    if (img != null)
        img.Dispose();
}

The possible reason you get the long buildup in memory use then a precipitous drop is that eventually the unmanaged resources of the undisposed images will get finalized, however because the GC is unaware of unmanaged memory owned by the undisposed Bitmap objects, it doesn't necessarily kick in, identify the bitmaps as unreferenced and pass them on to the finalizer thread for quite a while. It's not always easy to predict when or even if the finalizer thread will spin up and start working; see Are .net finalizers always executed? to which the answer is not necessarily. But by calling GC.WaitForPendingFinalizers(); you may be kickstarting that process.

Incidentally, a MemoryStream doesn't actually need to be disposed in the current implementation as it lacks unmanaged resources. See this answer by Jon Skeet to Is a memory leak created if a MemoryStream in .NET is not closed? for confirmation. (It's still good practice to do so, though in the case of Bitmap there's that pesky file/stream lock that makes it impossible.)