Freeing memory while iterating through list before reaching end of loop

Question

I'm working with various lists and types under memory constraints and would like to free up memory while iterating through a list, so I've set up a generic example:

public class Test
{
    List<PersonClass> personList = new(450000);

    public Test()
    {
        for(int i = 0; i < 450000; i++)
        {
            personList.Add(new PersonClass());
        }
    }

    public List<InfoClass> GetUpdatedList(List<PersonClass> personList)
    {
        List<InfoClass> secondList = new();
        
        for(int i=personList.Count - 1; i > -1; i--)
        {
            var aPerson = personList[i];
            
            secondList.Add(new InfoClass()
            {
                FirstName = aPerson.FirstName,
                LastName = aPerson.LastName
            });
            
            personList.RemoveAt(i);
        }

        return secondList;
    }
    
    public class PersonClass
    {
        public string FirstName { get; set; } = "Something Random";
        public string LastName { get; set; } = "Something Else Random";
    }

    public class InfoClass
    {
        public string FirstName { get; set; } = "Something Random";
        public string LastName { get; set; } = "Something Else Random";
        public string StateName { get; set; } = "Something Random";
        public string CityName { get; set; } = "Something Else Random";
    }
}

How can I free up memory while removing elements before reaching the end of the loop?

Doing your own memory management isn't advisable, but you can force garbage collection to happen: https://stackoverflow.com/questions/4257372/how-to-force-garbage-collector-to-run — computercarguy, Nov 18 '22 at 18:49
Well, how many hundreds of thousands of PersonsClass, InfoClass and other instances in your lists are you dealing with for memory consumption becoming an issue? Where are all these instances coming from; or in other words: Do all these instances really need to be live at once? — , Nov 18 '22 at 18:53
It's for generating reports, so I need to collate data and use a pdf writing service to generate the report. As soon as I read through a list element, it can be disposed and hopefully free up the memory. — AtomicallyBeyond, Nov 18 '22 at 19:06
Why you are concerned about memory? Did you run into a limit? How many element you have? — Aldert, Nov 18 '22 at 19:11
Person class is an example, there are about 30 - 40 different types, and the data is collated in various ways. I can't exceed over a couple hundred mb because I'm working with a legacy system. — AtomicallyBeyond, Nov 18 '22 at 19:11
@computercarguy by forcing garbage collection, will that free up the memory allocated for the removed elements? — AtomicallyBeyond, Nov 18 '22 at 19:15
@AtomicallyBeyond, C# should automatically free up memory on it's own timetable so you shouldn't have to deal with it. Garbage collection is that process of freeing up memory. I'd like to say it will 100% help you cleaning up the memory, but that may be something you only figure out by testing it. And forcing the memory collection may end up costing you speed or have other side effects you aren't intending, which is why it's not recommended to to it yourself and why extensive testing is necessary to make sure it doesn't have unintended consequences. — computercarguy, Nov 18 '22 at 19:25
@computercarguy if I remove items from the list, is there a possibility GC - if memory is too low - free up that memory from heap before ending the loop? — AtomicallyBeyond, Nov 18 '22 at 20:48
@AtomicallyBeyond, normal garbage collection frees up memory when it reaches certain limits. I'm not sure what those limits are, so I don't know if it'll help you or not, and different versions of C# use different reasoning for GC, so you'd have to research that yourself. But yes, generally, if a machine is running low on memory, GC will try to run more frequently to try to free up more space. However, with barrett777's answer, you don't have to worry about it. And MikeHofer's answer explains better what GC does and lots of other stuff that you need to consider. — computercarguy, Nov 18 '22 at 21:00
if memory is your great concern, .net may be not the platform for you. If you use .net, consider using things like `WeakReference` — T.S., Nov 18 '22 at 21:06
@AtomicallyBeyond, with your report, if you are reading the list from a DB, you may be able to use paging in your data query to reduce memory requirements. https://stackoverflow.com/questions/109232/what-is-the-best-way-to-paginate-results-in-sql-server It'll require multiple queries to the DB, but you'll be dealing with a lot less data at a single time. And, depending on what PDF service you are using, you might be able to generate smaller groups of pages to have in memory at a single time, then stitch them together before sending it to the client, which will also save on memory. — computercarguy, Nov 18 '22 at 21:08

score 1 · Answer 1 · answered Nov 18 '22 at 20:05

1

You can save a lot of memory if InfoClass contains a reference to the appropriate PersonClass, instead of creating new strings.

If InfoClass is a summary of many other objects, try just storing references to the other objects, instead of duplicating the properties

answered Nov 18 '22 at 20:05

barrett777

282
3
14

Some example code might make this easier to understand, but I think this is a good approach. – computercarguy Nov 18 '22 at 20:54

score 1 · Answer 2 · answered Nov 18 '22 at 20:57

A couple of things:

Variables will be freed automatically when they go out of scope. If you declare a variable within curly braces, that is its scope.
The garbage collector will automatically run when there is sufficient memory pressure to trigger it. Memory pressure can be caused by a number of things, but one of them is the inability to allocate enough contiguous free space on the heap for a new object.
Manually running GC.Collect() is frowned upon for a number of reasons, not the least of which is that it's slow. You certainly don't want to be doing it in a loop of any kind.
If you don't KNOW how much memory is being used by your application, and if that memory is higher than the amount of memory available to you at runtime, what you're proposing is premature optimization, the root of all evil. There are tools (some of them built right into Visual Studio) that will let you determine how much memory your application is using when it runs. Once you have that information, then you can make an informed decision about whether or not you should even worry about this sort of an optimization.
If you must optimize, you may want to consider using a struct instead of a class if you know that the class can be converted to one. structs are allocated on the stack instead of the heap, saving a little bit of memory, but they must be read-only and must meet specific size requirements. But they will be deallocated immmediately when they go out of scope, and heap memory won't be an issue.

Freeing memory while iterating through list before reaching end of loop

2 Answers2