C# exception OutOfMemory

Question

I would need some help how to handle this exception:

StackTrace :   at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult)
   at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject)
   at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
   at System.Configuration.Configuration.GetSection(String sectionName)
   at System.Configuration.ClientSettingsStore.GetConfigSection(Configuration config, String sectionName, Boolean declare)
   at System.Configuration.ClientSettingsStore.WriteSettings(String sectionName, Boolean isRoaming, IDictionary newSettings)
   at System.Configuration.LocalFileSettingsProvider.SetPropertyValues(SettingsContext context, SettingsPropertyValueCollection values)
   at System.Configuration.SettingsBase.SaveCore()
   at System.Configuration.SettingsBase.Save()
   at System.Configuration.ApplicationSettingsBase.Save()
   at BayesianSpamFilter.MainWindow.SaveSettings()

This is what I am doing:

string percentG = Properties.Settings.Default.PercentG;
string percentB = Properties.Settings.Default.PercentB;
string percentR = Properties.Settings.Default.PercentR;
string percentF = Properties.Settings.Default.PercentF;

string spamDbM = Properties.Settings.Default.SpamDbM;
string spamDbS = Properties.Settings.Default.SpamDbS;
string hamDbM = Properties.Settings.Default.HamDbM;
string hamDbS = Properties.Settings.Default.HamDbS;

private void SaveSettings()
{

    percentG = JsonConvert.SerializeObject(spamPercentageGraham);
    Properties.Settings.Default.PercentG = percentG;

    percentB = JsonConvert.SerializeObject(spamPercentageBurton);
    Properties.Settings.Default.PercentB = percentB;

    percentR = JsonConvert.SerializeObject(spamPercentageRobinson);
    Properties.Settings.Default.PercentR = percentR;

    percentF = JsonConvert.SerializeObject(spamPercentageFisher);
    Properties.Settings.Default.PercentF = percentF;

    spamDbM = JsonConvert.SerializeObject(spamDatabaseMulti);
    Properties.Settings.Default.SpamDbM = spamDbM;

    spamDbS = JsonConvert.SerializeObject(spamDatabaseSingles);
    Properties.Settings.Default.SpamDbS = spamDbS;

    hamDbM = JsonConvert.SerializeObject(hamDatabaseMulti);
    Properties.Settings.Default.HamDbM = hamDbM;

    hamDbS = JsonConvert.SerializeObject(hamDatabaseSingles);
    Properties.Settings.Default.HamDbS = hamDbS;


    Properties.Settings.Default.Save();
}

private void LoadSettings()
{
    spamPercentageGraham = JsonConvert.DeserializeObject<Dictionary<string, double>>(percentG);
    spamPercentageBurton = JsonConvert.DeserializeObject<Dictionary<string, double>>(percentB);
    spamPercentageRobinson = JsonConvert.DeserializeObject<Dictionary<string, double>>(percentR);
    spamPercentageFisher = JsonConvert.DeserializeObject<Dictionary<string, double>>(percentF);

    spamDatabaseMulti = JsonConvert.DeserializeObject<Dictionary<string, int>>(spamDbM);
    spamDatabaseSingles = JsonConvert.DeserializeObject<Dictionary<string, int>>(spamDbS);
    hamDatabaseMulti = JsonConvert.DeserializeObject<Dictionary<string, int>>(hamDbM);
    hamDatabaseSingles = JsonConvert.DeserializeObject<Dictionary<string, int>>(hamDbS);
}

Also: Count() for each of the dictionaries is around 300k. I am not running out of physical memory, nor RAM.. I am counting many diferent things with those dictionaries so I need all of them.

I had to smile a bit at the end of your question. Everything looks reasonable until you mention, almost as an afterthought hardly worth mentioning, that these dictionaries are GIGANTIC... :-D — stakx - no longer contributing, May 16 '15 at 12:57
Well, compiling for something "standard", never used to do things as big as those now. How can I change compiling method to 64b? — Ken'ichi Matsuyama, May 16 '15 at 13:00
[Check out this!](http://stackoverflow.com/questions/3661368/how-can-i-choose-between-32-bit-or-64-bit-build-in-c-sharp-express) — Simon, May 16 '15 at 13:03
There has to be a better solution than loading all of these big dictionaries into memory at the same time. — John, May 16 '15 at 13:05
You could try saving them one at a time. Currently you have all the dictionaries in memory and all the serialisations of them. One at a time would descope the serialisation and bin it. Personally I would never have thought of settngs to store data like this. I would have used settings to pick a folder and the loaded and saved with file stream. Simple name= value text would have done it. — Tony Hopkinson, May 16 '15 at 13:07
Settings are not meant to store data: there are databses and hard drive files for that. Settings are meant to store... settings, thats it to say some strings and boolean flags that change how certains part of your application behave but certainly not its data. — Kryptos, May 16 '15 at 13:23
You guys are probably right, gonna think about it, for now 64b worked. — Ken'ichi Matsuyama, May 16 '15 at 13:35

score 0 · Answer 1 · answered May 16 '15 at 12:58

0

Try setting the architecture of your application to 64bit. Then your application should be able to allocate more memory!

answered May 16 '15 at 12:58

Simon

1,244
8
21

1

What if it is to run on a 32bit machine? – User92 May 16 '15 at 13:00
Allocating more memory to a machine because a program has a bug is never a viable solution. – John May 16 '15 at 13:01
@Jevans92 Then he can't hold all these list in memory! – Simon May 16 '15 at 13:01
He's go all the lists in memory, Problem is in save is he has all the serialisations of them as well... – Tony Hopkinson May 16 '15 at 13:16

score 0 · Accepted Answer · edited May 23 '17 at 12:06

A few things to consider.

The architecture (32 bit / 64 bit) that you are targeting. Like this answer said already, try compiling for a 64 bit platform if that is an option. 32-bit .NET processes are limited to 2 GB no matter how much RAM your machine's got.

You can do this by creating a 64 bit configuration in Build → Configuration Manager → Active solution platform → New….

This of course requires that you are not going to develop or run your program on a 32 bit machine.

Even if the above is an option, I think you would do well to consider whether you can change your application design (or more precisely, the way how you persist that data):

Do you need to store all that data, or would a subset do? Is there any way you can reduce the data that has to be saved? For example, can some data be derived from other data by applying some fixed rule, so that you would only have to store the latter kind of data?
Can you persist your data to some file(s) other than Web/App.config? (Besides: the settings storage mechanism you are currently using should not be abused for storing any kind of data.) If you persisted your dictionaries using some other storage mechanism (such as one file per dictionary), you might be able to...
Serialize and persist one dictionary after the other, one at a time. That might enable you to dispose no-longer-needed dictionaries (and their serialized representations) early, you'd only ever have to keep one serialization output in memory at a time. This would likely relieve memory pressure.

C# exception OutOfMemory

2 Answers2