6

I am trying to parse the JSON files and insert into the SQL DB.My parser worked perfectly fine as long as the files are small (less than 5 MB).

I am getting "Out of memory exception" when trying to read the large(> 5MB) files.

if (System.IO.Directory.Exists(jsonFilePath))
                {
                    string[] files = System.IO.Directory.GetFiles(jsonFilePath);
                    foreach (string s in files)
                    {
                        var jsonString = File.ReadAllText(s);
                        fileName = System.IO.Path.GetFileName(s);
                        ParseJSON(jsonString, fileName);

                    }
                }

I tried the JSONReader approach, but no luck on getting the entire JSON into string or variable.Please advise.

user1046415
  • 779
  • 4
  • 23
  • 43
  • 2
    what's "large"? 5.1meg? 5555555555 megabytes? – Marc B Jul 25 '16 at 21:22
  • *I tried the JSONReader approach, but no luck on getting the entire JSON into string or variable.* If by "JSONReader" you mean [`JsonTextReader`](http://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonTextReader.htm) from Json.NET, can you share what you tried but did not work? – dbc Jul 25 '16 at 21:39
  • foreach (string s in files){ fileName = System.IO.Path.GetFileName(s); using (WebClient client = new WebClient()){using (StreamReader sr = new StreamReader(client.OpenRead(jsonFilePath + fileName))) {using (JsonReader reader = new JsonTextReader(sr)){ var jsonString= reader.Value.ToString();ParseJSON(jsonString, fileName) } } } – user1046415 Jul 26 '16 at 14:06
  • Can you share a sample of what your JSON looks like, and how you are parsing it? I.e., a [mcve] example of your problem giving an idea of what you are doing inside `ParseJSON` would make it much more likely for us to give a helpful answer. – dbc Aug 13 '16 at 17:45

2 Answers2

3

Use 64 bit, check RredCat's answer on a similar question:

Newtonsoft.Json - Out of memory exception while deserializing big object

NewtonSoft Jason Performance Tips

Read the article by David Cox about tokenizing:

"The basic approach is to use a JsonTextReader object, which is part of the Json.NET library. A JsonTextReader reads a JSON file one token at a time. It, therefore, avoids the overhead of reading the entire file into a string. As tokens are read from the file, objects are created and pushed onto and off of a stack. When the end of the file is reached, the top of the stack contains one object — the top of a very big tree of objects corresponding to the objects in the original JSON file"

Parsing Big Records with Json.NET

Community
  • 1
  • 1
A.J.Bauer
  • 2,803
  • 1
  • 26
  • 35
0

The json file is too large to fit in memory, in any form.

You must use a JSON reader that accepts a filename or stream as input. It's not clear from your question which JSON Reader you are using. From which library?

If your JSON reader builds the whole JSON tree, you will still run out of memory. As you read the JSON file, either cherry pick the data you are looking for, or write data structures to another on-disk format that can be easily queried, for example, an sqlite database.

Jeffrey Rennie
  • 3,193
  • 1
  • 18
  • 19