384

How does one read a very large JSON file into an array in c# to be split up for later processing?

I have managed to get something working that will:

  • Read the file Miss out headers and only read values into array.
  • Place a certain amount of values on each line of an array. (So I could later split it an put into 2d array)

This was done with the code below but it crashes the program after entering a few lines into the array. This might have to do with the file size.

// If the file extension was a jave file the following 
// load method will be use else it will move on to the 
// next else if statement
if (fileExtension == ".json") 
{
    int count = 0;
    int count2 = 0;
    int inOrOut = 0;
    int nRecords=1; 
    JsonTextReader reader = new JsonTextReader(new StreamReader(txtLoaction.Text));
    string[] rawData = new string[5];
    while (reader.Read())
    {
        if (reader.Value != null)
            if (inOrOut == 1)
            {
                if (count == 6)
                {
                    nRecords++;
                    Array.Resize(ref rawData, nRecords);
                    //textBox1.Text += "\r\n";
                    count = 0;
                }
                rawData[count2] += reader.Value + ","; //+"\r\n"
                inOrOut = 0;
                count++;
                if (count2 == 500)
                {
                    MessageBox.Show(rawData[499]);
                }
            }
            else
            {
                inOrOut = 1;
            }
    } 
}

A snippet of the JSON I am working with is:

[ 
    { "millis": "1000", 
      "stamp": "1273010254", 
      "datetime": "2010/5/4 21:57:34", 
      "light": "333", 
      "temp": "78.32", 
      "vcc": "3.54" }, 
]

I need the values out of this JSON. For example, I need "3.54", but I would not want it to print the "vcc".

How can one read a JSON file in and only extract the data needed to be put into an array?

ΩmegaMan
  • 29,542
  • 12
  • 100
  • 122
Chris Devine
  • 3,859
  • 2
  • 13
  • 7
  • 1
    What exception does your program throw when it crashes? – tmesser Nov 08 '12 at 21:01
  • 1
    Does this answer your question? [How can I parse JSON with C#?](https://stackoverflow.com/questions/6620165/how-can-i-parse-json-with-c) – Heretic Monkey Jan 03 '20 at 18:15
  • Does this answer your question? [Can Json.NET serialize / deserialize to / from a stream?](https://stackoverflow.com/questions/8157636/can-json-net-serialize-deserialize-to-from-a-stream) – Liam Sep 15 '20 at 08:56

13 Answers13

685

How about making everything easier with Json.NET?

    public void LoadJson()
    {
        using (StreamReader r = new StreamReader("file.json"))
        {
            string json = r.ReadToEnd();
            List<Item> items = JsonConvert.DeserializeObject<List<Item>>(json);
        }
    }

    public class Item
    {
        public int millis;
        public string stamp;
        public DateTime datetime;
        public string light;
        public float temp;
        public float vcc;
    }

You can even get the values dynamically without declaring Item class.

    dynamic array = JsonConvert.DeserializeObject(json);
    foreach(var item in array)
    {
        Console.WriteLine("{0} {1}", item.temp, item.vcc);
    }
Robert Christopher
  • 4,940
  • 1
  • 20
  • 21
L.B
  • 114,136
  • 19
  • 178
  • 224
  • I think that may be working, ill test it out further and let you know. thanks alot for all the help – Chris Devine Nov 08 '12 at 21:25
  • got an error on line "List items = JsonConvert.DeserializeObject>(json); " it was "Unexpected character encountered while parsing value: C. Path '', line 0, position 0." – Chris Devine Nov 08 '12 at 22:13
  • 3
    @ChrisDevine I hope you didn't put the path as `json`. It must be the content of your file. – L.B Nov 08 '12 at 22:24
  • It's actually faster if you declare the output type, so returning dynamic objects is no optimization. If you don't know the type, then you have no choice of course. – Molibar May 23 '13 at 22:24
  • 6
    StreamReader("file.json") needs a Stream not string – v.g. May 22 '15 at 07:40
  • 18
    In C# DotNet Core, use: using (StreamReader r = File.OpenText("file.json")) – Fred Nov 14 '16 at 06:37
  • What if I only want to save the time part of the datetime – Roel Jul 20 '17 at 12:30
  • 32
    For the folks who don't like reading the other answers to understand this one: this solution requires the Json.net package (Newtonsoft.Json) – Tydaeus Jan 18 '18 at 17:35
  • new StreamReader("file.json") i can't able to access the file... error : can't covert string to System.IO.Stream... – Yalamandarao Mar 07 '18 at 04:55
  • 2
    Since you have a `StreamReader` anyway, it would be better to deserialize directly from the stream using `JsonTextReader` as shown in [Can Json.NET serialize / deserialize to / from a stream?](https://stackoverflow.com/a/17788118). The `r.ReadToEnd()` is not needed. – dbc Jun 11 '19 at 19:05
  • @dbc or simply, File.ReadAllText :) – L.B Jun 20 '19 at 21:01
  • it should be `Console.WriteLine("{0} {1}", item.Name, item.Value); ` instead of `Console.WriteLine("{0} {1}", item.temp, item.vcc);` – vicport Jul 15 '19 at 00:29
52

Answer for .NET Core

You can just use the built-in System.Text.Json instead of the 3rd-party Json.NET. To promote reuse, the JSON-file-reading functionality belongs in its own class and should be generic rather than hard-coded to a certain type (Item). Here's a full example:

using System;
using System.IO;
using System.Text.Json;
using System.Threading.Tasks;

namespace Project
{
    class Program
    {
        static async Task Main()
        {
            Item item = await JsonFileReader.ReadAsync<Item>(@"C:\myFile.json");
        }
    }

    public static class JsonFileReader
    {
        public static async Task<T> ReadAsync<T>(string filePath)
        {
            using FileStream stream = File.OpenRead(filePath);
            return await JsonSerializer.DeserializeAsync<T>(stream);
        }
    }

    public class Item
    {
        public int millis;
        public string stamp;
        public DateTime datetime;
        public string light;
        public float temp;
        public float vcc;
    }
}

Or, if you prefer something simpler/synchronous:

class Program
{
    static void Main()
    {
        Item item = JsonFileReader.Read<Item>(@"C:\myFile.json");
    }
}

public static class JsonFileReader
{
    public static T Read<T>(string filePath)
    {
        string text = File.ReadAllText(filePath);
        return JsonSerializer.Deserialize<T>(text);
    }
}
MarredCheese
  • 17,541
  • 8
  • 92
  • 91
  • i tried your solution but CSC doesn't want Main() to be async : error CS4009: "A void or int returning entry point cannot be async" – geriwald Jun 17 '21 at 08:46
  • @geriwald Sorry, `async Main` needs to return `Task`, not `void`. I updated it. – MarredCheese Jun 17 '21 at 13:18
  • Worth a mention, perhaps: this is first available with .NET 5.0. If you're stuck with something earlier, see the Json.net solution. – JamieB May 05 '23 at 17:03
  • @JamieB Based on which versions of the documentation page exist, I believe it is available starting with [.NET Core 3.0](https://learn.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.0). But you are right to point out it wasn't there right from the start. – MarredCheese May 06 '23 at 03:09
50

Doing this yourself is an awful idea. Use Json.NET. It has already solved the problem better than most programmers could if they were given months on end to work on it. As for your specific needs, parsing into arrays and such, check the documentation, particularly on JsonTextReader. Basically, Json.NET handles JSON arrays natively and will parse them into strings, ints, or whatever the type happens to be without prompting from you. Here is a direct link to the basic code usages for both the reader and the writer, so you can have that open in a spare window while you're learning to work with this.

This is for the best: Be lazy this time and use a library so you solve this common problem forever.

Lauren Rutledge
  • 1,195
  • 5
  • 18
  • 27
tmesser
  • 7,558
  • 2
  • 26
  • 38
  • 1
    I am using Json.net but i do not understand how it works correctly. when i read the information using JsonTextReader into a textbox i get every bit of data but also headers etc. I just want the vaules in the headers. I have tried to read the Json.NET documentation but i did not find it exsplains everything enough for me to use it in the way i would like – Chris Devine Nov 08 '12 at 21:03
  • @ChrisDevine "Json headers"? Are you referring to the keys? Perhaps this would be easier if you posted a short (~10-15 lines) snippet of JSON and point to precisely what you're trying to extract. – tmesser Nov 08 '12 at 21:05
  • @ChrisDevine I just added your comment here to your question, so if you could kindly delete the comment above this one that'd be great. – tmesser Nov 08 '12 at 21:23
  • @ChrisDevine Also, if you could answer the comment I have on your question that would also be awesome. – tmesser Nov 08 '12 at 21:24
  • I did post a snippet of the data i am trying to recover – Chris Devine Nov 08 '12 at 22:27
  • 1
    @ChrisDevine Yes, I am saying I put it in your question so it's no longer necessary here. – tmesser Nov 08 '12 at 22:32
33

This can also be done in the following way:

JObject data = JObject.Parse(File.ReadAllText(MyFilePath));
Adrita Sharma
  • 21,581
  • 10
  • 69
  • 79
18
string jsonFilePath = @"C:\MyFolder\myFile.json";
            
string json = File.ReadAllText(jsonFilePath);
Dictionary<string, object> json_Dictionary = (new JavaScriptSerializer()).Deserialize<Dictionary<string, object>>(json);

foreach (var item in json_Dictionary)
{
    // parse here
}
live2
  • 3,771
  • 2
  • 37
  • 46
Kanad Mehta
  • 371
  • 3
  • 4
16

Based on @L.B.'s solution, the (typed as Object rather than Anonymous) VB code is

Dim oJson As Object = JsonConvert.DeserializeObject(File.ReadAllText(MyFilePath))

I should mention that this is quick and useful for constructing HTTP call content where the type isn't required. And using Object rather than Anonymous means you can maintain Option Strict On in your Visual Studio environment - I hate turning that off.

SteveCinq
  • 1,920
  • 1
  • 17
  • 22
9

For any of the JSON parse, use the website http://json2csharp.com/ (easiest way) to convert your JSON into C# class to deserialize your JSON into C# object. 

 public class JSONClass
 {
        public string name { get; set; }
        public string url { get; set; }
        public bool visibility { get; set; }
        public string idField { get; set; }
        public bool defaultEvents { get; set; }
        public string type { get; set; }        
 }

Then use the JavaScriptSerializer (from System.Web.Script.Serialization), in case you don't want any third party DLL like newtonsoft.

using (StreamReader r = new StreamReader("jsonfile.json"))
{
   string json = r.ReadToEnd();
   JavaScriptSerializer jss = new JavaScriptSerializer();
   var Items = jss.Deserialize<JSONClass>(json);
}

Then you can get your object with Items.name or Items.Url etc.

shailesh gavathe
  • 341
  • 3
  • 10
  • 9
    You can turn json to C# directly in VS now, `Edit` -> `paste special` -> `Paste json as Classes`. – Liam Sep 15 '20 at 09:00
8

Very Easiest way I found on online to work with .JSON file in C#(or any other Programming Language)

Prerequisite:-

and here is the URL -> https://app.quicktype.io/

Steps

1> go to this URL - https://app.quicktype.io/

2> Copy and Paste your JSON file structure into Left sidebar

app.quicktype.io

3> Select required Language (here C#) from Options menu

4> Copy generated code and go to your Project and Create a new .cs file with the same name(here "Welcome.cs")

Welcome.cs

5> Paste all generated code into the newly created class.

Welcome.cs pasted Code

6> that's it. :)

Steps to Access value

1> Go to Main Program .cs file or wherever you need to access it.

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Access Json values using Keys.>");

        String jsonString = new StreamReader("give <.json> file Path here").ReadToEnd();

        // use below syntax to access JSON file
        var jsonFile = Welcome.FromJson(jsonString);

        string FileName = jsonFile.File;
        long Lvl = jsonFile.Level;
        bool isTrue = jsonFile.CSharp;

        Console.WriteLine(FileName);//JSON
        Console.WriteLine(Lvl);//1
        Console.WriteLine(isTrue);//true
    }
}
mohamad zabiulla
  • 83
  • 1
  • 2
5

For finding the right path I'm using 

   var pathToJson = Path.Combine("my","path","config","default.Business.Area.json");
   var r = new StreamReader(pathToJson);
   var myJson = r.ReadToEnd();

   // my/path/config/default.Business.Area.json 
   [...] do parsing here

Path.Combine uses the Path.PathSeparator and it checks whether the first path has already a separator at the end so it will not duplicate the separators. Additionally, it checks whether the path elements to combine have invalid chars.

See https://stackoverflow.com/a/32071002/4420355

kuzdu
  • 7,124
  • 1
  • 51
  • 69
  • 2
    Better way to find absolute path regardless of the application: https://stackoverflow.com/questions/15653921/get-current-folder-path/23513793#23513793 – user3326078 Jan 17 '19 at 23:28
4

There is a faster way of parsing json then Json.Net . If you are using .net core 3.0 or up then you can use the System.Text.Json nuget package to serialize or deserialize.

you need to add:

using System.Text.Json

And then you can serialize as:

var jsonStr = JsonSerializer.Serialize(model);

And Deserialize as:

var model = JsonSerializer.Deserialize(jsonStr);
saad bin sami
  • 364
  • 4
  • 11
1

This code can help you:

string _filePath = Path.GetDirectoryName(System.AppDomain.CurrentDomain.BaseDirectory);

JObject data = JObject.Parse(_filePath );
Mojtaba Nava
  • 858
  • 7
  • 17
1

There is an easier way to get JSON from file or from the Web: Json.Net.Curl

Install-Package Json.Net.Curl

// get JObject from local file system 
var json = Json.Net.Curl.Get(@"data\JObjectUnitTest1.json");
var json = await Json.Net.Curl.GetAsync(@"data\JObjectUnitTest1.json")


// get JObject from Server  
var json = await Json.Net.Curl.GetAsync("http://myserver.com/data.json");

GitHub Project Nuget

Grzegorz Smulko
  • 2,525
  • 1
  • 29
  • 42
KC85
  • 11
  • 1
0

With Cinchoo ETL, an open source library, parsing of very large JSON file is iterative and simple to use

1. Dynamic Method: - No POCO class required

        string json = @"
[
  {
    ""millis"": ""1000"",
    ""stamp"": ""1273010254"",
    ""datetime"": ""2010/5/4 21:57:34"",
    ""light"": ""333"",
    ""temp"": ""78.32"",
    ""vcc"": ""3.54""
  },
  {
    ""millis"": ""2000"",
    ""stamp"": ""1273010254"",
    ""datetime"": ""2010/5/4 21:57:34"",
    ""light"": ""333"",
    ""temp"": ""78.32"",
    ""vcc"": ""3.54""
  }
] 
";
        
        using (var r = ChoJSONReader.LoadText(json))
        {
            foreach (var rec in r)
                Console.WriteLine(rec.Dump());
        }

Sample fiddle: https://dotnetfiddle.net/mo1qvw

2. POCO Method:

Define POCO class matching json attributes

public class Item
{
    public int Millis { get; set; }
    public string Stamp { get; set; }
    public DateTime Datetime { get; set; }
    public string Light { get; set; }
    public float Temp { get; set; }
    public float Vcc { get; set; }
}

Then using the parser to load the JSON as below

        string json = @"
[
  {
    ""millis"": ""1000"",
    ""stamp"": ""1273010254"",
    ""datetime"": ""2010/5/4 21:57:34"",
    ""light"": ""333"",
    ""temp"": ""78.32"",
    ""vcc"": ""3.54""
  },
  {
    ""millis"": ""2000"",
    ""stamp"": ""1273010254"",
    ""datetime"": ""2010/5/4 21:57:34"",
    ""light"": ""333"",
    ""temp"": ""78.32"",
    ""vcc"": ""3.54""
  }
] 
";
        
        using (var r = ChoJSONReader<Item>.LoadText(json))
        {
            foreach (var rec in r)
                Console.WriteLine(ChoUtility.Dump(rec));
        }

Sample fiddle: https://dotnetfiddle.net/fRWu0w

Disclaimer: I'm author of this library.

Cinchoo
  • 6,088
  • 2
  • 19
  • 34