Writing JSON to a stream without buffering the string in memory

Question

I would like to write JSON to a Stream by building the document up explicitly. For example:

var stream = ...;
var writer = new JsonWriter(stream);

writer.BeginArray();
{
  writer.BeginObject();
  {
    writer.String("foo");
    writer.Number(1);
    writer.String("bar");
    writer.Number(2.3);
  }
  writer.EndObject();
}
writer.EndArray();

This would produce:

[
  {
    "foo": 1,
    "bar": 2.3
  }
]

The benefit of this approach is that nothing needs to be buffered in memory. In my situation, I'm writing quite a lot of JSON to the stream. Solutions such as this one involve creating all your objects in memory, then serialising them to a large string in memory, then finally writing this string to the stream and garbage collecting, probably from the LOH. I want to keep my memory use low, writing out elements while reading data from another file/DB/etc stream.

This kind of approach is available in C++ via the rapidjson library.

I've searched around a fair bit for this and haven't found a solution.

Drew Noakes · Accepted Answer · 2013-08-16T09:29:33.770

25

Turns out I needed to Google for a bit longer.

JSON.NET does indeed support this via its JsonWriter class.

My example would be written:

Stream stream = ...;

using (var streamWriter = new StreamWriter(stream))
using (var writer = new JsonTextWriter(streamWriter))
{
    writer.Formatting = Formatting.Indented;

    writer.WriteStartArray();
    {
        writer.WriteStartObject();
        {
            writer.WritePropertyName("foo");
            writer.WriteValue(1);
            writer.WritePropertyName("bar");
            writer.WriteValue(2.3);
        }
        writer.WriteEndObject();
    }
    writer.WriteEndArray();
}

edited Aug 16 '13 at 09:29

answered Aug 16 '13 at 09:24

Drew Noakes

300,895
165
679
742

Why do you use those unnecessary `{`, `}`s? – I4V Aug 16 '13 at 09:33
1

And why should I explicitly write all elements manually instead of simply doing `jsonSerializer.Serialize(jsonTextWriter,obj)` ? – I4V Aug 16 '13 at 09:41
6

@I4V -- the `{}` are for readability and are optional. In a non-trivial example they can really help. Writing elements manually is a good idea if you have many, many elements to write that follow a simple repeating format. For example, if you are writing JSON to an HTTP response stream using 100,000 elements from a `SqlDataReader`, writing in this fashion avoids hitting `OutOfMemoryException` and, even if you don't have an OOME, your web server will perform much better. So, it depends upon the demands of your situation. – Drew Noakes Aug 16 '13 at 09:46
@I4V, also, in my case I'm using a visitor pattern type arrangement where using a `JsonWriter` provides a nice abstraction, allowing different handlers to write whatever they want. – Drew Noakes Aug 16 '13 at 09:47
3

@I4V Moreover is this a way more performant approach. Serialization always takes a lot of overhead with it. By manually writing only the data that is necessary you can avoid a lot of work. After all automatic serialization does need to check for certain attributes and analyze the object's structure which does take considerably more time than manuall writing it. In my projects I do write my files manually because it could take ages with huge data amounts. And sorry, I know this thread is old. But I thought this is worth mentioning. – SharpShade May 30 '17 at 13:37

Writing JSON to a stream without buffering the string in memory

1 Answers1

Linked