Streaming one JSON object after another having a JSON array in one huge line in the file

Question

I have a problem with huge JSON file (1GB) which contains an array of JSON objects but all of them are in one line. So it looks like...

[{JsonObject1},{JsonObject1},{JsonObject1},...,{JsonObject999999}]

I'm not able to save the content into the memory so I wanted to do this using streaming. I know how to stream line by line but if I have everything in one line only how can I stream JSON object one after another from this one line array? I tried to browse the internet but I couldn't find anything :(

@FrancisBartkowiak No I'm not, is too big and I want to split it into the smaller pieces — EdXX, Jan 11 '18 at 22:06
Ok. You said it is a file, so I was going to suggest opening it up in Notepad++ and doing some find & replace to add line breaks so it would be easier to read. — Francis Bartkowiak, Jan 11 '18 at 22:28
@FrancisBartkowiak No, its impossible I have 1GB of data and 700K JSON objects in an array — EdXX, Jan 11 '18 at 22:33
I suggest looking at this: https://stackoverflow.com/a/18209319/8972283 — Francis Bartkowiak, Jan 11 '18 at 22:43
I think this post may be helpful to you as well: https://stackoverflow.com/a/13569964/8972283 — Francis Bartkowiak, Jan 12 '18 at 13:07

smac89 · Answer 1 · 2018-02-03T07:12:36.007

Using google gson, you can whip up something that operates lazily:

Maintained as a Gist here:

<script src="https://gist.github.com/smac89/bdcb9b08fcdf9d055150824d57ab3513.js"></script>

import com.google.gson.Gson;
import com.google.gson.stream.JsonReader;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.reflect.Type;
import java.net.URL;
import java.util.Iterator;
import java.util.Spliterators;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;

public class ReadJsonFile {
    private static final class JsonIterator<T> implements Iterator<T> {
        private final Gson gson;
        private final Type objectType;
        private final JsonReader reader;

        private JsonIterator(JsonReader reader, Gson gson, Type objectType) {
            this.gson = gson;
            this.objectType = objectType;
            this.reader = reader;
        }

        @Override
        public boolean hasNext() {
            try {
                return reader.hasNext();
            } catch (IOException ioe) {
                return false;
            }
        }

        @Override
        public T next() {
            return gson.fromJson(reader, objectType);
        }
    }

    public static <J> Stream<J> readJsonFromFile(Gson gson, URL jsonFile, Type type) throws IOException {
        JsonReader reader = new JsonReader(
                new BufferedReader(new InputStreamReader(jsonFile.openStream())));
        reader.beginArray();
        if (!reader.hasNext()) {
            return Stream.empty();
        }
        return StreamSupport.stream(Spliterators.spliteratorUnknownSize(
                new JsonIterator<J>(reader, gson, type), 0), false).onClose(() -> {
            try {
                reader.close();
            } catch (IOException ioe) {
            }
        });
    }
}

data.json

<script src="https://gist.github.com/smac89/15fc3bffa4f965d18587eec2db4972dd.js"></script>

Streaming one JSON object after another having a JSON array in one huge line in the file

1 Answers1