Iterate over a large JSON Array with JSONPath

Question

I have a simple Java app that needs to traverse a large JSON array (with around 20K items), and within each array, I parse a sub-array. Each item looks like this:

{"index":0,"f1":[2,16,16,16,16,16,32,16],"f2":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"startTime":0.0}

I am using JSONPath to iterate over each item. What I do is first I read the length, and simply iterate over the whole array. But it is very slow (like, 1 item per second).

int length = JsonPath.read(response, "$.result.length()");
for (int i = 0; i < length; i++) {
    double start_time = JsonPath.read(response, "$.result["+i+"].startTime");
    ArrayList<Integer> f1= JsonPath.read(response, "$.result["+i+"].f1");
    //...other things
}

Is there a way to optimise it?

Yes - first parse your JSON file into a data structure only once, and the you access the data structure. What you currently do is parsing the JSON every time you retrieve a value from it, which is, as you have noticed, very, very slow. See https://stackoverflow.com/questions/2591098/how-to-parse-json-in-java — Erwin Bolwidt, Mar 26 '19 at 21:36

score 3 · Answer 1 · answered Mar 26 '19 at 23:17

You should minimise number of read operation. First time you scan the whole file and and next you scan n times file partially. Reading from disk is slower than from memory: Latency Numbers Every Programmer Should Know, so you should load the file to memory once and then iterate over items. Also, from JsonPath documentation:

If you only want to read once this is OK. In case you need to read an other path as well this is not the way to go since the document will be parsed every time you call JsonPath.read(...). To avoid the problem you can parse the json first.

String json = "...";
Object document = Configuration.defaultConfiguration().jsonProvider().parse(json);

List<Integer> f10 = JsonPath.read(document, "$.result[0].f1");
List<Integer> f11 = JsonPath.read(document, "$.result[1].f1");

You can improve your JsonPath: $.result and read only what you need by: $.result..['f1','startTime'].

Example app which loads only required fields:

import com.jayway.jsonpath.JsonPath;

import java.io.File;
import java.util.List;
import java.util.Map;

public class JsonPathApp {

    public static void main(String[] args) throws Exception {
        File jsonFile = new File("./resource/test.json").getAbsoluteFile();
        List<Object> array = JsonPath.read(jsonFile, "$.result..['f1','startTime']");
        for (Object item : array) {
            Map<String, Object> map = (Map<String, Object>) item;
            System.out.println(map.get("f1"));
            System.out.println(map.get("startTime"));
        }
    }
}

Thanks. Yes this is what I do now. Lighting speed now...unbelievable! — Tina J, Mar 27 '19 at 14:43
@TinaJ, I'm glad to hear that. We have more tools than only `JsonPath`, take a look on similar question with `Jackson` example: [How do I read a data from a JSON file with high efficiency in Java with Jackson?](https://stackoverflow.com/questions/54906877/how-do-i-read-a-data-from-a-json-file-with-high-efficiency-in-java-with-jackson). In case, my answer was helpful take a look on [How does accepting an answer work?](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work) — Michał Ziober, Mar 27 '19 at 19:23

score 1 · Accepted Answer · answered Mar 26 '19 at 23:10

Got it. Thanks to Erwin, I can parse the whole JSON at once into a HASHMap simply like this:

ArrayList<HashMap> json= JsonPath.read(response, "$.result");

And then we can simply call get(i) to access a specific item in the loop:

double start_time = (double) json.get(i).get("startTime");

Iterate over a large JSON Array with JSONPath

2 Answers2

Linked