How can REST API pass large JSON?

Question

I am building a REST API and facing this issue: How can REST API pass very large JSON?

Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.

What methods we can use to solve this problem?

DBTraining trainingdata = new DBTraining();
@GET
@Produces("application/json")
@Path("/{cat_id}")
public Response getAllDataById(@PathParam("cat_id") String cat_id) {
    List<TrainingData> list = new ArrayList<TrainingData>();
    try {
        list = trainingdata.getAllDataById(cat_id);
        Gson gson = new Gson();
        Type dataListType = new TypeToken<List<TrainingData>>() {
        }.getType();
        String jsonString = gson.toJson(list, dataListType);
        return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();

    } catch (SQLException e) {
        logger.warn(e.getMessage());
    }
    return null;
}

You're probably going to need to page the data, like /request/?page=1&itemsPerPage=100 and make multiple calls in a loop to get all of the data. — Paul Abbott, Nov 06 '14 at 18:03
maybe json is not the right choice for you. it is intended to move small chunks of data, not the whole db... just my 2c — Z ., Nov 06 '14 at 18:09
@Zdravko Danev So except for JSON, what other methods can I use? Should I upload a file? — Freya Ren, Nov 06 '14 at 18:13
@paul.abbott.wa.us Yeah, I think maybe this is the most common way to do. Just like Amazon shows item pages. — Freya Ren, Nov 06 '14 at 18:14
well, it depends... :) based on what db you are using and what you are trying to accomplish there are different and better options. json is for communicating between a web page and the web server to move small amounts of data. — Z ., Nov 06 '14 at 18:33
REST is about using all of the HTTP standard, which includes HTTP status 202 and the ranges header - http://benramsey.com/blog/2008/05/206-partial-content-and-range-requests/ — MatthewMartin, Nov 06 '14 at 21:07
Maybe this would work for you - https://github.com/typicode/json-server — vsync, May 30 '15 at 18:56

score 4 · Answer 1 · answered Sep 22 '17 at 00:22

The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.

By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.

The GitHub API provides a good example of this.

score 0 · Answer 2 · answered Nov 10 '17 at 20:03

My suggestion is no to pass the data as a JSON but as a file using multipart/form-data. In your file, each line could be a JSON representing a data record. Then, it would be easy to use a FileOutputStream to receive te file. Then, you can process the file line by line to avoid memory problems.

A Grails example:

    if(params.myFile){
           if(params.myFile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
           def fileName = "/tmp/myReceivedFile.txt"
           new FileOutputStream(fileName).leftShift(params.myFile.getInputStream())
       }
       else
           //print or signal error
    }

You can use curl to pass your file:

    curl -F "myFile=@/mySendigFile.txt" http://acme.com/my-service

More details on a similar solution on https://stackoverflow.com/a/13076550/2476435

score 0 · Answer 3 · answered Nov 10 '17 at 20:09

0

HTTP has the notion of chunked encoding that allows you send a HTTP response body in smaller pieces to prevent the server from having to hold the entire response in memory. You need to find out how your server framework supports chunked encoding.

answered Nov 10 '17 at 20:09

Darrel Miller

139,164
32
194
243

How can REST API pass large JSON?

3 Answers3