How to parse newline delimited Json response in Android?

Question

Sample of NdJson data:

{"type":"data","id":"xyz"}
{"type":"value","id":"xcf"}
....
....

Here is my Retrofit and RxJava code which is working fine for fetching data with the limit=1 i.e. {"type":"data","id":"xyz"}.

 adapter.create(API.class).getData()
        .subscribeOn(Schedulers.newThread())
        .observeOn(AndroidSchedulers.mainThread())
        .subscribe(new Observer<APIData>() {
         @Override
         public void onCompleted() {}

         @Override
         public void onError(Throwable e) {}

         @Override
         public void onNext(APIData apidata) {}

         });

My model class

Model class have only two parameter:

public class APIData(){
  private String type;
  private String id;

  ..// Getter and setter for above two fields
}

Api class

public interface WarehouseAPI {
  @GET("/search?limit=1")
  public Observable<APIData> getdata ();
}

The error I am getting while replacing @GET("/search?limit=1") with @GET("/search") is Malformed JSON: Syntax error.

Question

How to proper parse the NdJson ?

Is there any way to store the response in List<APIData>

Edit-1

Now I am trying to accept the generic Response in observer:

adapter.create(APIData.class).getdata()
       .subscribeOn(Schedulers.newThread())
       .observeOn(AndroidSchedulers.mainThread())
       .subscribe(new Observer<Response>() {
        @Override
        public void onCompleted() {}

        @Override
        public void onNext(Response response) {}
 });

But getting no suitable method found for subscribe(<anonymous Observer<Response>>)

Edit-2

However, I was doing some silly mistake and the error what I am getting in "Edit-1" section is fixed now. Now I am getting

Error:retrofit.RetrofitError: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 2 column 2 path $

What about just receive json as the string . Split it to array like response.split("\n"). and just iterate through it and creating new new object using gson? — ProblemSlover, Oct 19 '15 at 04:23
@ProblemSlover Imagine I do have 2000 delimiter in one response. Running loop will be time consuming (this is the first activity I need to show in android) — Amit Pal, Oct 19 '15 at 04:30
I would get a `Response`, then split it on `\n`, then flatmap to get each `APIData` as an event of the stream — njzk2, Oct 19 '15 at 05:21
or get the Response, then readline on the body, then emit each line (and parse in another operator) — njzk2, Oct 19 '15 at 05:22
@njzk2 Can you please help me with the pseudo code. I am new in Reactive x BTW I just tried to do what you suggested but got this error: `error: no suitable method found for subscribe(>) ` — Amit Pal, Oct 19 '15 at 05:23

ProblemSlover · Answer 1 · 2015-10-19T09:57:27.113

2

I assume that server response has the following format

{"type":"data","id":"xyz"}\n{"type":"data","id":"xyz"}

The basic idea is to receive the response from the server as the String. Split it into array like response.split("\n"). Iterate through an Array creating a new json object for every array element.

I do realize it's quite time consuming as you've described in the comments. You can also try to play with String replaceAll method to transform each line into an array element and parse the whole string. like

String myResponse = "[" + response.replaceAll("/\n/", ",") + "]";
Gson gson = new Gson();
MyEntity[] arr = gson.fromJson(myResponse, MyEntity[].class);

In case of Retrofit. You will have to use a custom Response Converter. I won't write up the complete solution since you have found the way to get it done using custom Converter.

edited Oct 19 '15 at 09:57

answered Oct 19 '15 at 04:30

ProblemSlover

2,538
2
23
33

Please checkout my comment in the section of Question. Iterating will be more time consuming, Can we do it in much better way i.e. store all string in `List` – Amit Pal Oct 19 '15 at 04:34
Your answer is perfect when I am dealing with normal REST calls but not with Reactive/Rx-java and retrofit in android – Amit Pal Oct 19 '15 at 05:16
I cannot see your previous comment where you suggested to use custom converter. I just want to upvote it :D – Amit Pal Oct 19 '15 at 08:59
@AmitPal Please accept my answer, and you will earn more 2 scores more . and I will be rewarded for a couple hours of research :P Kidding. kiddig. reputation 10 is enough. even though I was aiming to reach 400 today:P – ProblemSlover Oct 19 '15 at 09:30
Lol. You just reached to 400 :P – Amit Pal Oct 19 '15 at 09:56
The whole point of streaming like this is to process large datasets... which means converting them to an array is a bad idea. – Brill Pappin Jun 05 '18 at 13:55

Amit Pal · Accepted Answer · 2015-10-19T09:00:19.827

To make things work I had to add Custom converter implements by Retrofit.Converter:

public class CustomConverter implements Converter {

@Override
public Object fromBody(TypedInput body, Type type) throws ConversionException {
    String text = null;
    try {
        text = fromStream(body.in());
    } catch (IOException e) {
        e.printStackTrace();
    }
    return text;

}

@Override
public TypedOutput toBody(Object object) {
    return null;
}

// Custom method to convert stream from request to string
public static String fromStream(InputStream in) throws IOException {
    BufferedReader reader = new BufferedReader(new InputStreamReader(in));
    StringBuilder out = new StringBuilder();
    String newLine = System.getProperty("\n");
    String line;
    while ((line = reader.readLine()) != null) {
        out.append(line);
        out.append(newLine);
    }
    return out.toString();
}
}

And it works perfectly!

That's exactly what I meant. It's the second time when I followed my doubts and didn't include my answer! — ProblemSlover, Oct 19 '15 at 09:14
you are just getting a string, here. no need for a custom converter for that. — njzk2, Oct 19 '15 at 13:54

score 1 · Answer 3 · answered Oct 19 '15 at 13:50

For starters, your json is not json that Gson understand, so you can't use the Gson converter in Retrofit directly. So

public Observable<APIData> getdata ();

must be

public Observable<Response> getdata ();

Then, you have

adapter.create(WarehouseAPI.class).getdata()
   .subscribeOn(Schedulers.newThread())
   .observeOn(AndroidSchedulers.mainThread())
   .map((Response res) -> new BufferedStreamReader(res.body.body().bytesStream()))
   .flatMap(stream -> {
       return Observable.create(subscriber -> {
           String line;
           while((line = stream.readLine() != null) {
               subscriber.onNext(gson.fromJson(line, APIData.class));
           }
           subscriber.onCompleted();
       });
   });

You can subscribe to that and receive each individual item of your json list.

This is not tested, and error cases are not handled (and validity of subscription is not tested, for shortness).

It should mostly be in the right direction though.