How to use Observables for pagination at REST-API endpoint?

Question

I'm using Angular 7 and building a webapp retrieving data from a REST-API endpoint. Users put their search string into an common formControl field which is observed. If the value changes in the input field a http request will send to the API endpoint. Finally I get the http response as an observable that I can subscribe. But the query result can also have more than 100 items. Then the API endpoint send a NEXT link to the second page and so on (pagination at API endpoint).

My problem is right now, I don't find the right way to observe the search input field AND interate through all NEXT pages I get from the API endpoint. Separately it works like a charm.

Does anybody have a good practice for my use case?

My component file:

export class GeomapComponent implements OnInit {
  searchTerm = new FormControl();
  constructor(private geomapService: GeomapService) { }

  ngOnInit() {
    this.searchTerm.valueChanges.pipe(
      debounceTime(400),
      distinctUntilChanged(),
      switchMap(term => this.geomapService.retrieveApi(term))
    ).subscribe(data => {
      if (data['results']) {
       ......
      }
    }
  }
}

My service file:

export class GeomapService {
  retrieveUrl = new Subject<string>();

  constructor(private http: HttpClient) { }

  public retrieveApi(searchTerm: string): Observable<ApiDataResponse> {
    this.retrieveUrl.next(baseApiUrl + '?q=' + searchTerm);
    return this.retrieveUrl.pipe(
      switchMap(url => this.sendRequest(url)),
    ).subscribe(data => {
      if (data['next']) {
        this.retrieveUrl.next(data['next']);
      }
    });
  }

  public sendRequest(url: string, httpOptions = {}): Observable<ApiDataResponse> {
    return this.http.get<ApiDataResponse>(url);
  }

}

Unfortunately I get the following error:

TS2322: Type 'Subscription' is not assignable to type 'Observable'. Property '_isScalar' is missing in type 'Subscription'.

UPDATE: Right now I am a bit further because I realized that I have to merge / concart sequential incoming observables in the service (provided by a loop).

  public requestApi(requestUrl: string): Observable<ApiDataResponse> {
    return this.sendRequest(requestUrl).pipe(
      mergeMap(result => {
        if (result['next']) {
          return this.requestApi(result['next']);
        } else {
          return of(result);
        }
      })
    );
  }

Nevertheless I still hanging with the right combination / transformation operator, because in this stage with mergeMap I get on subscription just the result of the last response. But I also want to get all responses received before and merged as one value.

Does anybody have an idea how should the returns in the operator should look like?

UPDATE: Code update, removed array issue (ApiDataResponse[] --> ApiDataResponse)

In your code `data` is of type `ApiDataResponse[]`, right?! How come you have the properties `data['next']` and `data['result']` on an array? — frido, Jan 12 '19 at 20:15
@fridoo: `data['next']` and `data['result']` are values of the API response. — Mike42, Jan 12 '19 at 20:49
yes, I thought so. But you always return / switchMap / mergeMap to an`Observable`. So your `data` or `result` variable will be of type `ApiDataResponse[]`. <-- note that's an array. — frido, Jan 12 '19 at 21:06
@fridoo: Your are right with the mentioned array issue. I fixed it. Thanks. However this was not the solution for my problem (notice for anybody else). — Mike42, Jan 13 '19 at 09:54

score 1 · Answer 1 · answered Jan 12 '19 at 19:03

So, if I understand well, you have an observable on the search field, and when you search, then you call the API for results. If results are > 100, then your API sends the first 100 results and tell you to do another request for the next 100, until there's no more results.

To me it's a little weird that you need to get ALL results at once, isn't the point to send the 100 first results to wait for the user to request the 100 next ones (for instance, by clicking on a "next page button" or by scrolling to a certain limit) ?

I would solve that problem by storing the initial first 100 results in the component which displays them (probably in an BehaviourSubject), and when it's time to get the next part of results (for instance, when scroll position reach a certain amount or when the user click on a next button), I would then request the next results if there are any...

Thanks for your idea. I had the same idea as well but I need all data at once to visualize it as a whole. — Mike42, Jan 12 '19 at 21:04

frido · Answer 2 · 2019-08-20T16:41:25.453

0

I agree with @LaetitiaDallinge on whether it's really necessary or appropriate to request all data at once. You should generally try to minimize the amount of http requests you make and only get the data when it's really needed.

If you have to get all the data at once, you can use expand:

import { EMPTY } from 'rxjs';
import { expand, reduce, map } from 'rxjs/operators';

// I presumed that you get an object of this type back from your API.
interface ApiDataResponse {
  next: string,
  results: Things[]
}

public fetchAllData(searchTerm: string): Observable<Things[]> {
  return this.http.get<ApiDataResponse>(baseApiUrl + '?q=' + searchTerm)
    .pipe( 
      // we recursively call the GET requests until there is no 'next' url
      expand(apiResponse => apiResponse.next
        ? this.http.get<ApiDataResponse>(apiResponse.next)
        : EMPTY;
      }),
      // we map the api response to the data we actually want to return
      map(apiResponse => apiResponse.results),
      // we reduce the data of all GET requests to a single array
      reduce((accData, data) => accData.concat(data), [])
    )
}

edited Aug 20 '19 at 16:41

answered Jan 12 '19 at 22:35

frido

13,065
5
42
56

Thanks for your response. You are absolutely right, the issue with the array was just a mistake on my side with the catchError operator which returned a empty array. I fixed it already. Regarding your code I get an error with the `empty()` return statement (TS2349: Cannot invoke an expression whose type lacks a call signature. Type 'Observer' has no compatible call signatures.) – Mike42 Jan 13 '19 at 09:50
After a little bit research I found out that in RxJS 6 I have to use `EMPTY` instead of `empty()` (Source: https://stackoverflow.com/questions/38548407/how-to-return-an-empty-observable-in-rxjs) With this your code works pretty well. THANKS! Finally I will modify it for my use case and publish an update to my code as soon as possible. – Mike42 Jan 13 '19 at 14:58
@Mike42 Ok, I just read that `empty()` is deprecated in favor of the constant `EMPTY` in RxJS 6. I updated my code though `empty()` still works me in RxJS 6. I also added a `map` function to the pipe and modified my `reduce` function accordingly, as I found this to be necessary. Glad I could help. If my answer solved your problem please accept and/or upvote this answer. Thanks. – frido Jan 13 '19 at 17:46

score -2 · Answer 3 · answered Jan 12 '19 at 14:16

-2

You cannot return subscription. Either use switchMap to return observable or subsciebe

answered Jan 12 '19 at 14:16

Derviş Kayımbaşıoğlu

28,492
4
50
72

Yeah, I noticed. But what could be a solution in this case, working with two observables? – Mike42 Jan 12 '19 at 14:55

How to use Observables for pagination at REST-API endpoint?

3 Answers3