How to avoid net::ERR_INSUFFICIENT_RESOURCES when ngrx/data is making remote api calls?

Question

I am updating a lot of records under @ngrx/data, which calls a remote API in the background to sync the database and the local store.

  dataList.forEach((entity) => {
    const p = this.entitySvc
      .getEntityCollectionService(storeName)
      .upsert(entity)
      .toPromise();
    promises.push(p);
  });
  return Promise.all(promises);

The issue I have is that the remote API call happens outside of my code, and it happens so fast the connections overwhelm the browser with:

net::ERR_INSUFFICIENT_RESOURCES

Throttling the code above doesn't help because the remote API calls happen outside of my control.

I there a way to throttle the ngrx/data remote API calls, or another way to address this issue?

Answer 1

I would use mergeMap rxjs operator in order to add concurrency between the server requests:

updateEntities(): Observable<any> {
  return from(dataList).pipe(
    mergeMap(entity => this.entitySvc
     .getEntityCollectionService(storeName)
     .upsert(entity), 2)
  );
}

Then call the function when needed:

this.updateEntities().subscribe();

Note that mergeMap's concurrent parameter value is 2, meaning that only 2 requests will be sent in parallel, preventing multiple calls at the same time to the server. You may adjust this number as you want.

PS: Keep in mind that when you call toPromise() on an Observable it will be executed immediately.

Answer 2

Option 1

Below approach using Observable without having to convert to promise first should reduce the resource usage, give it a try

  import { forkJoin } from 'rxjs'
  ...
  myFunction() {
     return forkJoin(dataList.map(entity => this.entitySvc
      .getEntityCollectionService(storeName)
      .upsert(entity)))
  }
 

NOTE: We are returning an Observable, so where you initially called this Promise will need to change from a .then() to a subscribe()

myFunction().subscribe()

Option 2

If the above doesn't solve try below. The approach chains the Observable and only processes the next request after the previous request completes.

NOTE: This may be slower and you may need to indicate to the user the progress of the operation

    this.dataList
      .pipe(
        tap(() => 0),
        tap(({ length }) => (this.totalItems = length)),
        flatMap(item => item)
      )
      .pipe(
        concatMap(entity =>
          this.entitySvc.getEntityCollectionService(storeName).upsert(entity)
        )
      )
      .pipe(tap(() => (this.progress += 1)))
      .subscribe({
        next: console.log
      });

See this Demo