Iterating through multiple pages of an API response in a JS promise function

Question

I have the following promise function which uses fetch to get data from an API:

const getContacts = token =>
  new Promise((resolve, reject) => {
    fetch(url, {
      method: 'GET',
      headers: {
        'Content-Type': 'application/json',
      },
    })
    .then(response => response.json())
    .then((data) => {
      resolve(data);
    })
    .catch(err => reject(err));
  });

This function is then called in a different file:

getContacts(token)
.then((data) => {
  const contacts = data.data;
  console.log(contacts);
})
.catch(err => console.error(err));

When there is a larger amount of data returned from the API, it is paginated. The response includes a link that needs to be fetched in order to get the next page. I want my code to first iterate through all pages and collect all data, then resolve the promise. When execution reaches the const contacts = data.data line, it should have data from every page (currently it returns only the first page).

What would be the best way to achieve this?

EDIT:

I tried recursion inside the getContacts function. This way I can iterate through all pages and get all data in one object, but I don't know what's the right way to resolve this back to the code, which initially called the function. The code below doesn't resolve correctly.

const getContacts = (token, allData, startFrom) =>
  new Promise((resolve, reject) => {
    if (startFrom) {
      url = `${url}?${startFrom}`; // the api returns a set of results starting at startFrom (this is an id)
    }
    fetch(url, {
      method: 'GET',
      headers: {
        'Content-Type': 'application/json',
      },
    })
    .then(response => response.json())
    .then((data) => {
      let nextPageExists = false;
      Object.assign(allData, data.data);

      data.links.forEach((link) => {
        if (link.rel === 'next') {
          nextPageExists = true;
          getContacts(token, allData, link.uri);
        }
      });
      if (!nextPageExists) {
        resolve({ data: allData });
      }
    })
    .catch(err => reject(err));
  });

Avoid the [`Promise` constructor antipattern](https://stackoverflow.com/q/23803743/1048572?What-is-the-promise-construction-antipattern-and-how-to-avoid-it)! — Bergi, Jan 18 '18 at 11:49
There is just one rel=next link among a few other link types. — rok, Jan 18 '18 at 12:15

score 7 · Accepted Answer · answered Jan 18 '18 at 12:24

First of all, do not use the new Promise constructor when fetch already returns a promise.

Then, just use a recursive approach and chain your promises with then:

function getContacts(token, allData, startFrom) {
  return fetch(startFrom ? url + '?' + startFrom : url, {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json',
    },
  }).then(response => response.json()).then(data => {
    Object.assign(allData, data.data);
    const nextPage = data.links.find(link => link.rel === 'next');
    if (!nextPage)
      return allData;
    else 
      return getContacts(token, allData, nextPage.uri);
  });
}

Thank you for correcting my approach and providing a solution. It worked perfectly. — rok, Jan 18 '18 at 13:12

score 0 · Answer 2 · answered Jun 30 '21 at 23:46

Here's a generic function using async/await syntax.

It returns itself until currentPage equals totalPages. You can retrieve these keys from your API response.

async function getData(perPage, page, options, allData = []) {

  // fetch data
  let base = 'https://api.example.com';
  let url = `${base}?perPage=${perPage}&page=${page}`;
  let response = await fetch(url, options);
  let data = await response.json();

  // push this data object (or data.data... whatever) into allData array
  allData.push(data); 

  // get 'totalPages' and 'currentPage' (or whatever your API names these)
  let totalPages = data.pagination.total_pages;
  let currentPage = data.pagination.current_page;
  if (currentPage == totalPages) {
    // you're done
    return allData;
  } else {
    // get the next page and repeat
    page++;
    return getData(perPage, page, options, allData);
  }
}

Calling it:

const options = {
  method: 'GET',
  headers: {
    Accept: 'application/json',
    appId: 'APP_ID',
    apiKey: 'APP_KEY',
    'Content-Type': 'application/json'
  }
};

let perPage = 100;
let page = 1;

getData(perPage, page, options).then((data) => {
  console.log(data)
}).catch((error) => {
  console.log(error)
})

Iterating through multiple pages of an API response in a JS promise function

2 Answers2