recursively fetch paginatedData from an API using a continuation token

Question

The Sonatype REST API paginates data in its response with a continuation token placed inside the response body; if the continuation token is null, you know you're on the last page of results.

I would like to recursively fetch all records from the endpoint with fetch (unless there is a better strategy?), updating the url using a query parameter on each recursive iteration and calling the recursive fetch on it, and building a storage structure containing the complete list of records in the function. I have been unsuccessful.

To get the first page of records, this works:

const requestUrl = `https://{baseURL}/service/rest/v1/components?repository=myRepo&group=myGroup`;


// returns first 10 results
async function fetchData(url) {
  let headers = new Headers();
  headers.append("Accept", "application/json");
  headers.append(
    "Authorization",
    "Basic <BASE64-STRING-HERE>"
  );
  const requestOptions = {
    method: "GET",
    headers: headers,
    redirect: "follow",
  };
  // Fetch request and parse as JSON
  const response = await fetch(url, requestOptions);
  let data = await response.json();

  return data;
}

// called at `/api/get-data`
export default async function getData(req, res) {
  try {
    let result = await fetchData(requestUrl);
    res.status(200).json({ result });
  } catch (err) {
    res.status(500).json({ error: "failed to load data" });
  }
}

This returns an object containing an array of 10 records and the continuation token, a string, e.g.

{"items":[{"id": 1, "name": "itemOne"}, {"id": 2, "name": "itemTwo"}, etc.],"continuationToken":"07929bf0c78cf6c92d4acd0bf5823d80"}}

On the last page of records, the continuationToken is null:

{"items":[{"id": 99, "name": "itemNinetyNine"}, {"id": 100, "name": "itemOneHundred"}, etc.],"continuationToken": null}}

To get the complete set of records using recursion, this is the strategy I have tried, but haven't been successful:


async function recursivelyFetchData(url) {
  let headers = new Headers();
  headers.append("Accept", "application/json");
  headers.append(
    "Authorization",
    "Basic <BASE-64-STRING>"
  );
  const requestOptions = {
    method: "GET",
    headers: headers,
    redirect: "follow",
  };
  try {
    // Fetch request and parse as JSON
    const response = await fetch(url, requestOptions);
    let data = await response.json();
    // create a storage array
    let storage = [];
    // Set a variable for the next page url
    let nextPageUrl;
    // if the initial response contains a continuation token, start to recurse
    if (data.continuationToken) {
      // remove the query param so we don't keep adding it to the url string on each iteration
      let strippedUrl = url.replace(/&continuationToken=.*/g, "");
      // create the url with the query param
      nextPageUrl =
        strippedUrl + `&continuationToken=` + data.continuationToken;
      // store the next response
      let nextPageResponse = await recursivelyFetchData(
        nextPageUrl,
        requestOptions
      );
      // parse
      let nextPageData = await nextPageResponse.json();
      // add to storage
      storage = [...data, ...nextPageData.items];
    }
    // break recursion
    return storage;
  } catch (err) {
    res.status(500).json({ error: "failed to load data" });
  }
}

export default async function getData(req, res) {
  try {
    let result = await recursivelyFetchData(requestUrl);
    res.status(200).json({ result });
  } catch (err) {
    res.status(500).json({ error: "failed to load data in getPackages" });
  }
}

I've tried the approaches found here, here, here, here, and here, but this API is different from other paginated responses in that it relies on a continuation token in the response body, and I can't for the life of me figure out how to recursively add the query parameter to the url, continue iterating until the continuation token is null, then break recursion.

Any help is appreciated.

score 2 · Accepted Answer · answered Jun 14 '22 at 16:10

Here is what I think is a simple approach. fetchAllPages fetches the first page, and then checks to see if the result has a non-nil continuation token. If it doesn't we just returns the items from the result. But if it does, we merge that token into a new url and recur, combining the current items with the newly-fetched ones into a single array.

Relying on a helper addToken function that builds our URL, it looks like this:

const fetchAllPages = (url) => fetch (url)
  .then (res => res .json ())
  .then (
    ({items, continuationToken: token}) => token
      ? fetchAllPages (addToken (url, token)) .then (newItems => [...items, ...newItems])
      : items
  )

We alter it slightly to show where your header manipulation and such would fit, in a snippet that uses a dummy fetch function that returns item0 - item33 paginated in groups of ten and an implementation of addToken that does URL manipulation using DOM methods (rather than string parsing/hacking):

const addToken = (base, token) => {
  const url = new URL (base)
  const searchParams = new URLSearchParams (url .search)
  searchParams .set ('continuationToken', token)
  url .search = searchParams
  return url .toString ()
}

const fetchAllPages = (url) => {
  // configure fetch parameters (headers, etc.)
  return fetch (url, /*headers, whatever*/)
    .then (res => res .json ())
    .then (
      ({items, continuationToken: token}) => token 
        ? fetchAllPages (addToken (url, token)) .then (newItems => [...items, ...newItems])
        : items
    )
}

fetchAllPages ('https://baseURL.com/service/rest/v1/components?repository=myRepo&group=myGroup')
  .then (console .log)
  .catch (console .warn)

.as-console-wrapper {max-height: 100% !important; top: 0}

<script>
// dummy version of `fetch` to demo without a real server
const fetch = (() => {
  const data = Array .from ({length: 34}, (_, id) => ({id, name: `item ${id}`}))
  const pageSize = 10
  return async (s) => {
    console .log (`fetching '${s}'`)
    const url = new URL (s)
    const searchParams = new URLSearchParams (url .search)
    const start = searchParams .get ('continuationToken') || '0'
    const startIndex = data .findIndex (({id}) => id == start)
    const token = startIndex < 0 || (startIndex + pageSize >= data.length) ? null : data [startIndex + pageSize] .id
    return {json: () => ({
      items: data .slice (startIndex, startIndex + pageSize), 
      continuationToken: token ? String (token) : null
    })}
  }
})()
</script>

recursively fetch paginatedData from an API using a continuation token

1 Answers1