How do you fetch multiple pages of data into an array?

Question

Here is my code

let url;
let planetCount;
let planetData = [];

//turn through the pages
for (let p = 1; p < 7; p++) {
  url = `https://swapi.boom.dev/api/planets?page=${p}`;

  //fetch data
  for (let j = 0; j < 1; j++) {
    fetch(url).then(res => res.json())
      .then(data => {

        //push data to array
        for (let i = 0; i < data.results.length; i++) {
          planetData.push(data.results[i]);
        }
      })

    console.log(planetData)

  }
}

And here is the output: https://i.stack.imgur.com/Hivwr.png

Problem: How do I make it so it all goes into one array, instead of 6?

your `console.log` is still in a for loop so you display 6 times same array, but each time this array is empty — ekans, Sep 13 '21 at 14:36
You have a console.log in your first for loop that runs 6 times. I think the code looks correct otherwise :) — Wojtek322, Sep 13 '21 at 14:37

thgie · Accepted Answer · 2021-09-13T14:46:43.373

You do have only one array. You just get an output six times because you run console.log() within a loop.

I expanded your code with the proper way to handle several async calls and wait until they are all finished:

let url;
let planetCount;
let planetData = []
let promises = [];

//turn through the pages
for (let p = 1; p < 7; p++) {
url = `https://swapi.boom.dev/api/planets?page=${p}`;

//fetch data
for (let j = 0; j < 1; j++) {
    promises.push(fetch(url).then(res => res.json())
        .then(data => {

            //push data to array
            for (let i = 0; i < data.results.length; i++) {
                planetData = planetData.concat(data.results[i]);
            }

        }));
    }
}

Promise.all(promises)
.then(() => {
    console.log(planetData.length, '=>', planetData);
})

score 1 · Answer 2 · answered Sep 13 '21 at 14:49

You can use async/await and Promise.all() to return an array of the pages returned. Once we have this, we can use Array.flat() to create a contiguous array of planet data:

async function getPlanets() {
    const urls = Array.from( { length: 7 }, (v,i) => `https://swapi.boom.dev/api/planets?page=${i + 1}` );
    const promises = urls.map(url => fetch(url).then(res => res.json()).then(data => data.results));
    const planetData = (await Promise.all(promises)).flat();
    console.log(`Results for ${planetData.length} planets downloaded...`);
    console.log('Results:', planetData);
}

getPlanets()

This is extremely beautiful! I tried to stay as close as possible to the original code to not add additional reading-complexities. — thgie, Sep 13 '21 at 15:08

score 0 · Answer 3 · answered Sep 13 '21 at 14:45

0

Does this answer your question? How can I fetch an array of URLs with Promise.all?

Promise all should be the right tool for you to use so that you wait until all promises are resolved before handling responses

answered Sep 13 '21 at 14:45

David Biderman

188
8

score 0 · Answer 4 · answered Sep 13 '21 at 14:48

The issue is related to how async code works.

the fetch() function will eventually return a result, but that result is not available at the time you are trying to log it.

You must wait for the result to be retrieved. Use Aysnc/Await to accomplish that:

async function turnThroughThePges() {
  let url;
  let planetCount;
  let planetData = [];


  //turn through the pages
  for (let p = 1; p < 7; p++) {
    url = `https://swapi.boom.dev/api/planets?page=${p}`;
    //fetch data
    for (let j = 0; j < 1; j++) {
      await fetch(url).then(res => res.json())
        .then(data => {
          for (let i = 0; i < data.results.length; i++) {
            planetData.push(data.results[i]);
          }
        })
    }
  }
  return planetData;
}

turnThroughThePges().then(console.log);

Vitaliy Rayets · Answer 5 · 2021-09-13T15:33:26.613

let url;
let promises = [];

//turn through the pages
for (let p = 1; p < 7; p++) {
  url = `https://swapi.boom.dev/api/planets?page=${p}`;

  //fetch data
  promises.push(
    fetch(url)
      .then((res) => res.json())
      .then((data) => data.results)
  );
}

function handleRejection(p) {
  return p.catch((err) => ({ error: err }));
}
async function requests() {
  return await Promise.all(promises.map(handleRejection));
}
requests().then((planetData) => console.log(planetData.flat()));

How do you fetch multiple pages of data into an array?

5 Answers5