How do you loop through nested arrays with async operations?

Question

How would you optimizely loop through the items and nestedItems of each item? The target is to download and store images and PDFs in file system to make them available without internet connection.

Let's assume I have the following data structure:

const items = [
  {
    link: 'example.com/image1.png',
    nestedItems: [
      {
        nestedItemLink: 'example.com/pdf11.pdf',
      },
      {
        nestedItemLink: 'example.com/pdf12.pdf',
      },
    ]
  },
  {
    link: 'example.com/image2.png',
    nestedItems: [
      {
        nestedItemLink: 'example.com/pdf21.pdf',
      },
      {
        nestedItemLink: 'example.com/pdf22.pdf',
      },
    ]
  }
]

How I loop through the higher order array:

const promises = items.map(async (item) => {
  if (!item.link) {
    return item;
  }

  const {
    uri
  } = await FileSystem.downloadAsync(
    item.link,
    `${FileSystem.documentDirectory}${item.link
          .split('/')
          .pop()}`,
  );

  item.link = uri;

  return item;
})

const data = await Promise.all(promises);

Don't know what is better. Looking for some advise. I receive ```items``` from the server and want to modify it before storing to the redux store. — Maksym Bezruchko, Dec 15 '20 at 13:42
I think downloads should run in parallel in order to complete faster. I don't expect the file size to be huge — Maksym Bezruchko, Dec 15 '20 at 14:16

score 1 · Answer 1 · answered Dec 15 '20 at 14:59

There's a few things to consider:

the number of items in your list. You don't want to execute an unbounded number of promises in parallel and crash your node with high CPU/memory, or hammer your disk due to a very high volume of concurrent downloads. If this is a concern, check out the answers here for ideas on limiting promise concurrency: What is the best way to limit concurrency when using ES6's Promise.all()?. Or if optimal speed is not a concern, you could just download them serially.
how large the files are, and whether you could run out of disk space
failure scenarios. How do you expect the system to behave if one of the downloads fails - would you like the operation to fail fast, or perform as many downloads as it can? If you want to get results from all promises, then you could use Promise.allSettled as explained here: Wait until all promises complete even if some rejected

async function downloadItemLinks() {
  if (!item.link) {
    return item;
  }

  const {
    uri
  } = await FileSystem.downloadAsync(
    item.link,
    `${FileSystem.documentDirectory}${item.link
          .split('/')
          .pop()}`,
  );

  item.link = uri;

  if (!item.nestedItems || !item.nestedItems.length) {
    return item;
  }

  // This will download all nested items serially
  for (const nested of item.nestedItems) {
    await <your download promise>
  }

  return item;
}

// This will download all links in parallel, but for each link its nested links will be downloaded serially.  Depending on the volume of links, the concurrency may still be too high
const promises = items.map(async (item) => {
  return await downloadItemLinks(item);
});

const data = Promise.allSettled(promises);

How do you loop through nested arrays with async operations?

1 Answers1