Nodejs and Mongoose: Await stops loop

Question

I am parsing a CSV and process each record to insert it into my MongoDB using Mongoose. I already get an array out of the CSV correctly but when I start iterating over it using forEach (and async/await) it just stops there.

Here is my code:

const csv = require('csv-parser');
const fs = require('fs');
const Customer = require('../../app/models/Customers');

const projects = [];

const processRecords = async () => {
  try {
    const usableProjects = projects.filter((project) => project['cust_number customer']);
    const customerNames = [...new Set(usableProjects.map((item) => item['Parent Name']))];
    await customerNames.forEach(async (customerName) => {
      console.log('Point 1');
      const existingCustomer = await Customer.find({Name: customerName});
      console.log('Point 2'); //<======= THIS CODE IS NEVER REACHED
      if (existingCustomer.length > 0) {
        console.log(`${customerName} already exists. Skipping...`);
        return;
      }
      const customerRecord = usableProjects.find((project) => project['Parent Name'] === customerName);
      const newCustomer = {
        Name: customerName,
        Source: 'CSV',
        Initials: customerRecord.Initials,
        Code: customerRecord.Codename,
      };
      const newCustomerRecord = await Customer.create(newCustomer);
      if (newCustomerRecord) {
        console.log(`Customer ${newCustomerRecord._id} created`);
      }
    });
  } catch (err) {
    console.log(err);
  }
};

fs.createReadStream('customer_table.csv')
  .pipe(csv())
  .on('data', async (data) => projects.push(data))
  .on('end', async () => {
    processRecords();
  });

And this is the output:

Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1
Point 1

I know this might have something to do with synchronous/asynchronous code not being handled corrected by me. But I've not been able to fix it. Thanks in advance.

array forEach doesn't return a Promise, so `await`ing it doesn't do what you want - easiest and neatest solution is to use a regular for loop instead of forEach - however, this may not explain why `Customer.find` never resolves - but it's a step in the right direction — Jaromanda X, Dec 28 '19 at 01:12
Does this answer your question? [Using async/await with a forEach loop](https://stackoverflow.com/questions/37576685/using-async-await-with-a-foreach-loop) — Get Off My Lawn, Dec 28 '19 at 02:15

score 3 · Accepted Answer · edited Jan 10 '20 at 05:59

First, let's put try-catch on a Customer.find() (I'll simplify your code for clarity. And I'll replace Customer.find() with f())

async function success() { return "Hurrah!" }
async function failure() { throw new Error("Oops!") }

const customerNames = [1, 2, 3]

const processRecords1 = async (f) => {
    try {
        await customerNames.forEach(async (customerName) => {
            try {
                console.log('Point 1');
                const existingCustomer = await f()
                console.log('Point 2', existingCustomer);
                // ...
            } catch (err) {
                console.log('Point 3', err);
            }
        });
    } catch (err) {
        console.log('Point 4', err);
    }
};
setTimeout(() => processRecords1(success), 0);

Output:

Point 1
Point 1
Point 1
Point 2 Hurrah!
Point 2 Hurrah!
Point 2 Hurrah!

As you can see, if f = success the 'Point 2' is reached. So that is the first problem: your Customer.find() fails and you do not see the exception. Let's try to throw from f(), just to prove the point...

setTimeout(() => processRecords1(failure), 100);

Output:

Point 1
Point 1
Point 1
Point 3 Error: Oops!
Point 3 Error: Oops!
Point 3 Error: Oops!

Yes, if f() fails we never get to 'Point 2'. But now we do see an error, in 'Point 3'. So we can stop here.

But let's try and catch exception on a top level of processRecords() and get to the 'Point 4'. As it was already mentioned forEach() does not return a value. Let's try map()

const processRecords2 = async (f) => {
    try {
        await customerNames.map(async (customerName) => {
            console.log('Point 1');
            const existingCustomer = await f()
            console.log('Point 2', existingCustomer);
            // ...
        });
    } catch (err) {
        console.log("Point 4", err);
    }
};
setTimeout(() => processRecords2(failure), 200);

Output:

Point 1
Point 1
Point 1
Uncaught (in promise) Error: Oops!
Uncaught (in promise) Error: Oops!
Uncaught (in promise) Error: Oops!

No luck. It's because map() do return a value, but it is an Array, not a Promise. You can not await for an array of Promise-s, but you can use Promise.all() https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

const processRecords3 = async (f) => {
    try {
        await Promise.all(customerNames.map(async (customerName) => {
            console.log('Point 1');
            const existingCustomer = await f()
            console.log('Point 2', existingCustomer);
            // ...
        }));
    } catch (err) {
        console.log("Point 4", err);
    }
};
setTimeout(() => processRecords3(failure), 300);

Output:

Point 1
Point 1
Point 1
Point 4 Error: Oops!
Point 4 Error: Oops!
Point 4 Error: Oops!

That it. Replace await customerNames.forEach(...) with await Promise.all(customerNames.map(...) and you good to go.

score 1 · Answer 2 · answered Dec 28 '19 at 01:30

1

I agree with @JaromandaX, forEach is not promise-aware and can not support async/await. Use map instead.

answered Dec 28 '19 at 01:30

Kevin N.

163
5

score 1 · Answer 3 · answered Dec 28 '19 at 01:45

Array forEach doesn't return anything, and thus, you can't await them. If you want to await an array of promises, map is the way to go. However, note that you should combine that with Promise.all

async function example(arr) {
  await Promise.all(
    // Assume item.fetch() returns a promise
    arr.map(item => item.fetch())
  );
}

Now, this will run all of item.fetch calls in parallel. This is the preferred way in most cases. However if you want to run the item.fetch calls in serial, you should use a for loop instead.

async function example(arr) {
  for (let item of arr) {
    await item.fetch();
  }
}

Nodejs and Mongoose: Await stops loop

3 Answers3