Nodejs Async db.end() called too soon

Question

I have a function that uses a for loop to insert data in database. But db.end() is being called before the loop is finished executing. I've also tried to convert db.query into a promise using the util library, and same thing happens. What do i need to change to ensure that .end() doesn't get called until the for loop is finished?

async function processData(err, data) {
  if (err) {
    throw err;
  }
  let db = mysql.createConnection({ ... });

  let statement = "INSERT INTO Kijiji SET ?";
  db.connect();
  for (let i = 0; i < data.length; i++) {
    let row = data[i];
    scrape(row.url).then(adInfo => {
      //console.log(adInfo);
      let columns = {
        uuid: adInfo.adId
      };
      db.query(statement, columns, function(error) {
        if (error) throw error;
      });
    });
  }
  db.end();
}

Cannot enqueue Query after invoking quit.

Does this answer your question? [Asynchronous Process inside a javascript for loop](https://stackoverflow.com/questions/11488014/asynchronous-process-inside-a-javascript-for-loop) — Alexander Staroselsky, Jan 27 '20 at 17:57
@AlexanderStaroselsky Yes, you can. It just won't be run in parallel. — zero298, Jan 27 '20 at 17:57
Adding `await` before `scrape` makes the program much much slower. — Telenoobies, Jan 27 '20 at 17:59
@Alexander, I've been told not to use `ForEach` with asynchronous process... — Telenoobies, Jan 27 '20 at 18:05
It depends on the your needs, but I retract by previous statement. — Alexander Staroselsky, Jan 27 '20 at 18:07

zero298 · Answer 1 · 2020-01-27T18:01:33.143

0

You aren't awaiting anything. Should be something like this:

async function processData(err, data) {
  if (err) {
    throw err;
  }
  let db = mysql.createConnection({ ... });

  let statement = "INSERT INTO Kijiji SET ?";
  await db.connect();
  for (let i = 0; i < data.length; i++) {
    let row = data[i];
    const adInfo = await scrape(row.url);
    
    let columns = {
      uuid: adInfo.adId
    };
    
    await db.query(statement, columns)
  }
  db.end();
}

Or try it in parallel:

async function processData(err, data) {
  if (err) {
    throw err;
  }
  let db = mysql.createConnection({ ... });

  let statement = "INSERT INTO Kijiji SET ?";
  await db.connect();
  
  await Promise.all(data.map(async row => {
    const adInfo = await scrape(row.url);
    
    let columns = {
      uuid: adInfo.adId
    };
    
    await db.query(statement, columns)
  }));
  
  db.end();
}

edited Jan 27 '20 at 18:01

answered Jan 27 '20 at 17:55

zero298

25,467
10
75
100

Putting `await` infront of the `scape` makes the code extremely slow... – Telenoobies Jan 27 '20 at 17:58
@Telenoobies Well, yes. Your code isn't written to be run in parallel and it's slow now because it actually waits to do the operations before closing the DB. Your alternative is to put it in a `Promise.all`. – zero298 Jan 27 '20 at 17:59
@Telenoobies try with the parallel version. – zero298 Jan 27 '20 at 18:02
Using `promise.all`? Looking into it right now... Thanks zero – Telenoobies Jan 27 '20 at 18:03
VsCode warns that using `await query()` is unnecessary. And there is no error handling for `query()` either. – Telenoobies Jan 27 '20 at 18:09

Nodejs Async db.end() called too soon

1 Answers1