No data written in Mongo database after Process.Exit()

Question

I'm writing a node.js script to write in a MongoDB database some data from a MySQL database. It is not a web application, but a local script (both my MySQL and Mongo are local).

The problem is that when all data is written, the script doesn't terminate. I have to close it with CTRL+C. But all my data is written in my MongoDB database.

I've added a "process.exit()" line at the end of the callback called after writing in MongoDB. The app terminates, but no data is written ! And I don't understand why.

Here is the relevant portion of code

(...)
var req = "SELECT * FROM geometry";

// "connection" is a MySQL connection
connection.query(req, function(err, rows, fields) {
    console.log("number of lines :" + rows.length);
    connection.end();

    if (err) {
        throw err;
    }
    // "Entry" is my model
    for (var i = 0; i < rows.length; i++) {
        Entry.create({
            "code" : rows[i].code,
            "name" : rows[i].name,
            "coords" : rows[i].coords,
            "type" : rows[i].type
        }, function(err, doc){
            if (err) {
                throw (err);
            }
        });
    }

    console.log("end");
    process.exit(); // no data written ! why ?
});

Can you help me ? Thanks in advance.

Answer 1

The problem is that you're terminating you process too early. You're not waiting for mongoose to actually send any of your documents to MongoDB. Here are some examples of how to do it properly.

N.B. Please, first read this question to understand why no data is written, then read my answer to know how to deal with it.

---

Using promises and when.js module

First we should require when module (or any other promise-based control flow module, e.g. Q, bluebird).

when = require('when');

then we could use it to simplify our code:

promises = rows.map(function(row) {
  return Entry.create(row);
})
when.all(promises).otherwise(console.error).ensure(function() {
  process.exit()
})

or using when.map:

when.map(rows, function(row) {
  return Entry.create(row);
}).otherwise(console.error).ensure(function() {
  process.exit()
})

N.B. This solution is possible dye to mongoose build-in promises support.

---

Using async.js module

async.js is a callback-based control flow module for node.js:

async = require('async');

but it is almost as powerful as promises are:

async.map(roes, function(row, next) {
  Entry.create(row, next);
}, function(err){
  if (err) throw (err);
  process.exit();
});

---

Using pure JavaScript

But we can achieve the same results using pure callbacks-based JavaScript:

var pending = 0;
rows.forEach(function(row) {
  pending++;
  Entry.create(row, function(err) {
    if (err) throw (err);
    if (--pending === 0) process.exit();
  });
});

---

Note on mongoose

Bu default mongoose uses strict mode, so I simply call

Entry.create(row);

knowing that mongoose will strip any property of row which is not in present in Entry schema.

---

Note on asynchronous code and control flow modules

Control flow modules (promises, async.js, etc) helps you deal with asynchronous code, but if you're new to node.js and asynchronous programming I would recommend you to start from pure JS approach in order to understand node.js better. Then it'll be safe to adopt some control flow solution.

You should also note that promises provide a huge abstraction layer for your asynchronous code with a lot of syntactic sugar, while callbacks-based modules like async.js provide just a little bit of it.

So, in my opinion the best way to master node.js is the following:

Pure JS -> callback-based modules (async.js) -> promises

Latest node.js 0.11.x (with --harmony flag) also support generators, providing yet another way to deal with asynchronous code.