How do I send multiple queries from one endpoint with Express?

Question

I am trying to query my database several times and construct an object which stores every response from my database in a field. Here is my code:

router.post('/search', (req, res) => {
    var collection = db.get().collection('styles')
    var data = [];

    collection.distinct('make.name', (err, docs) => {
      data.push({'make': docs });
    });

    collection.distinct('model', (function (err, docs) {
        data.push({'model': docs });
    }))

    res.send(data);
});

Since NodeJS/Express is asynchronous, this isn't working as I would like. How can I reconstruct this endpoint to make several database calls (from the same collection) and return an object containing it?

Answer 1

There's more than one way to do it:

Nested callbacks

Without promises you could nest the callbacks:

router.post('/search', (req, res) => {
    var collection = db.get().collection('styles')
    var data = [];

    collection.distinct('make.name', (err, docs) => {
      if (err) {
        // ALWAYS HANDLE ERRORS!
      }
      data.push({'make': docs });
        collection.distinct('model', (function (err, docs) {
          if (err) {
            // ALWAYS HANDLE ERRORS!
          }
          data.push({'model': docs });
          res.send(data);
        }))
    });
});

This would be the easiest way, but note that it is not efficient if those two requests could be done in parallel.

The async module

You can use the async module:

router.post('/search', (req, res) => {
    var collection = db.get().collection('styles')
    var data = [];

    async.parallel({
      make: cb => collection.distinct('make.name', cb),
      model: cb => collection.distinct('model', cb),
    }, (err, responses) => {
      if (err) {
        // ALWAYS HANDLE ERRORS!
      }
      data.push({'make': responses.make });
      data.push({'model': responses.model });
      res.send(data);
    });
});

See: https://caolan.github.io/async/docs.html#parallel

But this may still not be the most convenient method.

ES2017 async/await

The most flexible way of doing that if you have 30 calls to make would be to:

1. Use functions that return promises instead of functions that take callbacks
2. Use async/await if you can or at least generator based coroutines
3. Await on promises (or yield promises) when the logic needs to run in sequence
4. Use Promise.all() for anything that can be done in parallel

With async/await your code could look like this:

    // in sequence:    
    var make = await collection.distinct('make.name');
    var model = await collection.distinct('model');
    // use 'make' and 'model'

Or:

    // in parallel:
    var array = await Promise.all([
      collection.distinct('make.name'),
      collection.distinct('model'),
    ]);
    // use array[0] and array[1]

A big advantage of async/await is the error handling:

try {
  var x = await asyncFunc1();
  var array = await Promise.all([asyncFunc2(x), asyncFunc3(x)]);
  var y = asyncFunc4(array);
  console.log(await asyncFunc5(y));
} catch (err) {
  // handle any error here
}

You can only use it inside of a function created with the async keyword. For more info, see:

• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await

For support in browsers, see:

• http://caniuse.com/async-functions

For support in Node, see:

• http://node.green/#ES2017-features-async-functions

In places where you don't have native support for async and await you can use Babel:

• https://babeljs.io/docs/plugins/transform-async-to-generator/

or with a slightly different syntax a generator based approach like in co or Bluebird coroutines:

• https://www.npmjs.com/package/co
• http://bluebirdjs.com/docs/api/promise.coroutine.html

See those answers for more info:

• try/catch blocks with async/await
• node.js ~ constructing chained sequence of Promise resolves
• How to run Generator Functions in Parallel?
• node.js ~ constructing chained sequence of Promise resolves
• Using async/await + Bluebird to promisifyAll
• jQuery: Return data after ajax call success

Answer 2

You can do it with Promises

router.post('/search', (req, res) => {
    var collection = db.get().collection('styles');
    // Create promise for "make.name" query
    let firstQuery = new Promise((resolve, reject) => {
        collection.distinct('make.name', (err, docs) => {
            if (!err) {
                resolve(docs);
            } else {
                reject(err);
            }
        });
    });
    // Create promise for "model" query
    let secondQuery = new Promise((resolve, reject) => {
        collection.distinct('model', (function (err, docs) {
            if (!err) {
                resolve(docs);
            } else {
                reject(err);
            }
        }))
    })
    // Run both queries at the same time and handle both resolve results or first reject
    Promise.all([firstQuery, secondQuery])
        .then((results) => {
            res.send({ "make.name": results[0], "model": results[1] });
        })
        .catch((err) => {
            // Catch error 
            res.send({});
        });
});

Also you can use destructuring in callback functions like that:

Promise.all([firstQuery, secondQuery])
    .then(([makeName, model]) => res.send({ "make.name": makeName, model }))

UPD: If you have a bunch of collection to request you can create an array of collections name, map it to promise requests and handle with Promise.all, for example

let collections = ["firstCollection", "secondCollection", "nCollection"];
let promises = collections.map((collectionName) => {
    return new Promise((resolve, reject) => {
        collection.distinct(collectionName, (err, docs) => {
            if (!err) {
                resolve(docs)
            } else {
                reject(err);
            }
        });
    })
});
Promise.all(promises)
    .then(results => {
        // Do what you want to do
    })
    .catch(error => {
        // or catch 
    });