Can't find a easy way out of multiple async for each node js (sails)

Question

So here's the deal :

I have an array of objects with a child array of objects

askedAdvices 
   askedAdvice.replayAdvices

I'm looping trough the parent and foreach looping trough the childs and need to populate() two obejcts (I'm using sails)

The child looks like :

askedAdvices = {
     replayAdvices : [{
       bookEnd : "<ID>",
       user : "<ID>"
    }]
  }

So my goal is to cycle and populate bookEnd and user with two findOne query, but I'm going mad with the callback hell. Here's the Models code :

AskedAdvices Model

module.exports = {
  schema : false,
  attributes: {
    bookStart : {
        model : 'book'
    },
    replayAdvices : {
      collection: 'replybookend'

    },
    user : {
        model : 'user',
      required : true
    },
    text : {
      type : "text"
    }
  }
};

ReplyBookEnd Model

module.exports = {
  schema : false,
  attributes: {
    bookEnd : {
        model : 'book'
    },
    user : {
        model : 'user',
        required : true
    },
    text : {
      type : "text"
    }
  }
};

Here's the Method code :

getAskedAdvices : function(req, res) { 

    var queryAskedAdvices = AskedAdvices.find()
        .populate("replayAdvices")
        .populate("user")
        .populate("bookStart")

    queryAskedAdvices.exec(function callBack(err,askedAdvices){

        if (!err) {
            askedAdvices.forEach(function(askedAdvice, i){

                askedAdvice.replayAdvices.forEach(function(reply, i){

                     async.parallel([
                        function(callback) {
                            var queryBook = Book.findOne(reply.bookEnd);
                            queryBook.exec(function callBack(err,bookEndFound) {
                                if (!err) {
                                    reply.bookEnd = bookEndFound;
                                    callback();
                                }                                   
                            })  
                        },
                        function(callback) {
                            var queryUser = User.findOne(reply.user)
                            queryUser.exec(function callBack(err,userFound){
                                if (!err) {
                                    reply.user = userFound; 
                                    callback();
                                }

                            })
                        }
                     ], function(err){
                        if (err) return next(err);

                        return res.json(200, reply);
                     })


                })  
            })



        } else {
            return res.json(401, {err:err})
        }
    })
}

I can use the async library but need suggestions

Thanks folks!

You are doing that because there is no deep populate in sails? — Bonanza, May 04 '16 at 10:54
@Asso http://stackoverflow.com/questions/36675683/sails-workaround-for-deep-populate/36679801#36679801 - try this solution. It will take few minutes to add this to Sails and can reduce this code to few lines. — Bonanza, May 04 '16 at 11:56
@Bonanza :Thats great! Thanks dude, I found one similar to taht tha was like a fork on ghitub from sails official — Asso, May 04 '16 at 12:12
@Asso : that solution saved me a lot of time. And we can avoid such [callback hell](http://callbackhell.com/) as you wrote before :) Just remember that if you deal with huge with huge amount of data it will take lots of resources. — Bonanza, May 04 '16 at 12:26
@Bonanza last question : do you suggest to replace the orm or mergin ? — Asso, May 04 '16 at 12:50
@Asso in my project i replaced it. There are some additional pros. For example Offshore is not changing $eq: 'string' to regex which is not using mongodb indexes. And caching mechanism. It solved 3 of mine problems with one update. I haven't tested it on production yet, but i hope it will work well. — Bonanza, May 04 '16 at 12:56
@Bonanza Working for deep polulate but having trouble with routes. Even a simple post to /user is returning 404 — Asso, May 05 '16 at 08:36
@Asso : I just checked that. It creates only routes to controller. I will create a bug report. So Blueprint API is not working with this ORM right now. — Bonanza, May 05 '16 at 09:14
@Bonanza Yes, I have to custom routes, even for Blueprint. It's kinda ok anyway and thanks for the advice! — Asso, May 05 '16 at 09:54

score 1 · Accepted Answer · answered May 06 '16 at 04:01

As pointed out in the comments, Waterline doesn't have deep population yet, but you can use async.auto to get out of callback hell. The trick is to gather up the IDs of all the children you need to find, find them with single queries, and then map them back onto the parents. The code would look something like below.

async.auto({

  // Get the askedAdvices 
  getAskedAdvices: function(cb) {
    queryAskedAdvices.exec(cb);
  },

  // Get the IDs of all child records we need to query.
  // Note the dependence on the `getAskedAdvices` task
  getChildIds: ['getAskedAdvices', function(cb, results) {
    // Set up an object to hold all the child IDs
    var childIds = {bookEndIds: [], userIds: []};
    // Loop through the retrieved askedAdvice objects
    _.each(results.getAskedAdvices, function(askedAdvice) {
      // Loop through the associated replayAdvice objects
      _.each(askedAdvice.replayAdvices, function(replayAdvice) {
        childIds.bookEndIds.push(replayAdvice.bookEnd);
        childIds.userIds.push(replayAdvice.user);
      });
    });
    // Get rid of duplicate IDs
    childIds.bookEndIds = _.uniq(childIds.bookEndIds);
    childIds.userIds = _.uniq(childIds.userIds);
    // Return the list of IDs
    return cb(null, childIds);
  }],

  // Get the associated book records.  Note that this task
  // relies on `getChildIds`, but will run in parallel with
  // the `getUsers` task
  getBookEnds: ['getChildIds', function(cb, results) {
    Book.find({id: results.getChildIds.bookEndIds}).exec(cb);
  }],

  getUsers: ['getChildIds', function(cb, results) {
    User.find({id: results.getChildIds.userIds}).exec(cb);
  }]

}, function allTasksDone(err, results) {

  if (err) {return res.serverError(err);

  // Index the books and users by ID for easier lookups
  var books = _.indexBy(results.getBookEnds, 'id');
  var users = _.indexBy(results.getUsers, 'id');

  // Add the book and user objects back into the `replayAdvices` objects
  _.each(results.getAskedAdvices, function(askedAdvice) {
    _.each(askedAdvice.replayAdvices, function(replayAdvice) {
      replayAdvice.bookEnd = books[replayAdvice.bookEnd];
      replayAdvice.user = users[replayAdvice.bookEnd];
    });
  });

});

Note that this is assuming Sails' built-in Lodash and Async instances; if you're using newer versions of those packages the usage of async.auto has changed slightly (the task function arguments are switched so that results comes before cb), and _.indexBy has been renamed to _.keyBy.

Thanks a lot! Finally undesrtood how async.auto works, greta explanation! — Asso, May 07 '16 at 19:46

Can't find a easy way out of multiple async for each node js (sails)

1 Answers1