nodejs send variable to asynchronous function and determine end of tasks

Question

I am making multiple asynchronous database queries, and am trying to store all this data in a large list, and when all the querying is finished i want to send this data as a response.

I am facing 2 problems,

How to determine the end of the asynchronous calls
How to pass a variable to an asynchronous function such that it's results would be stored in this variable that i can later on use

Function Explanation

I have an app which is being executed every k minutes in the background, and at every execution the task information is saved in a mongodb database. The function I am having problems completing is called fetchHistoryStatistics. This function is supposed to fetch the number of executions done every day.

Algo:

Fetch all distinct dates of execution (year-month-day)
Fetch all distinct accounts being executed
For every user fetch the number of exections for each day

Code:

function fetchHistoryStatistics(response){
    var dates = [], timeline = {}, query;
    //Distinct Dates
    db.collection('master')
        .distinct("date", function(err, ds){

            for(var i in ds){
                var d = ds[i].getFullYear() + " " + (ds[i].getMonth()+1) + " " + ds[i].getDate();
                if (dates.indexOf(d)==-1){
                    dates.push(d);
                }
            }

            //Distinct Users
            db.collection('master')
                .distinct("screen_id", function(err, users){
                    //Fetching History Data
                    for(var i in users){
                        timeline[users[i]]={};
                        for (var j in dates){
                            timeline[users[i]][dates[j]] = {};
                            if(dates[parseInt(j)+1] != undefined){
                                query = {"screen_id": {"$eq": users[i]}
                                    , "date": {"$gte": new Date(dates[j]), "$lt": new Date(dates[parseInt(j)+1])
                                    }};
                                db.collection('master')
                                    .find(query)
                                    .toArray(function(err, results){
                                       // **!! TODO !!** 
                                       // The problem is occuring here, when i try to store the data in the variable timeline
                                        timeline[users[i]][dates[j]] = results.length;
                                        console.log("-----------------", users[i], dates[j]);
                                        //console.log(query);
                                        //console.log(results);

                                        console.log("timeline :", timeline);
                                    })
                                ;
                            }else{
                                query = {"screen_id": {"$eq": users[i]}
                                    , "date": {"$gte": new Date(dates[j])
                                    }};

                                db.collection('master')
                                    .find(query)
                                    .toArray(function(err, results){
                                        timeline[users[i]][dates[j]] = results.length;
                                        console.log("-----------------", users[i], dates[j]);
                                        //console.log(query);
                                        //console.log(results);

                                        console.log("timeline :", timeline);
                                    })
                                ;
                            }
                        }
                    }
                    console.log("timeline :", timeline);
                })
            ;

        })
    ;


    // !! TODO !!
    // And another problem here
    // Because the functions are Asynchronous i need to be able to wait for all the executions to have terminated,
    // such that i can send all the collected data back in the response
    response.send({});

}

Checkout this answer, I think promises would make this easier to work with: http://stackoverflow.com/a/23771854/693275 — Rob M., May 27 '15 at 23:39
You lose asynchronicity when you run the FOR loops as they all execute without waiting. Have you tried using Caolan's asyn library to iterate those arrays instead? Something like async.eachSeries(dates, function(d, callback) { // process d, then callback to execute next array item })); — Brant, May 28 '15 at 02:52

score 0 · Answer 1 · answered May 27 '15 at 23:44

As my comment says, I think using bluebird to promisify the library would make this much easier to work with and reason about. If you aren't using promises (which allow you to treat asynchronous code synchronously), then the way this is typically handled is via a callback argument to your function.

function fetchHistoryStatistics(response, callback){ ...

Then once you have completed your data operations, you call the supplied callback with the final data:

//Fetching History Data
for(var i in users){
  ...
}
console.log("timeline :", timeline);
// fire callback function with timeline as argument
callback(timeline);

Your calling code would be of the form:

fetchHistoryStatistics(response, function(timeline) { ... });

nodejs send variable to asynchronous function and determine end of tasks

1 Answers1