learnyounode - juggling async - why doesnt my solution work

Question

Can someone explain why the following does not work as a solution to the "juggling async" lesson in the learnyounode workshop?

FYI I have left my log lines in if that helps. I am struggling to see any fundamental differences between my solution and an answer I found online here: https://github.com/olizilla/nodeschooling/blob/master/learnyounode-answers/09-juggling-async.js

Thanks in advance!

var urlList = process.argv.slice(2, process.argv.length);
//console.log(urlList);
var urlResponseList = {};

for(var i=0; i < urlList.length; i++)
{
    urlResponseList[urlList[i]] = '';
}

var http = require('http');


console.log(urlList);

for(var i = 0; i < urlList.length; i++) {
    //console.log(i);

    var url = urlList[i];

    console.log("1 " + url);

    http.get(url, function (response) {
        console.log("2 " + url);
        console.log("3 " + i);

        response.setEncoding('utf8');

        response.on('data', function (data) {
            urlResponseList[url] = urlResponseList[url] + data;
        });

        response.on('end', function () {
            //console.log(stringResponse);
            //console.log(url);
          });
    });
}
console.log(urlResponseList);

for(var i=0; i < urlList.length; i++){
    console.log(urlResponseList[urlList[i]]);
}

I also have a question about a solution I found posted online here: https://github.com/olizilla/nodeschooling/blob/master/learnyounode-answers/09-juggling-async.js

urls.forEach(function (item, index) {
  http.get(item, function (req) {
    req.setEncoding('utf8')
    req.pipe(concat(function (res) {
      data[index] = res;
      responseCount++
      if (responseCount === urls.length) {
        console.log(data.join('\n'));
      }
    }))
  })

if http.get is async and can the "index" variable be trusted in the http.get callback even though it is being set outside of the callback (in the foreach loop)?

I just wanted to post the updated solution below. Thanks for all the help. My issue was I didn't fully understand how closures worked.

var urlList = process.argv.slice(2, process.argv.length);
//console.log(urlList);
var urlResponseList = [];

for(var i=0; i < urlList.length; i++)
{
    urlResponseList.push('');
}

var http = require('http');

var responseCount = 0;

//console.log(urlList);

urlList.forEach(function(item, index){
    //console.log(i);
    var url = urlList[index];

    //console.log("1 " + url);

    http.get(item, function (response) {
        //console.log("2 " + url);
        //console.log("3 " + i);

        response.setEncoding('utf8');

        response.on('data', function (data) {
            urlResponseList[index] = urlResponseList[index] + data;
        });

        response.on('end', function () {
            responseCount++;
            if(responseCount == urlList.length)
            {
                //console.log("help");
                console.log(urlResponseList.join('\n'));
            }
          });
    });
});

//console.log(urlResponseList);

Answer 1

http.get() returns its result asynchronously (e.g. sometime later) in the callback. Your code execution does not wait for that result - the rest of your code keeps running even though the response is not yet here.

Thus, all requests will be sent at once from your first for loop, then your second for loop will run and THEN sometime later, the responses will arrive and the callbacks will be called. Your responses are going to arrive async and thus will NOT be available at the end of the first for loop. So, your first console.log(urlResponseList); will be empty and the second for loop will have nothing to do.

Async responses MUST be processed inside the callback in which they are delivered or they must be stored somewhere until the last response is done and then all responses can be processed at that time.

If you don't understand the issues behind asynchronous responses, then read these two references:

How do I return the response from an asynchronous call?

Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference

---

You also have issues with local variables declared before the callback that will have changed before the callback is called.

Here's an example of how you solve the for loop issue: JavaScript closure inside loops – simple practical example