async parallel calls using async js llin

Question

arrayOfFunctions is an array with 1000+ functions in it. Each function is making an async call to AWS S3 to get an object's metadata.

Function:

function (cb) {$.when(getMetadata("bucketname","objectkey"))
                    .done(function (data) {
                        cb(null, data)
                    });

For a single function, it takes less than a second to read the metadata, if I execute 1000s of them one by one that will take 1 second per function.

How I call with async.parallel:

    async.parallel(arrayOfFunctions, function (err, result) {
            console.log(result);
        }
    });

If I use async.parallel, does it mean that all the functions are executed at once and parallelly? should I get results of all of them at once and just 1-2 seconds?

thanks

score 0 · Answer 1 · edited May 23 '17 at 12:16

Not quite. There's a limit on how many connections you can open at the same time. For example the web browsers place a limit on this, each browser having a different limit.

Usually, with a small # of tasks, you wouldn't hit this limit and perceive. But having 1000s of tasks trying to fetch from AWS you probably won't be having the 1-2s execution time you are expecting.

Consider also the bandwidth limit of trying to download 1000s of images at the same time, how fast your server's internet connection would have to be to download it all in 1-2s?

For further reference see this stackoverflow discussion

async parallel calls using async js llin

1 Answers1