Get data from API and continue when done

Question

I have a problem with looping get request to apis in nodejs. I want to fetch data from multiple endpoints and continue when all request are done. I tried something like this but its running async and logging an empty array. Any tips how know for sure when all requests are ready?

var api_endpoints = { "1": "url1", "2": "url2", "3": "url3" };

var allApiSources = [];
_.each(api_endpoints, function (val, key) {
    request(val, function (error, response, body) {
        if (!error && response.statusCode == 200) {
            var data = JSON.parse(body);
            _.each(data.url, function (val, key) {
                allApiSources.push(value);
            });
        }
    });
});
console.log(allApiSources); // []

Thanks!

1. use promises or 2. use asyncjs or 3. use a counter to count how many values have been pushed to the array, and when they're all done, you'll have all the data (though, I wouldn't push to the array if the order is important - because asynchronous code is asynchronous) — Jaromanda X, Mar 02 '17 at 08:16
I cant really use a counter because the amount of values is dynamic and changes often. — Jack M., Mar 02 '17 at 08:20
you're using `_.each` ... so ... why is a counter inappropriate? — Jaromanda X, Mar 02 '17 at 08:23
Well, for that reason you can use api_endpoints.length since you don't know the exact length — Lys, Mar 02 '17 at 08:24
My bad then you gotta do smth like Object.keys(obj).length as mentioned here http://stackoverflow.com/questions/126100/how-to-efficiently-count-the-number-of-keys-properties-of-an-object-in-javascrip — Lys, Mar 02 '17 at 08:27
in `allApiSources.push(value);` ... where does `value` come from? — Jaromanda X, Mar 02 '17 at 08:29

Jaromanda X · Accepted Answer · 2017-03-02T08:39:57.477

Using promises, and Object.values

One assumption ... data.url is an Object, not an array

Another assumption is that in your original code allApiSources.push(value); should've been allApiSources.push(val);

var api_endpoints = { "1": "url1", "2": "url2", "3": "url3" };

Promise.all(Object.values(api_endpoints).map(value => new Promise((resolve, reject) => {
    request(value, function (error, response, body) {
        if (!error && response.statusCode == 200) {
            var data = JSON.parse(body);
            // you can remove the Object.values call if data.url is an Array
            return Object.values(data.url);
        }
        reject(error || response.statusCode);
    });
})))
.then(results => [].concat(...results)) // flattens the array of arrays
.then(allApiSources => {
    console.log(allApiSources);
});

Thanks this looks promising :) – Jack M. Mar 02 '17 at 08:45 — Jack M., Mar 02 '17 at 08:45
ahhh, I see what you did there :p – Jaromanda X Mar 02 '17 at 08:45 — Jaromanda X, Mar 02 '17 at 08:45

score 0 · Answer 2 · answered Mar 02 '17 at 08:35

0

You need to use the async npm, since the calls are asynchronous you will not be able to print the values, the async module lets you run multiple async calls and then maps the result to a single array

http://caolan.github.io/async/

async.map(['url1','url2','url3'], request, function(err, allApiSources) {
      console.log(allApiSources); // []

    // allApiSources is now an array of responses for each request
});

answered Mar 02 '17 at 08:35

Sharjeel Ahmed

2,051
2
17
25

wow ... async handles the response.statusCode, parses the body and extracts the values for you!! how smart is that!!! – Jaromanda X Mar 02 '17 at 08:38
hehe..also is there a way to get the map values from array/object or do I have to manually write all urls there? – Jack M. Mar 02 '17 at 08:41
async.map will accept an OBJECT according to the documentation - there's much detail left out of this answer to explain how it could work for you! – Jaromanda X Mar 02 '17 at 08:43
`Object.values({ "1": "url1", "2": "url2", "3": "url3" })` => `['url1','url2','url3']` – Jaromanda X Mar 02 '17 at 08:44

Get data from API and continue when done

2 Answers2