How to make a synchronous delay inside a for loop?

Question

I'm writing a NodeJS script that calls a bunch of APIs via GET (using request from npm) and saves the responses in a JSON file. I'm using a for to loop through IDs to pass to the APIs, but I'm having trouble putting in a delay between call bursts so I don't spam the API server and make it mad at me (rate limiting). Does anyone know how to do this?

My current code (without any delay):

var fs       = require('fs');
var request  = require('request');

// run through the IDs
for(var i = 1; i <= 4; i++)
{
    (function(i)
    {
        callAPIs(i); // call our APIs
    })(i);
}

function callAPIs(id)
{
    // call some APIs and store the responses asynchronously, for example:
    request.get("https://example.com/api/?id=" + id, (err, response, body) =>
        {
            if (err)
                {throw err;}

            fs.writeFile("./result/" + id + '/' + id + '_example.json', body, function (err)
            {
                if (err)
                    {throw err;}
            });
        });
}

I'm looking for this behavior:

callAPIs(1); // start a burst of calls for ID# 1

// wait a bit...

callAPIs(2); // start a burst of calls for ID# 2

// wait a bit...

// etc

See this thread/duplicate: https://stackoverflow.com/questions/33289726/combination-of-async-function-await-settimeout — Jeppe, Jun 15 '19 at 19:54
How many requests are you limited to do? This could be a good case to batch groups of requests instead of doing all the requests serially... — Christian C. Salvadó, Jun 15 '19 at 19:54

Akxe · Answer 1 · 2019-06-15T19:38:29.490

In nodeJS you don't do pauses, you use it's asynchronous nature to await the result of preceding tasks, before resuming on next task.

function callAPIs(id) {
  return new Promise((resolve, reject) => {
  // call some APIs and store the responses asynchronously, for example:
    request.get("https://example.com/api/?id=" + id, (err, response, body) => {
      if (err) {
        reject(err);
      }

      fs.writeFile(`./result/${id}/${id}_example.json`, body, err => {
        if (err) {
          reject(err);
        }

        resolve();
      });
    });
  });
}

for (let i = 1; i <= 4; i++) {
  await callAPIs(array[index], index, array);
}

This code, will do request, write the file, and once it is written to disk, it will process the next file.

Waiting a fixed time before the next task is processed, what if would take a bit more time? What if you're wasting 3 seconds just to be sure it was done...?

This makes the code synchronous right? Yet it does not allow to 'limit' the rate (e.g. a delay) - it might still be blocked right? — Jeppe, Jun 15 '19 at 19:42
@Jeppe, this will make the function (or even body on newer node versions) seem synchronous, it will `await` blocks execution on that line, till it resolves/rejects, but in meantime, other things might happen. — Akxe, Jun 15 '19 at 19:44

score 3 · Accepted Answer · answered Jun 15 '19 at 19:38

You can use the new ES6's async/await

(async () => {
    for(var i = 1; i <= 4; i++)
    {
        console.log(`Calling API(${i})`)
        await callAPIs(i);
        console.log(`Done API(${i})`)
    }
})();

function callAPIs(id)
{
    return new Promise(resolve => {
        // Simulating your network request delay
        setTimeout(() => {
            // Do your network success handler function or other stuff
            return resolve(1)
        }, 2 * 1000)
    });
}

A working demo: https://runkit.com/5d054715c94464001a79259a/5d0547154028940013de9e3c

Jainil Gandhi · Answer 3 · 2019-06-16T12:20:44.767

You might also want to have a look at the async module. It consists of async.times method which will help you achieve the results you need.

var fs = require('fs');
var request = require('request');
var async = require('async');


// run through the IDs
async.times(4, (id, next) => {
    // call some APIs and store the responses asynchronously, for example:
    request.get("https://example.com/api/?id=" + id, (err, response, body) => {
        if (err) {
            next(err, null);
        } else {
            fs.writeFile("./result/" + id + '/' + id + '_example.json', body, function (err) {
                if (err) {
                    next(err, null);
                } else
                    next(null, null);
            });
        }
    });
}, (err) => {
    if (err)
        throw err
});

You can read about it from the below shared url: https://caolan.github.io/async/v3/docs.html#times

How to make a synchronous delay inside a for loop?

3 Answers3

Linked

Related