jQuery request a list of url while limit the max number of concurrent request

Question

Possible Duplicate:
Queue AJAX calls

I have a list of id:

var ids = [3738, 75995, 927, ... ]; // length is about 2000

I'd like to request the url http://xx/ + id with $.getJSON, like:

ids.forEach(function(id, index){
    $.getJSON('http://xx/' + id, function(data){
        // store the data in another array.
    });
});

However, this will make too much requests in one time, making the browser blocking for a while, so my question is, how could I limit the number of concurrent ajax request in jQuery? for example, I send 10 request and when each of them got the response I send another request.

If the requests are passed as async, it won't lock the browser. If the $.getJSON function works like the $.ajax function, then all the calls are async and you should not worry about it — Bruno Vieira, Oct 30 '12 at 17:08
@Bruno Too many asynch request can bring down the browser as well — Jasper, Oct 30 '12 at 17:08
I believe you then must break your each loop every ten times and keep track of the times you already requested until you do everything you must do. But since I don't know if this is a good approach I'll leave this as a comment — Bruno Vieira, Oct 30 '12 at 17:10
The solution should be how can I make the backend handle multiple ids. — epascarello, Oct 30 '12 at 17:11
@epascarello I can't. This is an api from another website which I have no control over. — wong2, Oct 30 '12 at 17:12
The other site is most likely going to block you/rate limit you if you make that many requests. — epascarello, Oct 30 '12 at 17:12

score 1 · Answer 1 · edited Oct 30 '12 at 17:41

1

shift() or pop() the ids off of the array as you start the requests. Start by firing off 10 requests. Then in the complete() handler for your ajax call, check for an array length. If it's greater than 0, setTimeout for a few hundred milliseconds (to free up the browser a bit) and then shift or pop off another ID and fire another request.

edited Oct 30 '12 at 17:41

KatieK

13,586
17
76
90

answered Oct 30 '12 at 17:21

Randy Hunt

435
3
10

score 1 · Answer 2 · edited May 23 '17 at 10:25

var $queue = $({});

ids.forEach(function(id, index) {
    $queue.queue("ajaxQueue", function( next ) {
        $.getJSON('http://xx/' + id, function(data){
            // store the data in another array.

            next();
        });
    });
});

$queue.queue("ajaxQueue", function() {
    // everything is saved
});

$queue.dequeue("ajaxQueue");

jQuery docs:

jQuery.queue
jQuery.dequeue

SO:

What are queues in jQuery?

Also:

The solution should be how can I make the backend handle multiple ids. – epascarello

~~##Ten request at the time:~~ Have some issues!

var $queue = $({}),
    handler;

ids.forEach(function(id, index) {
    if ( !(index % 10) && handler ) {
         $queue.queue("ajaxQueue", handler);
    }
    handler = (function(prev) {
        return function( next ) {
            prev();
            $.getJSON('http://xx/' + id, function(data){
                // store the data in another array.
            });
            if ( next ) {
                next();
            }
        }
    })(handler);
});

$queue.queue("ajaxQueue", function() {
    // everything is saved
});

$queue.dequeue("ajaxQueue");

x % y

(index % 10) => Math.floor(index/10)*10 === index;
!(index % 10) => Math.floor(index/10)*10 !== index;

Thanks, but how is this different from setting the async flag to false in jQuery? — wong2, Oct 30 '12 at 17:53
`async:false` doesn't work cross-domain & freezes the entire application while making the request. — Andreas Louv, Oct 30 '12 at 19:50

score 0 · Answer 3 · answered Oct 30 '12 at 17:14

0

This should do the trick:

var current;    

function fetchCurrentLast()
{
    if (current < ids.length)
    {
        var id = ids[current];
        current++;

        $.getJSON('http://xx/' + id, function(data){
            // store the data in another array.

            fetchCurrentLast();
        });
    }
}

current = 0;

for (var i = 0; i < 10; i++)
{
    fetchCurrentLast();
}

answered Oct 30 '12 at 17:14

Jasper

11,590
6
38
55

@mplungjan: Yep, we wrote the same code. At least, I think that's what you're saying. If not, I don't know what you are saying. – Jasper Oct 30 '12 at 17:20
But I'm doing the reccursive call in the *callback* function, which means it can only be done after the previous call is complete... – Jasper Oct 30 '12 at 17:28
But you are not! You are calling the function 10 times fast and then again each time, so at least 20 times – mplungjan Oct 30 '12 at 20:51
I am calling it ten times to start off with, so that we do 10 at the same time, as per the question. This code works on arrays of sizes below 20 just fine, because of the `(current < ids.length)` check – Jasper Oct 30 '12 at 20:58
Nope. Not convinced. I'm off to bed. Try your code for real – mplungjan Oct 30 '12 at 21:06
Let's change `for (var i = 0; i < 10; i++)` into `for (var i = 0; i < 2; i++)` and an empty array of ids, according to your logic it should go for at least 4 queries now. Let's see, for `i==0`: `current == 0`, `ids.length == 0`, `current < ids.length == false` as such, we don't go into the if of `fetchCurrentLast()` and as there is nothing outside the if, we don't do anything in that function. 0 requests sent so far. Then for `i==1` we get the exact same story, so again no requests and nothing that is going to happen later on. As far as I know, 0 requests is less than 4 requests – Jasper Oct 30 '12 at 21:22

mplungjan · Answer 4 · 2012-10-30T20:57:23.387

0

var cnt = 0;
function getEach() {
    if (cnt>=ids.length) return;
    $.getJSON('http://xx/' + ids[cnt], function(data){
    // store the data in another array.
    cnt++;
    getEach(); 
    // or setTimeout(getEach,1000); to wait a sec
    // or if (cnt%10==0) setTimeout(getEach,1000); to wait a sec every 10
    //    else getEach();
  });
}

edited Oct 30 '12 at 20:57

answered Oct 30 '12 at 17:15

mplungjan

169,008
28
173
236

The `cnt++` is in the wrong place, as with this code one can only do one request at a time, not 10 – Jasper Oct 30 '12 at 17:16
This will do all requests one at a time each one when the previous finished as asked for in the original question – mplungjan Oct 30 '12 at 17:54
From the question: `for example, I send 10 request and when each of them got the response I send another request.` – Jasper Oct 30 '12 at 17:56
MHmm I did not notice the 2000, but my code will work with or without batching – mplungjan Oct 30 '12 at 20:54
Because you update `cnt` only after the ajax request returns, when you use batching, you will be fetching the same result more than once. For example, if assuming each request takes exactly even long, you get ten times the result for `id = 0` and then ten times the result for `id=10` and so on (in reality they won't take equally long, so it becomes much more of a mess). – Jasper Oct 30 '12 at 21:02
Not true. If you set cnt to 10, it will get 10-19 so batching will work – mplungjan Oct 30 '12 at 21:03
Okay I was mistakes, only the original 10 will be getting the same request. Still, that's a pretty major flaw. – Jasper Oct 30 '12 at 21:07
I do not see the flaw. Another function is needed for the batching that isall – mplungjan Oct 30 '12 at 21:08
`getEach(); getEach()` does not work. Yes, `getEach(); cnt++; getEach(); cnt++;` does, but that's not trivial code, so in that case that code should be included in your answer. (And that's basically what I've been saying all along, except I used a different way to solve the same problem. – Jasper Oct 30 '12 at 21:13
Replace the return with a callback to the batching function and it will work. Shees this is getting too long – mplungjan Oct 31 '12 at 04:09

jQuery request a list of url while limit the max number of concurrent request

4 Answers4