Parallel.foreach() with WebRequest. 429 error

Question

I am currently running a script that is hitting an api roughly 3000 times over an extended period of time. I am using parallel.foreach() so that I can send a few requests at a time to speed up the process. I am creating two requests in each foreach iteration so I should have no more than 10 requests. My question is that I keep receiving 429 errors from the server and I spoke to someone who manages the server and they said they are seeing requests in bursts of 40+. With my current understanding of my code, I don't believe this is even possible, Can someone let me know If I am missing something here?

    public static List<Requests> GetData(List<Requests> requests)
    {
        ParallelOptions para = new ParallelOptions();
        para.MaxDegreeOfParallelism = 5;

        Parallel.ForEach(requests, para, request =>
        {
            WeatherAPI.GetResponseForDay(request);
        });

        return requests;
    }

    public static Request GetResponseForDay(Request request)
    {
        var request = WebRequest.Create(request);
        request.Timeout = 3600000;
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        StreamReader myStreamReader = new 
        StreamReader(response.GetResponseStream());
        string responseData = myStreamReader.ReadToEnd();
        response.Close();

        var request2 WebRequest.Create(requestthesecond);
        HttpWebResponse response2 = (HttpWebResponse)request2.GetResponse();
        StreamReader myStreamReader2 = new 
        StreamReader(response2.GetResponseStream());
        string responseData2 = myStreamReader2.ReadToEnd();
        response2.Close();

        DoStuffWithData(responseData, responseData2)

        return request;
    }

I think 429 comes from request count per some time unit, so maybe you don't have parallely 40+ requests, but within 1 second you are generating more. Please add some measure how long 1 request is processed or ask server manager what is the limit. Anyway this code has to be reworked if you want to respect such limit (i.e. 40 requests per second maximum) or you need to reduce parallelism. — smartobelix, Jun 21 '17 at 14:34
Those coming from search: consider the approach in the answers to this question: [.net Core Parallel.ForEach issues](https://stackoverflow.com/q/39794498). It's not exactly the same question, but [this answer](https://stackoverflow.com/a/39796934) has some good advice. — Heretic Monkey, Jun 20 '18 at 21:08

score 1 · Accepted Answer · answered Jun 21 '17 at 16:49

as smartobelix pointed out, your degreeofparallelism set to 5 doesn't prevent you of sending too many requests as defined by servers side policy. all it does is prevents you from exhausting number of threads needed for making those requests on your side. so what you need to do is communicate to servers owner, get familiar with their limits and change your code to never meet the limits.

variables involved are:

number of concurrent requests you will send (parallelism)
average time one request takes
maximum requests per unit time allowed by server

so, for example, if your average request time is 200ms and you have max degree of parallelism of 5, then you can expect to send 25 requests per second on average. if it takes 500ms per request, then you'll send only 10.

mix servers allowed numbers into that and you'll get the idea how to fine tune your numbers.

score 1 · Answer 2 · answered Jun 22 '17 at 11:56

Both answers above are essentially correct. The problem I had was that the server manager set a rate limit of 100 requests/min where there previously wasn't one and didn't inform me. As soon as I hit this limit of 100 requests/min, I started receiving 429 errors nearly instantly and started firing off requests at that speed which were also receiving 429 errors.

Parallel.foreach() with WebRequest. 429 error

2 Answers2