Multi threaded web requests performance issue

Question

I want to use multi threaded web requests but i am not getting the performance I expect.

i am running with computer with 4 cores, with 1 Gbps of upload and 1 Gbps of download.

For the test i am downloading the google main page.

the size of the data from the google page is ~50 KB.

I am sending 1000 requests simultaneously and I except to complete all requests within ~2 seconds but it takes more than 20 seconds to complete

my code looks like this:

        bool success = ThreadPool.SetMinThreads(workerThreads: 2000, completionPortThreads: 1000);
        success = ThreadPool.SetMaxThreads(2000, 2000);
        DateTime dt = DateTime.UtcNow;
        Parallel.For(0, 1000, (num) =>
        {
            string url = "https://www.google.co.il/?gfe_rd=cr&dcr=0&ei=OZy3WcmoMY7b8Affj4F4&gws_rd=ssl";
            using (WebClient web = new WebClient())
            {
                byte[] bytes = web.DownloadData(url);
            }
        }
        );
        double sec = (DateTime.UtcNow - dt).TotalSeconds;
        Console.WriteLine(sec);

https://stackoverflow.com/questions/1361771/max-number-of-concurrent-httpwebrequests — Kevin Gosse, Sep 12 '17 at 09:18
Also, note that `Parallel.For` will only use one thread per core (so in your case, four threads). This is not the right tool to use for I/O bound operations — Kevin Gosse, Sep 12 '17 at 09:20
Also, the ServicePiontManager by default limits you to 2 concurrent requests for any host. You'll need to change the default connection limit, or look into adding a connectionManagement section to your app.config file. — Jim Mischel, Sep 12 '17 at 21:58

score 0 · Answer 1 · answered Sep 12 '17 at 10:02

I think the better choice in your case is using HttpClient with explicit asynchronous operations and let windows manage thread pool itself.

Stopwatch sw = new Stopwatch();
sw.Start();

for (int i = 0; i < 1000; i++)
{
    var httpClient = new HttpClient();
    results[i] = httpClient.GetByteArrayAsync(@"https://www.google.co.il/?gfe_rd=cr&dcr=0&ei=OZy3WcmoMY7b8Affj4F4&gws_rd=ssl");
}

var status = Task.WhenAll(results); //WhenAny if you can process results independently
var pages = status.Result;
sw.Stop();
double sec = sw.Elapsed.TotalSeconds;
Console.WriteLine(sec);

Side note: Please consider using Stopwatch for time measurement.

score -1 · Answer 2 · answered Sep 12 '17 at 11:48

-1

the solution is that:

        System.Net.ServicePointManager.DefaultConnectionLimit = int.MaxValue;

answered Sep 12 '17 at 11:48

asaf

958
1
16
38

4

This answer is confusing and lacks context. You should add some description – Sandy Gifford Sep 12 '17 at 14:30

Multi threaded web requests performance issue

2 Answers2