Parallel.ForEach and httpClient.GetAsync throws AggregateException (InnerException:"A task was cancelled")

Question

I am using Parallel.ForEach to execute about 2200 requests (httpClient.GetAsync) so I can get the data needed faster. While sometimes this process passes without exception often I end up with AggregateException (InnerException:"A task was cancelled").

Cancel() is not called and I am using the default request Timeout.

There are suggestions to remove the using of the HttpClient creation and use a single instance of it, but it did not help.

Any ideas what causes the exception and how to prevent it?

    public void GetData()
    {
       ConcurrentBag<MyItem> alltems = new ConcurrentBag<MyItem>();

       Parallel.For(0, listOfUrls, url =>
        {
           allItems.Add(MyMethod(this.Get(url));
        }
    }

    public HttpResponseMessage Get(string url)
    {
        httpClient = NewHttpClient();
        {
            var requestContent = httpClient.GetAsync(this.siteUrl + "/" + url).Result;
            return requestContent;
        }
    }

    HttpClient cachedHttpClient;
    private HttpClient NewHttpClient(bool skipAuthorization = false)
    {
        var h = new WebRequestHandler();

        if (cachedHttpClient == null)
        {
            var client = new HttpClient(h);
            if (!skipAuthorization)
            {
                client.DefaultRequestHeaders.Add("Authorization", "WRAP access_token=" + accessToken);
            }
            return client;
        }
        else
        {
            return cachedHttpClient;
        }
    }

Try to set `ServicePointManager.DefaultConnectionLimit = int.Max;` see https://msdn.microsoft.com/en-us/library/7af54za5(v=vs.110).aspx — , Aug 05 '16 at 13:12
Could this be an httpclient.timeout issue? http://stackoverflow.com/questions/29179848/httpclient-a-task-was-cancelled — sr28, Aug 05 '16 at 13:27
Are you passing a cancellation token in? Can we see some code? — Botonomous, Aug 05 '16 at 13:28
Is there a reason to use Parallel.For? Can you just create new tasks in a list and then waitAll them? — Botonomous, Aug 05 '16 at 13:46
Your code can't compile, you can't use `return requestContent;` in a `Action`, and [there is no `Func` overload](https://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.for(v=vs.110).aspx). If you want us to help you with the code that is not working you need to actually show us the code that is not working. — Scott Chamberlain, Aug 05 '16 at 13:56
Now your code does not make sense, you request the content but you immediately throw it way? Also, what is "new" about `NewHttpClient`, what did you change in it? That could be causing the problem. (Also you really should be disposing of it) — Scott Chamberlain, Aug 05 '16 at 14:20
@x... - I tried that DefaultConnectionLimit and for a while it seemed to work, but I am back to the errors. — checho, Aug 09 '16 at 10:34
@sr28 I had a look at this thread, I will try to enhance the timeout to see what happens. — checho, Aug 09 '16 at 10:35
@ScottChamberlain - I just made an example to show the idea initially, now I updated the code. I read some suggestions that the dispose might be causing this if I do not get the result by the time the code exits the using, so I remove it. Can you elaborate why should it get disposed? — checho, Aug 09 '16 at 10:36

Parallel.ForEach and httpClient.GetAsync throws AggregateException (InnerException:"A task was cancelled")

0 Answers0