0

I have a structure of nested Parallel.For and PLINQ statements in my small console app that basically performs network-bound operation(performing http requests) like following:

a list of users is filled from DB where then I do following:

Parallel.For(0,users.count(), index=>{

// here I try to perform HTTP requests for multiple users

});

Then inside this for loop I perform a plinq statement for fetching this user's info via HTTP requests.

So that now in total I get two nested loops like following:

Parallel.For(0,users.count(), index=>{

// Some stuff is done before the PLINQ statement is called... 
newFilteredList.AsParallel().WithDegreeOfParallelism(60).ForAll(qqmethod =>
    {
        var xdocic = new XmlDocument();
        xdocic.LoadXml(SendXMLRequestToEbay(null, null, qqmethod.ItemID, true, TotalDaysSinceLastUpdate.ToString(), null));
        int TotalPages = 0;
        if (xdocic.GetElementsByTagName("TotalNumberOfPages").Item(0) != null)
        {
            TotalPages = Convert.ToInt32(xdocic.GetElementsByTagName("TotalNumberOfPages").Item(0).InnerText);
        }
        if (TotalPages > 1)
        {
            for (int i = 1; i < TotalPages + 1; i++)
            {
                Products.Add(SendXMLRequestToEbay(null, null, qqmethod.ItemID, false, TotalDaysSinceLastUpdate.ToString(), i.ToString()));
            }
        }
        else
        {
            Products.Add(SendXMLRequestToEbay(null, null, qqmethod.ItemID, false, TotalDaysSinceLastUpdate.ToString(), "1"));
        }
    });
});

I tried using the outer for loop just as a regular one ,and I noticed that it was performing much faster and better than like this.

What worries me here mostly is that I was checking the utilization of CPU when running the console app like this, it's always nearby 0.5-3% of total CPU power...

So the way I'm trying to perform HTTP requests is like this:

15 users at a time * amount of HTTP requests for those 15 users.

What am I doing wrong here?

Abion47
  • 22,211
  • 4
  • 65
  • 88
User987
  • 3,663
  • 15
  • 54
  • 115
  • 1
    The Parallel classes are not for non CPU bound work. HTTP requests are I/O bound. Also what is the type of `Products`, if it is just a `List` that class is not thread safe and you are breaking the class while you write to it with multiple threads concurrently. – Scott Chamberlain Jan 31 '17 at 22:14
  • @ScottChamberlain What would you suggest me that I replace the first outer for loop with then ? I'd like to be able to process 15 and send 15 simultaneous http requests for those users, how could I do that ? – User987 Jan 31 '17 at 22:15
  • @ScottChamberlain the Products is concurrent bag =) – User987 Jan 31 '17 at 22:16
  • 2
    Use [Dataflow](https://msdn.microsoft.com/en-us/library/hh228603%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396) instead, you can have two blocks in a chain, one for sending the request, one for processing, you can set the max degree of parallelism for each block. – Scott Chamberlain Jan 31 '17 at 22:16
  • @ScottChamberlain could you help me out a bit with this ? How would I write a practical example for this particular case of mine ? – User987 Jan 31 '17 at 22:17
  • 1
    See the 2nd half of [this old answer of mine](http://stackoverflow.com/questions/38077791/can-not-use-await-inside-a-parallel-foreach-error-the-await-operator-can/38078120#38078120), it has two blocks, one that gets records in parallel then one that processes those records one at a time to add it to a list. you could transform the 2nd block to do your "processing" you need and set it to a higher degree of parallel than 1. – Scott Chamberlain Jan 31 '17 at 22:20
  • Side note: no amount of `Parallel.xxx` calls will change networking configuration... You may be more interested in http://stackoverflow.com/questions/1361771/max-number-of-concurrent-httpwebrequests – Alexei Levenkov Feb 01 '17 at 00:36

0 Answers0