-1

I am using Parallel.ForEach to call a rest api service in batch mode like 1000 request at a time. My MaxDegreeOfParallelism is set to 1000 , But it seems system only creating 10-15 request at a time althoug system CPU utilization is very normal (15%)

var maxDegree = new ParallelOptions() { MaxDegreeOfParallelism = MaxDegreeOfParallelism };
        Parallel.ForEach(parsedLines, maxDegree, obj =>
        {


            processIndividualRecord(obj);

            var currentCount = Interlocked.Increment(ref paraaleCounter);
            if (currentCount % 100 == 0)
            {
                logger.Debug("Reaming records to process is:" + (parsedLines.Count - currentCount));
            }
        });

Is there any way I can make 1000 request at a time to avoid large wait time in non blocking mode.

tajinder singh
  • 113
  • 1
  • 10
  • 1
    What sane web developer would want you to call their api 1000 at a time ? Additionally the TPL `ForEach` and `For` methods are totally and completely unsuited to this job, you are essentially making a very inefficient threadpool thread blocking tool, as such the task scheduler will take a dim view of these tasks and severely limited your IO work – TheGeneral Apr 03 '20 at 23:34
  • 1
    You need to research the async and await pattern, why its scalable, and how it benefits IO work – TheGeneral Apr 03 '20 at 23:46

1 Answers1

0

You are running short of ThreadPool threads. A quick workaround is to increase the initial pool of workers by using the ThreadPool.SetMinThreads method.

ThreadPool.SetMinThreads(1000, 10);

A better approach is to call the REST API asynchronously. This way the need for worker threads while the requests are in flight will be removed. An excellent tool for coordinating the asynchronous process is the TPL Dataflow library. You can see a usage example of this library here.

Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
  • 1
    Using `Parallel` / TPL for async http (IO) requests is likely overkill. There is no need to spin up those `Task`s. – mjwills Apr 04 '20 at 00:10