1

I'm running into a problem with TPL dataflow that I can't seem to figure out. My code will run anywhere from 15 minutes to a couple of hours before it just deadlocks. I've done as much digging as I can and I will try to post it here.

Also, I've read that you generally shouldn't mix asynchronous and blocking code. If I'm doing it wrong by isolating my TPL Dataflow code to a Task.Run(Func<Task<bool>>) call, let me know. Otherwise, I don't think this is the problem since it's running on its own thread.

My dataflow network is set up pretty much exactly as in this SO link, please refer to it for the code.

The TransformBlock is pretty involved with lots of various awaits. One of the functions calls await httpRequest.GetRequestStreamAsync(). This seems to be where the deadlock is occurring, because running a memory profiler reveals four compiler-generated IAsyncStateMachine structs stuck in memory (e.g. <ExampleAsync>d__0) that all represent a state immediately before it's called.

It also shows that 110 objects are stuck waiting to be processed in the TransformManyBlock (and by the next BatchedJoinBlock), and there are only 267 HTTP connections open (ServicePointManager.DefaultConnectionLimit is set to 512).

Checking the source code of the GetRequestStreamAsync function reveals that at least another thread is created with Task.Run<>(), which seems unnecessary to me. I'm thinking I could even just call the FromAsync function myself:

if (...)
{
    return Task.Run<Stream>(() => {
        TaskFactory<Stream> factory = Task<Stream>.Factory;
        WebRequest webRequest = this;
        WebRequest webRequest1 = this;
        return factory.FromAsync(
            new Func<AsyncCallback, object, IAsyncResult> (webRequest.BeginGetRequestStream), 
            new Func<IAsyncResult, Stream>(webRequest1.EndGetRequestStream), 
            null
        );
    });
}

Could it be that the thread pool is becoming exhausted?

My only other clue is that the BatchedJoinBlocks may be hogging resources, because even after each one is completed, they never get disposed -- which leads me to want to continue the BatchedJoinBlock.Completion task with one that will unlink itself upon completion. (Setting MaxNumberOfGroups to 1 didn't seem to do that.) I believe there were nearly 4000 lingering BatchedJoinBlocks.

I'm stumped. If something is timing out, an exception is caught and passed to the Exception target of the BatchedJoinBlock. Obviously, this isn't the case.

Community
  • 1
  • 1
Josh Wyant
  • 1,177
  • 1
  • 8
  • 13
  • Sounds like it could be thread pool exhaustion, or some complex interaction between thread pool limits and the block throttling. Notes: `Task.Run` only queues work (does not create a thread); and `Task.Run` is used in `GetRequestStreamAsync` because `BeginGetRequestStream` (incorrectly) blocks on DNS lookup etc. Also, do you really need to change the mesh shape based on input? I'd look at other ways of grouping your results before using a dynamic mesh. – Stephen Cleary Apr 07 '14 at 17:16
  • @StephenCleary Yeah, it's pretty set in stone. The result of the x number of child objects has to be stored in each parent object. I'd use some sort of external queue of lists or something but this seemed a lot cleaner. I really thought this would be the solution we needed :( Do you think I need to try toning down the parallelism? – Josh Wyant Apr 07 '14 at 19:39
  • @StephenCleary one more thing, if it is thread pool exhaustion, do you think another instance of the same assembly would still be able to keep running for an hour or two longer? I've added the code to unlink the `BatchedJoinBlock`s on each batch and hopefully that frees up some unused worker items. Very nice MSDN article by the way :) – Josh Wyant Apr 08 '14 at 18:21
  • The thread pool is shared between app domains, but each app domain does have a different queue of work for the thread pool. So I'm really not sure whether a different assembly would work. Since you are modifying the mesh shape at runtime, proper cleanup is probably your best bet. – Stephen Cleary Apr 08 '14 at 18:30
  • So I've done everything I can think of. The default maximum number of threads is high enough for this not to be an issue (> 1000). Not all the threads even get used, meaning the TransformBlock probably doesn't ever reach its max degree of parallelism. Setting the minimum number of threads to 300 only causes an access violation exception :/ – Josh Wyant Apr 14 '14 at 21:35
  • Try to create a minimal repro. – Stephen Cleary Apr 14 '14 at 23:57

0 Answers0