await Task.WhenAll() appears to be deadlocking

Question

In the following code, a deadlock happens on the Task.WhenAll line:

[Fact]
public async Task GetLdapEntries_ReturnsLdapEntries()
{
    var ldapEntries = _fixture.CreateMany<LdapEntryDto>(2).ToList();
    var creationTasks = new List<Task>();
    foreach (var led in ldapEntries)
    {
        var task = _attributesServiceClient.CreateLdapEntry(led);
        creationTasks.Add(task);
    }
    await Task.WhenAll(creationTasks); // <- deadlock here

    var result = await _ldapAccess.GetLdapEntries();

    result.Should().BeEquivalentTo(ldapEntries);
}

public async Task<LdapEntryDto> CreateLdapEntry(LdapEntryDto ldapEntryDto)
{
    using (var creationResponse = await _httpClient.PostAsJsonAsync<LdapEntryDto>("", ldapEntryDto))
    {
        if (creationResponse.StatusCode == HttpStatusCode.Created)
        {
            return await creationResponse.Content.ReadAsAsync<LdapEntryDto>();
        }

        return null;
    }
}

The xUnit test attempts to create test data asynchronously by calling an asynchronous method that itself awaits a response from a web service. _httpClient is a real HttpClient created off an in-memory TestServer via TestServer.CreateClient().

When setting a breakpoint on the using line in the CreateLdapEntry method, it is hit twice. A breakpoint on the status code check is never hit. When breaking on Task.WhenAll() and inspecting creationTasks, both tasks are in state WaitingForActivation:

creationTasks
Count = 2
    [0]: Id = 32, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}"
    [1]: Id = 33, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}"

When not using Task.WhenAll() but instead awaiting each task individually, no deadlock occurs:

foreach (var led in ldapEntries)
{
    await _attributesServiceClient.CreateLdapEntry(led);
}

I am aware that a similar question has been asked and answered, however the code examples there make use of .Result, not of await Task.WhenAll().

I'd like to understand why that deadlock is occuring when using Task.WhenAll().

EDIT: Added Call Stack of locked threads

Not Flagged     3992    11  Worker Thread   Worker Thread   Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke
                        [Managed to Native Transition]
                        Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)
                        ShibbolethAttributes.Service.dll!RoleManager.Service.Middleware.ApiKeyHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) Line 38
                        Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)
                        System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Startd__7>(ref Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.d__7 stateMachine)
                        Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)
                        Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.TestServer.ApplicationWrapper.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.HttpContextBuilder.SendAsync.AnonymousMethod__0()
                        System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Startc__DisplayClass10_0.b__0>d stateMachine)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.HttpContextBuilder.SendAsync.AnonymousMethod__0()
                        System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke()
                        System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
                        System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot)
                        System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()

Not Flagged     1496    10  Worker Thread   Worker Thread   Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke
                        Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)
                        ShibbolethAttributes.Service.dll!RoleManager.Service.Middleware.ApiKeyHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) Line 38
                        Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)
                        System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Startd__7>(ref Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.d__7 stateMachine)
                        Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)
                        Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.TestServer.ApplicationWrapper.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.HttpContextBuilder.SendAsync.AnonymousMethod__0()
                        System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Startc__DisplayClass10_0.b__0>d stateMachine)
                        Microsoft.AspNetCore.TestHost.dll!Microsoft.AspNetCore.TestHost.HttpContextBuilder.SendAsync.AnonymousMethod__0()
                        System.Private.CoreLib.dll!System.Threading.Tasks.Task.InnerInvoke()
                        System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
                        System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot)
                        System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()

Answer 1

(Conclusion summarised from the comments, for any future viewers).

You're not blocking anywhere, so the only possible explanation I can think of is that at least one of the requests isn't completing.

(If you were synchronously waiting for a Task to complete then I could well expect a deadlock, since you're not using ConfigureAwait(false), but since you only ever await your Tasks, I pretty sure this isn't the cause).

Given that your requests complete successfully when they're run individually, this implies that there's some concurrency issue when multiple requests are run in parallel, possibly to do with whatever _httpClient is, or to do with the server that requests are being made against (if they're running against a real server).

Given that your Task list doesn't show anything interesting, I'm inclined to think that one of the requests has synchronously blocked the thread that called into it.

Just see whether one of the requests has synchronously blocked. Open the Threads window, and double-click each thread in turn. Most won't be doing anything, but at least one might be running your code, or running a method called from your code. Look at the call stack to try and find out what's going on. You can double-click entries in the call stack to inspect variables in scope at each point.