How to resolve Tasks hanging c#?

Question

While going over some code in a console application, I saw the nested Task.WhenAll in the SecondInitialize function. I decided to test this function with a large Locations List and see how it reacted.

What I saw is that with about 100 locations, 100*100 = 10,000 Calculate calls, the t.Wait() inside of Start takes about 60 seconds to return or sometimes just hang completely. If i try to click Break All the console application doesn't even respond to my click and visual studio crashes.

When using my 'Easier to read version' inside of SecondInitialize, it also takes a while to return. Consistent behavior.

Now the weird part is, whenever I use the debugger and put a breakpoint inside of the SecondInitialize and then hit continue, it will finish in about 5-7 seconds.

So my question is, why is it hanging taking a long time normally when I see it being faster when I debug inside that function? Another question is whether or not the use of Tasks is being utilized correctly

public void Start()
{
    var t = CacheInitialize(locations, CancellationToken.None);
    t.Wait();
}

public Task CacheInitialize(IList<Location> locations, CancellationToken token)
{
    return SecondInitialize(locations, token);
}

public async Task SecondInitialize(IList<Location> locations, CancellationToken token)
{
    await Task.WhenAll(locations.Select(first =>
    {
        return Task.WhenAll(locations.Where(second => !second.Equals(first)).Select(second =>
        {
            return Calculate(first, second, token);
        }));
    }));

    //Easier to read version of ^
    //var tasks = locations.SelectMany(first => locations.Where(second => !second.Equals(first)).Select(second =>
    //{
    //  return Calculate(first, second, token);
    //}));
    //await Task.WhenAll(tasks);


    //No Tasks.
    //for (int x = 0; x < locations.Length; x++)
    //{
    //    for (int y = 0; y < locations.Length; y++)
    //    {
    //        if (x == y)
    //            continue;
    //        await Calculate(locations[x], locations[y], token).ConfigureAwait(false);
    //    }
    //}
}

public async Task<TripLength> Calculate(Location start, Location finish, CancellationToken token)
{
    if (start == finish)
        return TripLength.Zero;

    var parameters = new RouteParameters
    {
        Coordinates = new []
        {
            new Coordinate(start.Latitude, start.Longitude),
            new Coordinate(finish.Latitude, finish.Longitude)
        }
    };

    var route = await RunRoute(parameters, token);

    return ToTripLength(route);
}


protected Task<RouteResult> RunRoute(RouteParameters routeParams, CancellationToken token)
{
    return Task.Run(async () =>
    {
        var routingTask = Task.Run(() =>
        {
            RouteResult routeResults;
            var status = _routeService.Route(routeParams, out routeResults);
            return routeResults;
        }, token);
    return await routingTask.ConfigureAwait(false);

    }, token);
}

Answer 1

The problem seems to be how to calculate routes from all trips connecting a set of locations (origins and destinations?) and calculate the length (cost?) of each route. The expensive work seems to be the call to _routeService.Route and ToTripLength.

Calculating 10K combinations from 100 locations is trivial and doesn't need parallelization. A simple LINQ query would work:

var combinations=( from start in locations
                   from finish in locations
                   where start!=finish
                   select (start,finish))
                 .ToArray();

What happens after that depends on what _routeService.Route does. If it's a local library, this is a data parallelism problem that tries to calculate 10K data points in the most efficient manner. That can be handled with PLINQ

If it's a call to an external service it's a concurrency problem that shouldn't waste CPU time waiting for 10K remote requests to respond.

Assuming _routeService.Route is a local libary, one can use PLINQ. A couple of helper methods will make writing the query easier though :

RouteParameters locationsToParams((Location start,Location finish) combination)
{
    return new RouteParameters {
        Coordinates = new[]
        {
            new Coordinate( start.Latitude, start.Longitude ),
            new Coordinate( finish.Latitude, finish.Longitude )
        }
    };
}

RouteResult  callRoute(RouteParameters routeParams)
{
    _routeService.Route(routeParams, out var routeResults);
    return routeResults;
}

var tripLengths = from cmb in combinations.AsParallel()
                  let routeParams=locationsToParams(cmb)
                  let result=callRoute(routeParams)
                  select ToTripLength(result);
var finalResults = tripLengths.ToArray();

AsParallel() will take the input IEnumerable, in this case combinations, partition it to as many partitions as there are cores and then use one worker task per partition. Each partition's data is fed to its worker task, minimizing the synchronization cost.

This could be used as a quick and rather dirty way to make 10K remote requests, as each call to Route will run on one of the worker tasks. This is wasteful because it blocks a task only to wait for a response. WithDegreeOfParallelism could be used to use more worker tasks than cores, but this still wastes CPU time waiting for a response. Blocking calls start with a SpinWait before a thread is suspended, which means a blocking call to a remote service can use a CPU core while doing nothing. This can seriously harm scalability on a server environment.

var tripLengths = from cmb in combinations.AsParallel()
                                          .WithDegreeOfParalellism(10)
                  let routeParams=locationsToParams(cmb)
                  let result=callRoute(routeParams)
                  select ToTripLength(result);
var finalResults = tripLengths.ToArray();

Answer 2

Since your example is not complete and can’t be compiled it’s kind of hard to see what exactly you are trying to do.

But as far as I can tell there are several problems:

1. Calling Wait (or Result) on a Task can lead to deadlocks. Using ConfigureAwait( false ) will help avoid such issues but can’t eliminate all of them. So best always await a Task when you want to access it’s result.

2. I don’t see what you are trying to accomplish with nesting Task.WhenAll within Task.WhenAll. WhenAll returns a single Task which you can just await without Task.WhenAll. Each Task you create will add some performance overhead, so you should try to create as few task as possible.

3. Using Task.Run with an async delegate to await another Task(created by Task.Run) makes no sense, you are creating more Tasks than you need. You can just await a single Task.Run

I tried to create a working example (it won’t do any work) based on your code to show what you should change. Please note async Main method is only available in C# 7.1 or above.

public class Program
{
    public static async Task Main( String[] args )
    {
        var foo = new Foo();

        var sw = Stopwatch.StartNew();

        await foo.Start();

        sw.Stop();

        Console.WriteLine($"Elapsed {sw.Elapsed} {sw.ElapsedMilliseconds}ms");
        Console.ReadLine();
    }
}

public class Foo
{
    public async Task CacheInitialize( IList<Location> locations, CancellationToken token ) =>
        await SecondInitialize( locations, token )
            .ConfigureAwait( false );

    public async Task<TripLength> Calculate( Location start, Location finish, CancellationToken token )
    {
        if ( start == finish )
            return TripLength.Zero;

        var parameters = new RouteParameters
        {
            Coordinates = new[]
            {
                new Coordinate( start.Latitude, start.Longitude ),
                new Coordinate( finish.Latitude, finish.Longitude )
            }
        };

        var route = await RunRoute( parameters, token );

        return new TripLength();
    }

    public async Task SecondInitialize( IList<Location> locations, CancellationToken token )
    {
        var tasks = new List<Task>( locations.Count );

        foreach ( var outer in locations )
        foreach ( var inner in locations )
        {
            if ( inner.Equals( outer ) )
                continue;

            tasks.Add( Calculate( outer, inner, token ) );
        }

        await Task.WhenAll( tasks );
    }

    public async Task Start()
    {
        var locations = new List<Location>();
        await CacheInitialize( locations, CancellationToken.None )
            .ConfigureAwait( false );
    }

    protected async Task<RouteResult> RunRoute( RouteParameters routeParams, CancellationToken token )
    {
        return await Task
                     .Run( () =>
                           {
                               //RouteResult routeResults;
                               //var status = _routeService.Route( routeParams, out routeResults );
                               //return routeResults;
                               return new RouteResult();
                           },
                           token )
                     .ConfigureAwait( false );
    }
}

public class Coordinate
{
    public Double Latitude { get; }
    public Double Longitude { get; }
    public Coordinate( Double latitude, Double longitude )
    {
        Latitude = latitude;
        Longitude = longitude;
    }
}
public class RouteParameters
{
    public Coordinate[] Coordinates { get; set; }
}
public class TripLength
{
    public static TripLength Zero = new TripLength();
}
public class RouteResult
{
}
public class Location
{
    public Double Latitude { get; }
    public Double Longitude { get; }
}