Retrying C# HttpClient Unsuccessful Requests and Timeouts

Question

I'm trying to build in retrying in an HttpClient DelegatingHandler such that responses such as 503 Server Unavailable and timeouts are treated as transient failures and retried automatically.

I was starting from the code at http://blog.devscrum.net/2014/05/building-a-transient-retry-handler-for-the-net-httpclient/ which works for the 403 Server Unavailable case, but does not treat timeouts as transient failures. Still, I like the general idea of using the Microsoft Transient Fault Handling Block to handle the retry logic.

Here is my current code. It uses a custom Exception subclass:

public class HttpRequestExceptionWithStatus : HttpRequestException {
    public HttpRequestExceptionWithStatus(string message) : base(message)
    {
    }
    public HttpRequestExceptionWithStatus(string message, Exception inner) : base(message, inner)
    {
    }
    public HttpStatusCode StatusCode { get; set; }
    public int CurrentRetryCount { get; set; }
}

And here is the transient fault detector class:

public class HttpTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy {
    public bool IsTransient(Exception ex)
    {
        var cex = ex as HttpRequestExceptionWithStatus;
        var isTransient = cex != null && (cex.StatusCode == HttpStatusCode.ServiceUnavailable
                          || cex.StatusCode == HttpStatusCode.BadGateway
                          || cex.StatusCode == HttpStatusCode.GatewayTimeout);
        return isTransient;
    }
}

The idea is that timeouts should be turned into ServiceUnavailable exceptions as if the server had returned that HTTP error code. Here is the DelegatingHandler subclass:

public class RetryDelegatingHandler : DelegatingHandler {
    public const int RetryCount = 3;

    public RetryPolicy RetryPolicy { get; set; }

    public RetryDelegatingHandler(HttpMessageHandler innerHandler) : base(innerHandler)
    {
        RetryPolicy = new RetryPolicy(new HttpTransientErrorDetectionStrategy(), new ExponentialBackoff(retryCount: RetryCount,
            minBackoff: TimeSpan.FromSeconds(1), maxBackoff: TimeSpan.FromSeconds(10), deltaBackoff: TimeSpan.FromSeconds(5)));
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var responseMessage = (HttpResponseMessage)null;
        var currentRetryCount = 0;

        EventHandler<RetryingEventArgs> handler = (sender, e) => currentRetryCount = e.CurrentRetryCount;
        RetryPolicy.Retrying += handler;

        try {
            await RetryPolicy.ExecuteAsync(async () => {
                try {
                    App.Log("Sending (" + currentRetryCount + ") " + request.RequestUri +
                        " content " + await request.Content.ReadAsStringAsync());
                    responseMessage = await base.SendAsync(request, cancellationToken);
                } catch (Exception ex) {
                    var wex = ex as WebException;
                    if (cancellationToken.IsCancellationRequested || (wex != null && wex.Status == WebExceptionStatus.UnknownError)) {
                        App.Log("Timed out waiting for " + request.RequestUri + ", throwing exception.");
                        throw new HttpRequestExceptionWithStatus("Timed out or disconnected", ex) {
                            StatusCode = HttpStatusCode.ServiceUnavailable,
                            CurrentRetryCount = currentRetryCount,
                        };
                    }

                    App.Log("ERROR awaiting send of " + request.RequestUri + "\n- " + ex.Message + ex.StackTrace);
                    throw;
                }
                if ((int)responseMessage.StatusCode >= 500) {
                    throw new HttpRequestExceptionWithStatus("Server error " + responseMessage.StatusCode) {
                        StatusCode = responseMessage.StatusCode,
                        CurrentRetryCount = currentRetryCount,
                    };
                }
                return responseMessage;
            }, cancellationToken);

            return responseMessage;
        } catch (HttpRequestExceptionWithStatus ex) {
            App.Log("Caught HREWS outside Retry section: " + ex.Message + ex.StackTrace);
            if (ex.CurrentRetryCount >= RetryCount) {
                App.Log(ex.Message);
            }
            if (responseMessage != null) return responseMessage;
            throw;
        } catch (Exception ex) {
            App.Log(ex.Message + ex.StackTrace);
            if (responseMessage != null) return responseMessage;
            throw;
        } finally {
            RetryPolicy.Retrying -= handler;
        }
    }
}

The problem is that once the first timeout happens, the subsequent retries immediately time out because everything shares a cancellation token. But if I make a new CancellationTokenSource and use its token, no timeouts ever happen because I don't have access to the original HttpClient's cancellation token source.

I thought about subclassing HttpClient and overriding SendAsync but the main overload of it is not virtual. I could potentially just make a new function not called SendAsync but then it's not a drop-in replacement and I'd have to replace all the cases of things like GetAsync.

Any other ideas?

Answer 1

You may just want to subclass (or wrap) HttpClient; it seems cleaner to me to retry the requests at the HttpClient level rather than at the handler level. If this is not palatable, then you'll need to split up the "timeout" values.

Since your handler is actually doing multiple results in one, the HttpClient.Timeout applies to the entire process, including retries. You could add another timeout value to your handler which would be the per-request timeout, and use that with a linked cancellation token source:

public class RetryDelegatingHandler : DelegatingHandler {
  public TimmeSpan PerRequestTimeout { get; set; }
  ...
  protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  {
    var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
    cts.CancelAfter(PerRequestTimeout);
    var token = cts.Token;
    ...
        responseMessage = await base.SendAsync(request, token);
    ...
  }
}