Using exponential backoff in Polly Library with httpclient

Question

I just read about Polly library and I need to handle 3 retries when communicating from a desktop agent to a server.

Currently I like the exponential backoff.

However, I struggle to understand how to implement this in my code. This is what I have done so far:

using (HttpClient client = new HttpClient())
{
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", bearerToken);
    client.DefaultRequestHeaders.Add("TenantId", tenantId);
#if DEBUG
    var url = "http://localhost:5001/api/v1/RetrievePaymentDetails?orderpaymentid={orderPaymentId";
#else
    //TODO: add URL once the application is in production!
    var url = "intent2.api";
#endif
    Policy
        .Handle<HttpRequestException>()
        .WaitAndRetry(new[]
        {
            TimeSpan.FromSeconds(5),
            TimeSpan.FromSeconds(15),
            TimeSpan.FromSeconds(30)
        });

    using (HttpResponseMessage response = await client.GetAsync($"{url}"))
    {
        using (HttpContent content = response.Content)
        {
            HttpContentHeaders headers = content.Headers;
            var result = JsonConvert.DeserializeObject<PaymentDetailsDto>(response.Content.ReadAsStringAsync().Result);
            Currency currencyFromDb = (Currency)Enum.Parse(typeof(Currency), result.Currency);
            var result2 = await this.posManager.PayPos(result.Amount, currencyFromDb, result.SumUpUsername, result.SumUpPassword);

            if (result2.TransactionCode == null)
            {
                return this.Request.CreateResponse(HttpStatusCode.BadRequest, result2);
            }
            return this.Request.CreateResponse(HttpStatusCode.OK, result2);
            //return this.Request.CreateResponse<object>(result2);
        }
    }
}

I turned off my web server so I can simulate this exercise. Now, every time it hits that URL it throws a HttpRequestException and does not retry at all. How do I go about using this?

unrelated - You're using HttpClient wrong : `using (HttpClient client = new HttpClient())` — Fildor, Jul 15 '20 at 11:38

score 5 · Answer 1 · answered Jul 15 '20 at 11:43

Your are not using the policy. Defining one does not apply it to anything. Here is an example of a specialized transient error policy from the HttpPolicyExtensions. But this general pattern applies to any policy:

static AsyncRetryPolicy<HttpResponseMessage> transientRetry3Policy = HttpPolicyExtensions
          .HandleTransientHttpError()
          .RetryAsync(3);

And here is how you apply it to a request:

var someresult = await transientRetry3Policy
                .ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));

Peter Csala · Answer 2 · 2022-09-14T06:04:28.527

Your code could (and should) be improved in so many different ways.
Because we are not on code review that's why I will only focus on the Polly related stuff.

Retry in general

There is really important precondition for retry: the action (which might be called several times) should be idempotent and side-effect free. As an educated guess based on the provided code you are trying to retrieve information about a purchase which has already occurred. I suppose such query is written in an idempotent way.

I quite not sure that your posManager is idempotent and side-effect free (I suppose not). So my point is that you should choose wisely the scope of your resilient strategy, because it can cause damage if it is carried out without care.

Retry and its friends

Retry itself is just a policy not a resilient strategy. We can talk about strategy when we combine multiple policies in order to gain resilient behavior.

For example, imagine a situation when your payment service is highly overloaded and your response times goes insane. Let's say 99 seconds. Are you willing to wait for that? The default timeout of HttpClient is 100 seconds, which is a lot. If you don't control that your system will hang for a while. That's where Timeout comes into the picture.

It can be used for each individual query or as a global timeout for the whole process (including retries). So for example:

local timeout: 5 seconds
global timeout: 90 seconds

Another related policy is called CircuitBreaker which can help you to prevent flooding the downstream system (which is the payment service in your case). If your payment service already has hard time to serve requests then we should not send new ones to it.

CircuitBreaker monitors the successive failures and if a given threshold is exceeded then it will prevent all outgoing requests for a predefined amount of time to help the downstream system to get on foot again.

If you put all of this together then you will have a fairly resilient solution:

Global timeout

Retry

CircuitBreaker

Local timeout

Transient Http Errors

The HandleTransientHttpError method mentioned by Crowcoder handles either HttpRequestException or status code of 408 or 5xx. If your payment service returns another response status, for example 429 (Too Many Request) then it will not handle that. It's worth to bear in mind.

score 3 · Answer 3 · answered Dec 28 '20 at 07:24

Crowcoder's answer with exponential backoff policy

// Exponential backoff policy
AsyncRetryPolicy<HttpResponseMessage> exponentialBackoffPolicy = HttpPolicyExtensions
               .HandleTransientHttpError()
// Use sleepDurationProvider parameter
               .WaitAndRetryAsync(retryCount, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

// Using policy
var someresult = await exponentialBackoffPolicy
               .ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));

Using exponential backoff in Polly Library with httpclient

3 Answers3

Retry in general

Retry and its friends

Transient Http Errors

Linked