How to write a timeout for short and repeatedly action?

Question

I would like to write a timeout function for the BasicPublish method of the RabbitMQ C# client. For many reasons sometimes the queue is blocked, or rabbit is down or whatever. But I want to detect when the publish is failing right away. I do not want to block the site for any reason.

I'm worried to implement a timeout with Task or threads adding overhead for a simple publish, that we are doing it millions of times in production.

Does anyone have and idea how to write a quick timeout on a fast blocking method as BasicPublish?

Clarification: Also I'm working in .Net 4, I do not have async.

You can use `async`/`await` with .NET4.0 using the [Microsoft.BCL.Async](https://www.nuget.org/packages/Microsoft.Bcl.Async/) package. — mountain traveller, Feb 22 '17 at 18:16

mountain traveller · Accepted Answer · 2017-02-22T23:13:01.683

Polly has a TimeoutPolicy aimed at exactly this scenario.

Polly's TimeoutStrategy.Optimistic is close to @ThiagoCustodio's answer, but it also disposes the CancellationTokenSource correctly. RabbitMQ's C# client doesn't however (at time of writing) offer a BasicPublish() overload taking CancellationToken, so this approach is not relevant.

Polly's TimeoutStrategy.Pessimistic is aimed at scenarios such as BasicPublish(), where you want to impose a timeout on a delegate which doesn't have CancellationToken support.

Polly's TimeoutStrategy.Pessimistic:

[1] allows the calling thread to time-out on (walk away from waiting for) the execution, even when the executed delegate doesn't support cancellation.

[2] does so at the cost of an extra task/thread (in synchronous executions), and manages this for you.

[3] also captures the timed-out Task (the task you have walked away from). This can be valuable for logging, and is essential to avoid UnobservedTaskExceptions - particularly in .NET4.0, where an UnobservedTaskException can bring down your entire process.

Simple example:

Policy.Timeout(TimeSpan.FromSeconds(10), TimeoutStrategy.Pessimistic).Execute(() => BasicPublish(...));

Full example properly avoiding UnobservedTaskExceptions:

Policy timeoutPolicy = Policy.Timeout(TimeSpan.FromSeconds(10), TimeoutStrategy.Pessimistic, (context, timespan, task) => 
{
    task.ContinueWith(t => { // ContinueWith important!: the abandoned task may very well still be executing, when the caller times out on waiting for it! 
        if (t.IsFaulted) 
        {
            logger.Error($"{context.PolicyKey} at {context.ExecutionKey}: execution timed out after {timespan.TotalSeconds} seconds, eventually terminated with: {t.Exception}.");
        }
        else
        {
           // extra logic (if desired) for tasks which complete, despite the caller having 'walked away' earlier due to timeout.
        }
    });
});

timeoutPolicy.Execute(() => BasicPublish(...));

To avoid building up too many concurrent pending tasks/threads in the case where RabbitMQ becomes unavailable, you can use a Bulkhead Isolation policy to limit parallelization and/or a CircuitBreaker to prevent putting calls through for a period, once you detect a certain level of failures. These can be combined with the TimeoutPolicy using PolicyWrap.

score 0 · Answer 2 · answered Feb 16 '17 at 20:05

0

I would say that the easiest way is using tasks / cancellation token. Do you think it's an overhead?

public static async Task WithTimeoutAfterStart(
    Func<CancellationToken, Task> operation, TimeSpan timeout)
{
    var source = new CancellationTokenSource();
    var task = operation(source.Token);
    source.CancelAfter(timeout);
    await task;
}

Usage:

await WithTimeoutAfterStart(
    ct => SomeOperationAsync(ct), TimeSpan.FromMilliseconds(n));

answered Feb 16 '17 at 20:05

Thiago Custodio

17,332
6
45
90

1

Also you could wrap your code using Polly: https://github.com/App-vNext/Polly#retry and https://github.com/App-vNext/Polly#retry – Thiago Custodio Feb 16 '17 at 20:08
I do not have big knowledge about if Task really creates a thread or not and when exactly it creates a new thread. But yes I think that creating a Thread for each publish es a overhead for a publish action when everything is working ok. Also intersted library Polly, I will investigate it – elranu Feb 16 '17 at 20:14
Cool. In the end it's just a retry pattern encapsulated. If none of them fits your need, you can always write your own: https://msdn.microsoft.com/en-us/library/dn589788.aspx and http://stackoverflow.com/questions/1563191/cleanest-way-to-write-retry-logic – Thiago Custodio Feb 16 '17 at 20:18
Circuit breaker is goog idea in general and Polly is afaik lightweight yet comfortable implementation – Alexander Powolozki Feb 16 '17 at 20:32

How to write a timeout for short and repeatedly action?

2 Answers2