7

Here's what we have here:

  • Topic Exchange DLE, which is intended to be a Dead-Letter Exchange
  • Topic Exchange E, which is the "main" Exchange
  • Several Queues (EQ1, ..., EQn) bound to E (and initialized with x-dead-letter-exchange = DLE), each with own Routing Key. These queues are the ones being consumed from.
  • For each EQn, there's a DLEQn (initialized with x-dead-letter-exchange = E and x-message-ttl = 5000), bound to DLE with the same routing key as EQn. These queues are not being consumed from

What I want is the following: if a consumer cannot process a message from EQn, it Nacks the message with requeue: false and it gets to the DLEQn - that is, to an appropriate queue on the Dead-Letter Exchange. Now, I want this message to sit on the DLEQn for some time and then get routed back to the original queue EQn to be processed again.

Try as I might, I could not get the "redelivery to the original queue" working. I see that messages sit in the DLEQn with all the right headers and Routing Key intact, but after TTL expires they just vanish into thin air.

What am I doing wrong here?

Anton Gogolev
  • 113,561
  • 39
  • 200
  • 288

3 Answers3

9

Yes, you can do this. We are currently doing this in production and it works great. The code is too long to include here but I will show you the diagram I created that represents the process. The basic idea is that the First DLX has a TTL, once that TTL expires the message goes into a 2nd queue to be re-sent back into the original.

enter image description here

jhilden
  • 12,207
  • 5
  • 53
  • 76
  • How does it get resent to the original queue? Is it RabbitMQ that does the resending or is there a separate consumer on the retry queue? – Anton Gogolev Feb 20 '15 at 18:35
  • 3
    A separate consumer picks up the message from "retry redirect queue" and looks at the rabbit headers. The header of interest is "x-death", inside there it has the exchange and queue that the message came from. We use this to send messages back to many different queues, that way we only need one "retry redirect queue". Make sense? – jhilden Feb 20 '15 at 18:47
  • @jhilden Could you please tell how do you maintain count of the failed messages. Do you use some inbuilt functionality or you had maintain custom variable of count and you increment everytime it is retried – Naresh Apr 11 '16 at 11:35
  • Custom variable, here is the code that checks that count: https://gist.github.com/jayhilden/2078872a53c7df0fe45d661861ed2d45 – jhilden Apr 11 '16 at 15:02
  • @jhilden When you get x-death at "Retry Redirect Queue" wouldn't it be holding "Retry Holding Queue" name and not inbound queue - because that's where it came from? – Pavel Dubinin May 11 '17 at 11:09
  • It's been a while, but I'm pretty sure it will have an x-death of *both*, so I take the first one, which is the original queue. – jhilden May 11 '17 at 15:48
  • @jhilden sorry to comment on such an old answer(great answer by the way) but the 'x-death' key is one that you added to your headers for the message properties correct? I have read the docs here, https://pika.readthedocs.io/en/0.10.0/modules/spec.html#pika.spec.BasicProperties , and headers appears to be none. Does that header have to be called x-death and if so do you recall where in the docs this would be? – UCProgrammer Oct 11 '18 at 15:38
  • @jhilden I see my folly now. It shows when you access properties.headers after it has already been dead lettered. of course! As far as sending back to the original queue goes, that would be just another basic_publish I assume. – UCProgrammer Oct 11 '18 at 16:21
5

RabbitMQ detects message flow cycling (E -> DLE -> E -> DLE ...) and silently drops messages:

From DLX manual (Routing Dead-Lettered Messages section):

It is possible to form a cycle of dead-letter queues. For instance, this can happen when a queue dead-letters messages to the default exchange without specifiying a dead-letter routing key. Messages in such cycles (i.e. messages that reach the same queue twice) will be dropped if the entire cycle is due to message expiry.

pinepain
  • 12,453
  • 3
  • 60
  • 65
  • I wonder if "25107 permit dead-letter cycles" in RabbitMQ 3.1 ( http://www.rabbitmq.com/release-notes/README-3.1.0.txt ) actually allows them. – Anton Gogolev Feb 20 '15 at 18:37
  • I guess asking for clarification in RabbitMQ user group (https://groups.google.com/forum/#!forum/rabbitmq-users) will bring some light on this question. I don't think that it 25107 is your case while you are using message expiry, which is the case to drop messages according to doc (i made bold that part) – pinepain Feb 20 '15 at 20:29
  • Well, my cycle is decidedly _not_ entirely because of expiry. Nacking original message moves it from E to DLE, and only then does TTL comes into play. Nevertheless, thanks for taking time! – Anton Gogolev Feb 20 '15 at 21:06
3

That post is pretty old, but it took me days to find a solution for a similar problem, so I thought I should share my solution here.

We're receiving messages in TargetQueue (no TTL!!!, bound to TargetExchange) which may be nacked by the consumer. TargetQueue has a DLX defined (RetryExchange), which in turn has bound a corresponding queue (RetryQueue, with a TTL of 60 secs, TargetExchange defined as DLX).

So if the consumer nacks a message from TargetQueue, it gets queued up in the RetryQueue and because of the TTL, the message gets nacked again and requeued in the original TargetQueue. The clue was, that TargetQueue may not have a TTL defined, otherwise a message like this appears in the RabbitMQ log:

Dead-letter queues cycle detected: [<<"TargetQueue">>,<<"RetryQueue">>,<<"TargetQueue">>]

So in the end the solution is pretty straight forward (and only needs one consumer). I got the final inspiration from https://medium.com/@igkuz/ruby-retry-scheduled-tasks-with-dead-letter-exchange-in-rabbitmq-9e38aa39089b

nkr
  • 3,026
  • 7
  • 31
  • 39
Thomas
  • 71
  • 1
  • 4