10

What is the difference between the following two retention configurations?

  • offsets.retention.minutes
  • log.retention.minutes

I don't get how it differs or relate to each other. From my understanding, once the offset is removed, the record in the log is not accessible and vice versa. Is there something I have misunderstood?

Ben Watson
  • 5,357
  • 4
  • 42
  • 65
Nicolas Henneaux
  • 11,507
  • 11
  • 57
  • 82

1 Answers1

25

The offset is a pointer to the most recent message that has been consumed by a consumer. So if you read 10 messages, the offset moves 10 places. offsets.retention.minutes allows you to move the offset back to the beginning if it isn't changed within a set period of time.

To visualise it, let's assume that we put the letters a to g (in that order) in a Kafka topic, all at different times. Before we start consuming the messages, the offset points to the oldest message:

OFFSET:    *
MESSAGES:  a b c d e f g

Now we consume 3 messages (a, b, c) so the offset moves:

OFFSET:          *
MESSAGES:  a b c d e f g

Now let's pretend we've set log.retention.minutes=10, and we put a and b into the topic 11 minutes ago, but the other messages were inserted more recently. We'd see:

OFFSET:          *
MESSAGES:      c d e f g

Now let's set offsets.retention.minutes=1, and pretend it has been 90 seconds since we last consumed anything. We'd see:

OFFSET:        *  
MESSAGES:      c d e f g

because c is now the oldest message on the topic (and the first that will be consumed).

Ben Watson
  • 5,357
  • 4
  • 42
  • 65
  • 2
    Thanks, it is more clear now. I would add a reference to `auto.offset.reset` which can change the behavior for the last scenario. [SO answer about that](https://stackoverflow.com/a/39142035/1630604) , [Proposal to change the default value to 7 days instead of 1 day in Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days) and the [Jira ticket related to the latter change](https://issues.apache.org/jira/browse/KAFKA-3806) – Nicolas Henneaux Feb 28 '18 at 12:15
  • 2
    The Jira Issue [KAFKA-3806](https://issues.apache.org/jira/browse/KAFKA-3806) also states _Offset retention should be always greater than log retention._ – Nicolas Henneaux Feb 28 '18 at 12:26
  • Is `offsets.retention.minutes` specific only to consumers in consumer groups? – deed02392 Jan 27 '21 at 21:35