1

I have a test automation project. I'm trying to get kafka consumer records starting from the latest record with config ConsumerConfig.AUTO_OFFSET_RESET_CONFIG = "latest". But it doesn't work. Here is a code where I'm trying to poll data:

for(int i=0; i<20; i++) {
            ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofMillis(500L));
            value = findValue(key, consumerRecords);
            if(value != null){
                break;
            }
        }

In this code variable consumerRecords has 0 size in every iteration.

If I change ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to "earliest" then consumer.poll() works and variable consumerRecords has not 0 size, but elements in collection are starting since earliest offset, while I need elements which are starting since last offset.

How I can achieve consumerRecords with elements in decreasing order by offset ?

I tried to increase the timeout to polling up to 10 seconds - it didn't help.

kafka-clients:2.7.0

Igor Vlasuyk
  • 514
  • 2
  • 10
  • 22

2 Answers2

0

With consumer AUTO_OFFSET_RESET_CONFIG = "latest" the consumer will get any record only if after the consumer is started there are records getting produced to the topic. You are getting records for AUTO_OFFSET_RESET_CONFIG = "earliest" because there are records already present in the topic. You can get more information here

Prog_G
  • 1,539
  • 1
  • 8
  • 22
0

I need elements which are starting since last offset.

Then you'll want to run some producer after the consumer has started from the end of the topic, where there is no data (yet). During an automated test, though, having only 20 polls from the end of the topic might yield a race condition in which the consumer could finish before the producer actually sends anything.

with elements in decreasing order by offset ?

It is not feasible to iterate a Kafka topic backwards

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Well, if you really want to iterate a topic backwards you can use the Consumer.seek method together with batch size config. I wouldn't advise to do so though, maybe it could be fine in a test. – kanly Dec 20 '21 at 08:53
  • It'd be really slow since you'd need to consume and seek one record offset at a time – OneCricketeer Dec 20 '21 at 15:02