4

When Kafka Consumer is launched from terminal, it is possible to set --from-beginning, so that the consumer read messages from the beginning of a Kafka queue.

~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TutorialTopic --from-beginning

However, how can I set this parameter in Scala (or Java)? This is my sample ConsumerConfig:

  import kafka.consumer.ConsumerConfig

  def createConsumerConfig(zookeeper: String, groupId: String): ConsumerConfig = {
    val props = new Properties()
    props.put("zookeeper.connect", zookeeper);
    props.put("group.id", groupId);
    props.put("auto.offset.reset", "largest");
    props.put("zookeeper.session.timeout.ms", "400");
    props.put("zookeeper.sync.time.ms", "200");
    props.put("auto.commit.interval.ms", "1000");
    val config = new ConsumerConfig(props)
    config
  }
Klue
  • 1,317
  • 5
  • 22
  • 43
  • 5
    Possible duplicate of [How to read data using Kafka Consumer API from beginning?](http://stackoverflow.com/questions/28561147/how-to-read-data-using-kafka-consumer-api-from-beginning) – Chobeat Jul 09 '16 at 12:16
  • 1
    You want `props.put("auto.offset.reset", "smallest");` – Yuval Itzchakov Jul 09 '16 at 12:57
  • 3
    Setting auto.offset.reset is not good enough. First, a consumer looks for a committed offset and if found, it will read from there. auto.offset.reset only has an effect, if no (valid) committed offset is found. Thus, you need to get rid of the offset by deleting if from ZK (for 0.8.2) or from the consumer offsets topic (>= 0.9.0) -- or by using a new "group.id" (because offsets are associated with group ID). Last but not least, you can use "seek" to go back to offset zero. – Matthias J. Sax Jul 12 '16 at 11:48
  • @Klue As a continuation to Matthias answer, if you were to seek to beginning of assigned partitions and consume from there,callbacks in ConsumerRebalanceListener can be used for that purpose. See this answer here https://stackoverflow.com/questions/28561147/how-to-read-data-using-kafka-consumer-api-from-beginning/47530912#47530912 – skm Nov 28 '17 at 19:53

0 Answers0