2

I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-* would have min ISR factor value set to 2, logs-* equal to 1 but value for core-* would be 4.

Solution proposal

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value looks almost as a solution but unfortunately it is shadowed when min.insync.replicas is also present in kafka.properties. So I will explore if there is solution to combine those two props. source

Verifying min ISR value using

./kafka-configs.sh --bootstrap-server localhost:9092 \
 --entity-type topics --entity-name $TOPIC --describe --all
Michał Dygas
  • 69
  • 7
  • How are you creating topics? There's plenty of tooling outside of the kafka-topics command that you can programmatically create this. The Terraform Kafka provider, for example, would just be a variable change – OneCricketeer Jan 05 '21 at 15:43
  • @OneCricketeer - `auto.create.topics.enable` is set to true and custom producers are writing to kafka cluster – Michał Dygas Jan 08 '21 at 00:37

2 Answers2

3

Edit: In order to avoid 80's style solutions (such as shown in this answer), please take a look at Dirk's answer.

I just love bash/sh

For already created topics, you'd have to execute a partition reassignment operation.

If you wish to control the ISR for newly created topics, you could wrap the kafka-topics.sh script in a new script, lets call it createTopics.sh:

#!/bin/bash
isr=0

if [[ $1 = \logs* ]]; then
        isr=1
elif  [[ $1 = \app* ]]; then
        isr=2
elif  [[ $1 = \core* ]]; then
        isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2
  • $1 - topic name
  • $2 - bootstrap-server
  • $3 - partitions

For example:

  ./createTopics.sh apps-topic localhost:9092 1 && \
  ./createTopics.sh core-topic localhost:9092 1

This would create two new topics with their respective replication-factor:

  • apps-topic > ISR = 2.
  • core-topic > ISR = 4.

(max ISR value, assuming all replicas are in-sync)

The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0.

You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1, for example. This would lead to:

#!/bin/bash
isr=1

if  [[ $1 = \app* ]]; then
       isr=2
elif  [[ $1 = \core* ]]; then
       isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2
aran
  • 10,978
  • 5
  • 39
  • 69
  • 1
    Thanks for Your answer, I believe that it might work but I would like to bootstrap cluster using config file instead of running custom script – Michał Dygas Jan 08 '21 at 00:12
  • 1
    @MichałDygas You're more than welcome, for your use case, I believe you could find some use of the kafkactl tool (as you said, it looks neat).I just love bash, but this solution is very error-prone on production...just a primitive way to solve this. More than a solution, a temporal fix. But I had a good time writting it, really! – aran Jan 08 '21 at 00:30
3

You can also use kafkactl for this:

# first run with --validate-only to see what kafkactl will do
kafkactl get topics -o compact | grep '^app.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 2 --validate-only" | sh
kafkactl get topics -o compact | grep '^log.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 1 --validate-only" | sh
kafkactl get topics -o compact | grep '^core.*' | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 4 --validate-only" | sh

# then do the replica reassignment
kafkactl get topics -o compact | grep '^app.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 2" | sh
kafkactl get topics -o compact | grep '^log.*'  | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 1" | sh
kafkactl get topics -o compact | grep '^core.*' | xargs -i echo "echo Topic: {};kafkactl alter topic {} --replication-factor 4" | sh

Note that the Kafka API that kafkactl is using for this is only available for Kafka ≥ 2.4.0.

Disclaimer: I am contributor to this project

D-rk
  • 5,513
  • 1
  • 37
  • 55
  • Hello @Dirk, `kafkactl` looks neat and believe me I will try using this tool. But I was looking for config file, that would provide this functionality without unnecessary cmd calls. – Michał Dygas Jan 08 '21 at 00:17