1

I'm setting up monitoring for Kafka connect in our Kafka ecosystem. I have enabled JMX exporter for kafka brokers and is working fine. Now I am trying to enable JMX exporter for kafka connect. However, it is a bit unclear where to start.

I can only modify connect-distributed.sh to enable the change. Any pointers would be a great addition.

kafka-run-class.sh was modified to enable jmx_exporter to emit jmx metrics on http://<host>:9304/metrics

I expect kafka-connect to emit metrics on http://<host>:19000/metrics once the jmx_exporter has been enabled.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
floating_hammer
  • 409
  • 3
  • 10

2 Answers2

7

Modifying a script in the bin directory is highly unrecommended. When upgrading Kafka to the next version, extracting the new binaries would override the changes made in the script.

The preferred way should be to set the environment variable KAFKA_JMX_OPTS outside the script:

export KAFKA_JMX_OPTS="-javaagent:/opt/kafka/libs/jmx_prometheus_javaagent-0.12.0.jar=127.0.0.1:10902:/etc/kafka-connect/jmx_exporter.yaml"

If the var is set before starting Kafka via the script it will use the var instead of the default values defined in /bin/kafka-server-start.sh

This answer is inspered by How do I set the Java options for Kafka?

Iskuskov Alexander
  • 4,077
  • 3
  • 23
  • 38
0

Digging around I have found a solution to the issue. Java agent can be added to the connect-distributed.sh and starts to emit the metrics nicely.

So enable jmx_exporter on Kafka Connect framework

  1. Open connect-distributed.sh
  2. Modify the last line which calls the kafka-run-class.sh
exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS org.apache.kafka.connect.cli.ConnectDistributed "$@"

to the following

exec $(dirname $0)/kafka-run-class.sh $EXTRA_ARGS -javaagent:/path/to/jmx_prometheus_javaagent-0.11.0.jar=9408:/path/to/config/file/prometheus.yml org.apache.kafka.connect.cli.ConnectDistributed "$@"
floating_hammer
  • 409
  • 3
  • 10