Stack Overflow
Stack Overflow

Questions tagged [kcat]

kcat is a single-binary Apache Kafka CLI tool

kcat is a single-binary Apache Kafka CLI tool for doing Unix cat-like operations against topics, such as reading (consuming), writing (producing), as well as Kafka related network troubleshooting.

https://github.com/edenhill/kcat

49 questions
8
votes
3 answers

Kafkacat: how to delete a topic or all its messages?

I am looking for a way to delete a topic or all its messages using kafkacat. Is it possible or the only way is through the script listed here? bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic
1Z10
  • 2,801
  • 7
  • 33
  • 82
6
votes
1 answer

Kafkacat Produce message from a file with headers

I need to produce batch messages to Kafka so I have a file that I feed kafkacat: kafkacat -b localhost:9092 -t -T -P -l /tmp/msgs The content of /tmp/msgs is as follows -H "id=1" {"key" : "value0"} -H "id=2" {"key" : "value1"} When I…
Jenia Ivanov
  • 2,485
  • 3
  • 41
  • 69
6
votes
1 answer

How can I install and configure KafkaCat in windows machine?

I am trying to install kafkaCat tool in windows machine. While doing a quick build by running ./bootstrap.sh, I am getting following error: $ ./bootstrap.sh Directory librdkafka already exists, not downloading…
Praveen
  • 1,378
  • 2
  • 15
  • 26
6
votes
1 answer

How to resolve "Leader not available" Kafka error when trying to consume

I'm playing around with Kafka and using my own local single instance of zookeeper + kafka and running into this error that I don't seem to understand how to resolve. I started a simple server per the Apache Kafka Quickstart Guide $…
Jon Erickson
  • 112,242
  • 44
  • 136
  • 174
5
votes
2 answers

How copy some message from one kafka topic to another from bash?

pls help We have 2 kafka topic. I want copy 10 message from beginning from topic1 to topic2. I`m try do it with kafka-console-consumer and kafka-console-producer First i save 10 message from topic1 to some directory: for (( i=1; i<=10; i++ )); do…
Sergey
  • 179
  • 6
  • 17
4
votes
4 answers

MacOS throwing dyld Error when running kafkacat

I want to use kafkacat command line utility(https://docs.confluent.io/current/app-development/kafkacat-usage.html) on my MacOS Mojave(10.14.5). I executed the below steps brew install kafkacat kafkacat -C -b brokerID -t kafkaTopic -e -K '+' I am…
Sujan Davangere Sunil
  • 269
  • 1
  • 4
  • 14
3
votes
0 answers

Can you use `kafkacat` with AWS IAM Access Control

I'm trying to use IAM Access Policy for authorization with MSK. I was trying to follow https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#configure-clients-for-iam-access-control. Looks like kafkacat doesn't support it.…
idetyp
  • 486
  • 1
  • 3
  • 13
3
votes
1 answer

How to use kafkacat with message-hub

In trying to use kafkacat with message hub, I've used the following: kafkacat -X client.id=xxxxx \ -X sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxx" password="xxxx";' \ -X…
Zach
  • 63
  • 1
  • 8
2
votes
1 answer

Sending messages to Kafka unbuffered using kafkacat

I have single node Kafka instance running locally via docker-compose. (system: Mac/Arm64, image: wurstmeister/kafka:2.13-2.6.0) I want to use kafkacat (kcat installed via Homebrew) to instantly produce and consume messages to and from Kafka. Here…
Juve
  • 10,584
  • 14
  • 63
  • 90
2
votes
0 answers

Statically linked binary for kcat or kafkacat

While trying to build kcat (Github edenhill/kcat - formerly known as kafkacat), I've stumbled upon some issues with library dependencies after the build has been successful, even though the build script has specific parameters for a static…
00ssh
  • 21
  • 2
2
votes
0 answers

Consuming with Kafkacat shows no messages, although using -o beginning

I'm trying to consume from a topic using Kafkacat , using the following command: kafkacat -b *****:9093 -C -K\t -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=**** -X sasl.password=**** -t **** -o beginning The…
Shizzle
  • 961
  • 3
  • 10
  • 27
2
votes
2 answers

Force kafka to connect brokers through IPs, not via hostnames

We have following kafka-ssh-tunneling setup. ssh -N $JUMPHOST -L 2181:w.x.y.z:2181 -L 9092:a.b.c.d:9092 -L 9091:e.e.f.f:9092 broker IP is a.b.c.d , There is local lo0 device alias with same IP address zookeper IP is w.x.y.z , There is local device…
libxelar.so
  • 473
  • 4
  • 16
2
votes
1 answer

Cannot write to kafka, brokers are down

I'm running Kafka locally. When I try to write to Kafka I get the following error: kafkacat -b localhost:9092 -t req -T -P -l msgs hello world % ERROR: Local: Broker transport failure: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed:…
Jenia Ivanov
  • 2,485
  • 3
  • 41
  • 69
2
votes
2 answers

Connecting to kafka from another namespace inside k8s

I have the following configuration for my kafka and zookeeper in my minikube: apiVersion: v1 kind: Service metadata: name: kafka-service namespace: kafka spec: selector: app: kafka ports: - protocol: TCP port: 9092 name:…
Rodolfo
  • 1,091
  • 3
  • 13
  • 35
2
votes
1 answer

How to turn a kafkacat payload printout into binary

I can use kafkacat to get a tuple from a topic and print it out: kafkacat -b kafka10.myorg.com:9092 -t MyTopic -o -1 -f '%s\n' And I get something like this in my terminal: �ǐے�ғ�����������ǐے�ғ���������S1_153314S3_153314S4_5422973S2_5420991... Is…
breezymri
  • 3,975
  • 8
  • 31
  • 65
1
2 3 4