Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
19
votes
7 answers

Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

I am using the confluent-kafka Python client in my project. I'm trying to create a Docker image with this client. I am facing the following error:- #11 8.015 [pipenv.exceptions.InstallError]: In file included from…
Jilaba Hindga
  • 191
  • 1
  • 1
  • 5
16
votes
1 answer

Kafka offset management: enable.auto.commit vs enable.auto.offset.store

A Kafka Consumer by default periodically commits the current offsets unless it is turned off by disabling enable.auto.commit. According to the documentation you're then responsible for committing the offsets yourself. So when I want manual control,…
Bastian Venthur
  • 12,515
  • 5
  • 44
  • 78
15
votes
1 answer

Kafka consumer gets stuck after exceeding max.poll.interval.ms

When the consumer does not receives a message for 5 mins (default value of max.poll.interval.ms 300000ms) the consumer comes to a halt without exiting the program. The consumer process hangs and does not consume any more messages. The following…
esquarer
  • 373
  • 1
  • 4
  • 12
8
votes
1 answer

Difference between Kafka async and Kafka sync acks=0?

Based on my readings, Kafka with acks=0 basically just pushes the message into the producer buffer. It does not wait for any sort of acks. With that, I'm wondering how it is different from async producer? How does acks affect the async producer?
La Reina
  • 81
  • 1
  • 3
7
votes
1 answer

Python librdkafka producer perform against the native Apache Kafka Producer

I am testing Apache Kafka Producer with native java implementation against Python's confluent-kafka to see which has the maximum throughput. I am deploying a Kafka cluster with 3 Kafka brokers and 3 zookeeper instances using docker-compose. My…
Amit Tripathi
  • 7,003
  • 6
  • 32
  • 58
6
votes
1 answer

librdkafka consumer and ssl configuration

I'm using librdkafka as a client consumer, and I have configured the broker and the client to support SSL, for the broker: listeners =…
Ismail
  • 2,322
  • 1
  • 12
  • 26
6
votes
1 answer

bazel rules_go: linking go binary against a static c++ library (.a file) produced by another target in the workspace

I'm using the confluent-kafka-go library inside my go binary, and this library needs to be linked against librdkafka. Other targets in my project use librdkakfa, so I have produced the static librdkafka.a and librdkafka++.a using rules_foreign_cc's…
mancini0
  • 4,285
  • 1
  • 29
  • 31
5
votes
0 answers

LibrdKafkaError: Broker: Unknown member after running around about 2 hours randomly

Right now, i want to implement node-rdkafka into our service, but i faced this error many times Broker: Unknown member. The same issue on github was https://github.com/confluentinc/confluent-kafka-dotnet/issues/1464. they say our consumer using same…
Ibid Athoillah
  • 95
  • 2
  • 10
4
votes
1 answer

How to use librdkafka with OIDC and Azure AD as token provider for OAUTHBEARER?

Problem I want to use Kafka OIDC with Azure AD as a token provider, but I'm stuck on some strange errors. I have read many azure docs pages and tried their examples from https://github.com/Azure/azure-event-hubs-for-kafka - all those worked for me…
Hlib Pylypets
  • 328
  • 2
  • 8
4
votes
1 answer

How does RD_KAFKA_PARTITION_UA work in librdkafka?

I have a producer in php which publishes to queue. I am using php-rdkafka library for the same. Following is the code: $conf->set('log_level', LOG_DEBUG); $conf->set('debug', 'all'); $rk = new RdKafka\Producer($conf); …
Rohit Aggarwal
  • 53
  • 1
  • 6
4
votes
1 answer

Error: Invalid value "sasl_ssl" for configuration property "security.protocol")

When trying to connect to kafka consumer ( kafka_2.11-2.0.0 in Centos ) using golang, getting below error : [ 2019_12_12_12:45:05 ] ----> Kafka Consumer Connection Error (kafka.Error=Invalid value "sasl_ssl" for configuration property…
Vani Polnedi
  • 595
  • 2
  • 4
  • 19
4
votes
1 answer

Build Golang Application with librdkafka in a Debian Docker Image?

With Alpine, Alpine fully supports recent versions of librdkafka, I can just do apk add in my Dockerfile, and the following works: FROM golang:1.13-alpine3.10 as builder WORKDIR /app COPY go.mod go.sum ./ COPY src ./src/ RUN set -eux; \ apk add…
clay
  • 18,138
  • 28
  • 107
  • 192
4
votes
0 answers

node-rdkafka is not getting installed in local windows system as well as in remote linux server

node-rdkafka needs Python 2.7 as one of the dependency and it is already available in the system, path is also set for it. Still npm is throwing errors related to librdkafka as per my understanding form the stacktrace. Please find below the error…
Mrinal Deo
  • 81
  • 2
  • 10
3
votes
1 answer

ignore tests in librd kafka

My golang project depends on librd kafka When I try to run go vet ./... or go test ./... from my jenkin, I get the following error. I beleive that is due to the fact that I am running ./... but even if I have to ignore the vendor I am not sure what…
Gayatri
  • 453
  • 1
  • 6
  • 18
3
votes
1 answer

Cannot pipenv install confluent-kafka 1.4.0 from source (pypi) - no workaround seems to work

There seems to be an issue logged with the current Confluent-Kafka package on pypi: I have a Dockerfile with the foll code which used to work until the issue happened : RUN cd /tmp && git clone https://github.com/edenhill/librdkafka.git \ && cd…
banditKing
  • 9,405
  • 28
  • 100
  • 157
1
2 3
…
11 12