Run kafka using docker compose and expose a different port instead of default one

Question

I'm trying to start a kafka cluster using docker compose, I'm using the following configurations:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    ports:
      - "15092:9092"
    hostname: kafka
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:15092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://:15092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - "zookeeper"

Both services are up and running, but when I try to produce a message from an external source using broker external-ip:15092 I receive the following error:

dial tcp: lookup kafka: no such host

Can you help me figure out what the configuration is missing?

Thanks

From the external machine the name kafka is not resolvable. Can you show what is the connection string you are using? — Mohit Mutha, Dec 16 '19 at 06:10

score 4 · Answer 1 · edited Aug 11 '21 at 19:34

Here's a configuration that exposes different ports:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "49815:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: wurstmeister/kafka
    container_name: broker
    ports:
      - "49816:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://broker:9092,INTERNAL://broker:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:49816,INTERNAL://broker:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Key points:

Your external source needs to connect to the address that is exposed by Docker (localhost:49816) and bound to the broker using the port 9092. So you need to set KAFKA_ADVERTISED_LISTENERS with that value for external connections.
The broker should be listening to 9092 to get the connections coming from the outside through Docker port binding. So you need to set PLAINTEXT://broker:9092 as an external listener in KAFKA_LISTENERS.
You still need to listen for inter container connections for Brokers and Zookeeper to function, so you need to set KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS with internal endpoints INTERNAL://broker:29092 that are visible from inside Docker.

Here's the complete configs link: https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#brokerconfigs_listeners (it's for Confluence image but it works the same with wurstmeister's)

With this method all the containers also inside docker-compose tries to connect to localhost:49816 even though i've defined them to connect to broker:9092. Such as Kafka Connect which has CONNECT_BOOTSTRAP_SERVERS: "broker:9092". Any advice? — Ville, Apr 12 '22 at 18:04

OneCricketeer · Answer 2 · 2019-12-16T06:03:48.927

2

You're getting "no such host", which occurs before the port is even used. You need to run code in another container in the same Docker network for the service name to resolve

Kafka doesn't work like that (a simple port forward), anyway

Both the listeners are still set set at 9092

You'll need to add / change the advertised listener containing externalIP:15092 for it to work, and you can find multiple places where the difference of listeners is documented (including the wiki pages for that container)

So, KAFKA_LISTENERS: INSIDE://:9092 is all you need (or more appropriately, put INSIDE://0.0.0.0:9092)

But you need to edit

KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://<your_external_IP>:15092

edited Dec 16 '19 at 06:03

answered Dec 15 '19 at 16:13

OneCricketeer

179,855
19
132
245

Thank you for your response. Can you give me some examples with documentation of difference of listeners? I can't figure out what is the problem, I updated the post with what I'm trying now – Dorin Dec 16 '19 at 05:14
Just search [listeners vs advertised listeners](https://stackoverflow.com/questions/42998859/kafka-server-configuration-listeners-vs-advertised-listeners) ... I have updated my answer – OneCricketeer Dec 16 '19 at 06:03
@cricket_007 I ran druid in docker, now when i run kafka in docker [https://github.com/wurstmeister/kafka-docker] bit it has its own zooker, how can i specify both kafka and druid needs to use the same zookeeper. – George Thomas Dec 16 '19 at 10:45
You would 1) Use the same docker-compose file 2) Point both at `zookeeper:2181` – OneCricketeer Dec 17 '19 at 00:33

Run kafka using docker compose and expose a different port instead of default one

2 Answers2