Docker: How to connect to a kafka container which is not defined in docker-compose.yml file

Question

I have 3 docker-compose files. One to start the kafka and the other two are consumer and producer. Added external_links in the other docker-compose files to kafka, but still unable to access kafka from inside containers. From outside the container, I can access through localhost:9092, but what about inside docker container.

# docker-compose1.yml
version: "3.6"
services:
  zookeeper:
    image: 'docker.io/bitnami/zookeeper:3.7'
    container_name: zookeeper
    ports:
      - '2181:2181'
    volumes:
      - 'zookeeper_data:/bitnami'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'docker.io/bitnami/kafka:3'
    container_name: kafka
    ports:
      - '9092:9092'
    volumes:
      - 'kafka_data:/bitnami'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ADVERTISED_HOST_NAME=localhost
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
    depends_on:
      - zookeeper


volumes:
  zookeeper_data:
    external: true
  kafka_data:
    external: true

# docker-compose2.yml
version: "3.6"
services:
  web:
    hostname: ocp-transmitter
    image: 'ocp/transmitter'
    command: bash -c "bundle install && foreman start"
    ports:
      - '3000:3000'
    volumes:
      - .:/app:cached
    stdin_open: true
    tty: true
    external_links:
      - kafka

[Communication between multiple docker-compose projects](https://stackoverflow.com/questions/38088279/communication-between-multiple-docker-compose-projects) describes how to get different Compose files to share a network; does that meet your needs? (Anything called "links" is obsolete at this point and you can usually safely delete it with no impact.) — David Maze, Apr 05 '22 at 10:45
@DavidMaze I tried the link which you shared but it's not working. May be it works only for http — naveed, Apr 05 '22 at 11:58
Why do you need more than one compose file, anyway? You can selectively start individual services in one file or you can extend from other compose files and run them with one command — OneCricketeer, Apr 05 '22 at 13:09

score 1 · Accepted Answer · answered Apr 05 '22 at 13:07

First, remove these, they are deprecated

  - KAFKA_ADVERTISED_HOST_NAME=localhost
  - KAFKA_ADVERTISED_PORT=9092

Second, read the bitnami image documentation more carefully, all the Kafka properties start with KAFKA_CFG_, then read the section about internal/external listeners

The linked answer(s) are correct Communication between multiple docker-compose projects

Run docker network create with a name to setup an external bridge network separately from Compose, then add networks section to each service in that network (Zookeeper, Kafka, and your Kafka clients). Then make sure it's external

networks:
  example-net:
    external: true

Then you'd use kafka:29092 in your apps, not localhost, and not port 9092

Docker: How to connect to a kafka container which is not defined in docker-compose.yml file

1 Answers1