Connect to kafka cluster running on ubuntu 20.04 running in VirtualBox on Windows host

Question

I start the Kafka cluster in docker containers on Ubuntu 20.04 guest running on Windows VirtualBox host.

When it runs locally I can connect to kafka cluster with kcat using the following bootstrap.servers=localhost:29092,localhost:29093,localhost:29094

The Web UIs kafdrop and provectuslabs/kafka-ui are also working fine from localhost.

Windows machine connected to the modem via WiFi. On the modem side, I assigned the static address to Windows host 192.168.0.25 and created advertised listeners EXTERNAL_REMOTE_HOST://192.168.0.25:29082 (see my docker-compose below)

I created inbound rules to pass Internet traffic on Windows host for ports 29082-29084 and forwarded this ports to Ubuntu 20.04 guest changing VirtualBox settings for Ubuntu guest virtual machine.

I have another computer, lets call it remote host, which I assigned the IP address 192.168.0.21 on the modem side. It is connected to the modem by cable and shares the same network 192.168.0.*.

The connection from remote host 192.168.0.21 to Ubuntu guest running on Windows VirtualBox is successful.

# Ubuntu 20.04 on 192.168.0.21
ip a
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:94:62:db brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.21/24 brd 192.168.0.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::bd3c:b937:87b8:8a05/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

# try to connect to Ubuntu guest on Windows VirtualBox
nc -zvw10 192.168.0.25 29082
Connection to 192.168.0.25 29082 port [tcp/*] succeeded!

But connection from remote host 192.168.0.21 to Kafka cluster fails

# Ubuntu 20.04 on 192.168.0.21
cat ~/.config/kcat.conf 
bootstrap.servers=192.168.0.25:29082,192.168.0.25:29083,192.168.0.25:29084
kcat -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

What is wrong with my Kafka brokers configuration?

My docker-compose is:

version: '3.0'

services:
  zookeeper:
    image: myown/kafka:1.0
    hostname: zookeeper
    container_name: zookeeper
    networks: 
      - kafka
    links: 
     - "proxy"
    volumes:
      - /opt/kafka/zookeeper:/opt/kafka/zookeeper
    environment:
      ZOOKEEPER_ADDRESS: localhost
      ZOOKEEPER_PORT: 29090
      ZOOKEEPER_ID: 0
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka-broker-0:
    image: myown/kafka:1.0
    build: .
    hostname: kafka-broker-0
    container_name: kafka-broker-0
    depends_on:
      - zookeeper
    networks: 
      - kafka
    links:
      - "zookeeper"
    ports:
      - "29092:29092"
      - "29082:29082"
    volumes:
      - /opt/kafka/log:/opt/kafka/log
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka-broker-0:9082,INTERNAL_NETWORK://kafka-broker-0:9092,EXTERNAL_HOST://0.0.0.0:29092,EXTERNAL_REMOTE_HOST://0.0.0.0:29082
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-broker-0:9082,INTERNAL_NETWORK://kafka-broker-0:9092,EXTERNAL_HOST://localhost:29092,EXTERNAL_REMOTE_HOST://192.168.0.25:29082
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,INTERNAL_NETWORK:PLAINTEXT,EXTERNAL_HOST:PLAINTEXT,EXTERNAL_REMOTE_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      DELETE_TOPIC_ENABLE: "true"

  kafka-broker-1:
    image: myown/kafka:1.0
    hostname: kafka-broker-1
    container_name: kafka-broker-1
    depends_on:
      - zookeeper
    networks: 
      - kafka
    links:
      - "zookeeper"
    ports:
      - "29093:29093"
      - "29083:29083"
    volumes:
      - /opt/kafka/log:/opt/kafka/log
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka-broker-1:9082,INTERNAL_NETWORK://kafka-broker-1:9092,EXTERNAL_HOST://0.0.0.0:29093,EXTERNAL_REMOTE_HOST://0.0.0.0:29083
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-broker-1:9082,INTERNAL_NETWORK://kafka-broker-1:9092,EXTERNAL_HOST://localhost:29093,EXTERNAL_REMOTE_HOST://192.168.0.25:29083
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,INTERNAL_NETWORK:PLAINTEXT,EXTERNAL_HOST:PLAINTEXT,EXTERNAL_REMOTE_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      DELETE_TOPIC_ENABLE: "true"
  kafka-broker-2:
    image: myown/kafka:1.0
    hostname: kafka-broker-2
    container_name: kafka-broker-2
    depends_on:
      - zookeeper
    networks: 
      - kafka
    links:
      - "zookeeper"
    ports:
      - "29094:29094"
      - "29084:29084"
    volumes:
      - /opt/kafka/log:/opt/kafka/log
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka-broker-2:9082,INTERNAL_NETWORK://kafka-broker-2:9092,EXTERNAL_HOST://0.0.0.0:29094,EXTERNAL_REMOTE_HOST://0.0.0.0:29084
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-broker-2:9082,INTERNAL_NETWORK://kafka-broker-2:9092,EXTERNAL_HOST://localhost:29094,EXTERNAL_REMOTE_HOST://192.168.0.25:29084
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,INTERNAL_NETWORK:PLAINTEXT,EXTERNAL_HOST:PLAINTEXT,EXTERNAL_REMOTE_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      DELETE_TOPIC_ENABLE: "true"
  kafdrop:
    image: obsidiandynamics/kafdrop:3.30.0
    hostname: kafdrop
    container_name: kafdrop
    restart: "no"
    networks: 
      - kafka
    ports:
      - 9000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka-broker-0:9092,kafka-broker-1:9092,kafka-broker-2:9092
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - kafka-broker-0
      - kafka-broker-1
      - kafka-broker-2
  kafka-ui:
    image: provectuslabs/kafka-ui:7837622d5eaacae9bade7342f14732967e1b48d0
    container_name: kafka-ui
    networks: 
      - kafka
    depends_on:
      - kafka-broker-0
      - kafka-broker-1
      - kafka-broker-2
    ports:
      - "8080:8080"
    restart: always
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-broker-0:9092,kafka-broker-1:9092,kafka-broker-2:9092
networks:
  kafka:
    external: true

Start with getting one broker working, not 3. Plus, just run Docker for Windows. Why do you need a VM? Also, you don't need more than two listeners for each broker (one for the compose service name, and one more for the external host IP, given by the router) — OneCricketeer, Dec 27 '22 at 21:02
I created VM for testing environment. We need Linux and also in production we need three brokers each running on the separate node. This configuration is for testing. Regarding the listeners. So, you believe that for the listener "EXTERNAL_HOST://0.0.0.0:29092" it is possible and sufficient to advertise two addresses, like the following EXTERNAL_HOST://localhost:29092,EXTERNAL_REMOTE_HOST://192.168.0.25:29092 ? I will try it. — Alexander Borochkin, Dec 28 '22 at 14:48
Docker containers are Linux too and you can still run three of them, but you still shouldn't run three brokers on one machine, VM or container. Your testing code will work, whether you run one or more. But no, not localhost as a **advertised** listener. Refer https://www.confluent.io/blog/kafka-listeners-explained/ — OneCricketeer, Dec 28 '22 at 22:01

Ehsan Rezaee · Answer 1 · 2022-12-28T06:04:16.860

0

I think your kafka server don't know hosts of kafka brokers.

You could add them by change /etc/hosts in your kafka server like below.

<your_kafka_broker_ip> <your_kafka_broker_host>

edited Dec 28 '22 at 06:04

answered Dec 27 '22 at 11:48

Ehsan Rezaee

19
3

I added it to `/etc/hosts` on `192.168.0.21` from which I try to connect, but with no luck. The addresses `kafka-broker-0` are only reachable from docker network. But I need to reach the Kafka cluster from outside. It is possible to do from 192.168.0.25 host where Kafka cluster is running, but not from remote machine. – Alexander Borochkin Dec 27 '22 at 11:59
1

could you ```ping 192.168.0.25``` in your remote machine? – Ehsan Rezaee Dec 27 '22 at 12:02
The problem was that I can't reach 192.168.0.25 from remote host. I created VirtualBox VM on Windows with NAT interface. In such a case the bridged network is a requirement, so that my Ubuntu VM has its own IP address, different from my Windows machine address. I followed instructions on how to do that in "Bridged networking not working in Virtualbox under Windows 10" https://stackoverflow.com/questions/31922055/bridged-networking-not-working-in-virtualbox-under-windows-10 – Alexander Borochkin Dec 27 '22 at 13:05
Never suggest using /etc/hosts to fix Kafka listener issues. The router has a DNS server. So does VirtualBox – OneCricketeer Dec 27 '22 at 23:06

Connect to kafka cluster running on ubuntu 20.04 running in VirtualBox on Windows host

1 Answers1