Spring Boot app is not able to publish events to Kafka in Docker

Question

Hello fellow programmers!

I am learning Docker and Kafka, and would be very grateful for any help. The problem is that the application is able to connect to Kafka, but not to publish any events. Error:

org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for wages-local-0:120001 ms has passed since batch creation

This problem arises when Spring Boot app in one Docker container tries to publish to Kafka in other container:

version: '3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    depends_on:
      - zookeeper
  publisher_app:
    build:
      context: ./kafka-publisher
    ports:
      - 8080:8080
    depends_on:
      - "kafka"
    environment:
      kafka.wages-topic.bootstrap-address: kafka:9092
    links:
      - kafka:kafka

Here is the link to source code: https://github.com/aleksei17/springboot-rest-kafka-mysql/blob/master/docker-compose.yml

are you able to ping to that container from your host ? This appears to be an issue with networking in docker container. follow below link for the solution:- https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach — Manish Kumar, Nov 27 '21 at 17:20
`kafka.wages-topic.bootstrap-address` does not look like an environment variable — OneCricketeer, Nov 29 '21 at 16:15
This is a variable from application.yml and it is used here: https://github.com/aleksei17/springboot-rest-kafka-mysql/blob/master/kafka-publisher/src/main/java/ee/aleksei/gvozdev/kafka/WagesTopicPublisherProperties.java — Aleksei, Nov 29 '21 at 17:28

score 0 · Answer 1 · answered Nov 29 '21 at 16:21

0

Your config is hard-coded to use localhost:9092

You want to externalize the config like so (remove your kafka.wages-topic section since bootstrap servers are not a topic-specific setting)

spring:
  kafka:
    bootstrap-servers: ${BOOTSTRAP_SERVERS}

Then in Compose,

environment:
  BOOTSTRAP_SERVERS: kafka:9092

To configure topics, see https://docs.spring.io/spring-kafka/reference/html/#configuring-topics

answered Nov 29 '21 at 16:21

OneCricketeer

179,855
19
132
245

Thanks for your answer! Seems there are no problem with `localhost:9092`, because this is overwritten in docker-compose by `kafka.wages-topic.bootstrap-address`. I would prefer to leave application.yml so that running from IntelliJ is easier – Aleksei Nov 29 '21 at 17:26
You can add environment variables to IntelliJ Run Configurations to make it work in both situations – OneCricketeer Nov 29 '21 at 19:40

score 0 · Accepted Answer · answered Nov 29 '21 at 17:35

Well, it seems that I needed to delete containers after usage. When I did so, I got an error when starting Kafka container like: unknown listener PLAINTEXT. So I changed PLAINTEXT to INTERNAL, and it worked! Here is the final version of docker-compose for Kafka:

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    depends_on:
      - zookeeper

It would be great if it worked both for Dockerized apps and for apps run on the same host outside Docker, but this is a different story. Played with this tutorial, but no success so far: https://www.baeldung.com/kafka-docker-connection

You only have one listener and one broker, and PLAINTEXT is the built-in, so you don't need a mapping or inter-broker listener — OneCricketeer, Nov 29 '21 at 19:41

Spring Boot app is not able to publish events to Kafka in Docker

2 Answers2