Docker + Airflow - Unable to connect to MySQL on host from docker container

Question

I'm using dockerized Airflow based on 'puckel/docker-airflow:1.10.9' image.

docker-compose file is the following:

version: '3.5'

services:
  postgres:
    image: postgres:12.3-alpine
    environment:
      POSTGRES_DB: airflow
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
    ports:
      - 5432:5432
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

  webserver:
    build:
      context: .
      dockerfile: docker/development/webserver/Dockerfile
    restart: always
    depends_on:
      - postgres
    environment:
      ...
    volumes:
      ...
    ports:
      - 8080:8080
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

Connection to MySQL on host is required in one of the tasks in DAG.

Connection details are the following:

When I'm trying to trigger DAG, the task instance fails with _mysql_exceptions.OperationalError: (2006, "Unknown MySQL server host 'host.docker.internal' (-2)")

I've tried the same as in this issue, but the problem still persists.

Also, here are some failed docker network checks:

$ docker run --rm webserver ping 'host.docker.internal'

ping: bad address 'host.docker.internal'

and

$ docker run --rm alpine nslookup host.docker.internal

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df20fa9351a1: Pull complete 
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest
Server:         192.168.100.1
Address:        192.168.100.1:53

** server can't find host.docker.internal: NXDOMAIN

** server can't find host.docker.internal: NXDOMAIN

which Host OS are you using? `host.docker.internal` this special DNS not available on linux — Adiii, Jun 30 '20 at 09:00
@Adiii ubuntu 19.10 and supposed to be running on ubuntu 18.04 in production — arturkuchynski, Jun 30 '20 at 09:01
okay, this not supported by ubuntu. this only supported by window and mac — Adiii, Jun 30 '20 at 09:03
you want to access host DB from which container? or you want to connect with `postgres` from another container? — Adiii, Jun 30 '20 at 09:05

score 2 · Answer 1 · answered Jun 30 '20 at 09:21

host.docker.internal is a special DNS that only work with Window and Mac.

You can try below workaround to resolve this DNS inside container with help of --add-host and where IP will refer to you Host IP address 192.168.9.100

docker run -it --rm=True --add-host=host.docker.internal:192.168.9.100 alpine sh -c "apk add curl --no-cache; curl host.docker.internal:8081"

or Better to use environment varaible

docker run -it --rm -e HOST_DB=192.168.9.100 myserver

and then you can use HOST_DB in the webserver container to connect with HOST DB.

score 0 · Accepted Answer · answered Jun 30 '20 at 12:29

I've added HOST_IP and HOST_DOMAIN to /etc/hosts inside webserver container. To do so, I have written a custom entrypoint.sh script

HOST_DOMAIN="host.docker.internal"
if ! ping -q -c1 $HOST_DOMAIN > /dev/null 2>&1
then
  HOST_IP=$(ip route | awk 'NR==1 {print $3}')
  # shellcheck disable=SC2039
  echo "$HOST_IP   $HOST_DOMAIN" >> /etc/hosts
fi

and now I can ping 'host.docker.internal' from inside webserver container:

# root@04cee44b7c1e:/usr/local/airflow# ping host.docker.internal

PING host.docker.internal (192.168.240.1): 56 data bytes
64 bytes from 192.168.240.1: icmp_seq=0 ttl=64 time=0.042 ms
64 bytes from 192.168.240.1: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 192.168.240.1: icmp_seq=2 ttl=64 time=0.060 ms
64 bytes from 192.168.240.1: icmp_seq=3 ttl=64 time=0.052 ms
64 bytes from 192.168.240.1: icmp_seq=4 ttl=64 time=0.055 ms
64 bytes from 192.168.240.1: icmp_seq=5 ttl=64 time=0.050 ms
64 bytes from 192.168.240.1: icmp_seq=6 ttl=64 time=0.045 ms

Why you need to maintain entrypoint when docker provides this functionality out of the box? Check the answers — Adiii, Jun 30 '20 at 14:40
@Adiii because I need to determine HOST_IP for dev and prod environment during CI/CD process, which can be different for a particular host. — arturkuchynski, Jul 01 '20 at 08:25

Docker + Airflow - Unable to connect to MySQL on host from docker container

2 Answers2