Docker Network Nginx Resolver

Question

I am trying to get rid of deprecated Docker links in my configuration. What's left is getting rid of those Bad Gateway nginx reverse proxy errors when I recreated a container.

Note: I am using Docker networks in bridge mode. (docker network create nettest)

I am using the following configuration snippet inside nginx:

location / {
      resolver 127.0.0.1 valid=30s;
      set $backend "http://confluence:8090";
      proxy_pass $backend;

1. I started a container with hostname confluence on my Docker network with name nettest.
2. Then I started the nginx container on network nettest.
3. I can ping confluence from inside the nginx container
4. confluence is listed inside the nginx container's /etc/hosts file
5. nginx log says send() failed (111: Connection refused) while resolving, resolver: 127.0.0.1:53
6. I tried the docker network default dns resolver 127.0.0.11 from /etc/resol.conf
7. nginx log says confluence could not be resolved (3: Host not found)

Anybody knows how to configure nginx resolver with Docker Networks or an alternative on how to force Nginx to correctly resolve the Docker network hostname?

Answer 1

First off, you should be using the Docker embedded DNS server at 127.0.0.11.

Your problem could be caused by 1 of the following:

1. nginx is trying to use IPv6 (AAAA record) for the DNS queries.

   See https://stackoverflow.com/a/35516395/1529493 for the solution.

   Basically something like:

   http {
       resolver 127.0.0.11 ipv6=off;
   }
   

   This is probably no longer a problem with Docker 1.11:

   Fix to not forward docker domain IPv6 queries to external servers (#21396)

2. Take care that you don't accidentally override the resolver configuration directive. In my case I had in the server block resolver 8.8.8.8 8.8.4.4; from Mozilla's SSL Configuration Generator, which was overriding the resolver 127.0.0.11; in the http block. That had me scratching my head for a long time...

Answer 2

Maybe you should check your container's /etc/resolv.conf

It shows your container's correct DNS config and then use that DNS server IP for resolver.

127.0.0.11 does not works in Rancher

Answer 3

I was running "node:12.18-alpine" with angular frontend and hit the same problem with proxy_pass.

Locally it was working with:

resolver 127.0.0.11;

As simple as that! Just execute:

$ cat /etc/resolv.conf | grep nameserver

In your container to get this ip address.

However, when deploying to kubernetes (AWS EKS) I got the very same error:

failed (111: Connection refused) while resolving, resolver: 127.0.0.11:53

Solution:

First solution was to find out the IP of the kube-dns service like below:

$ kubectl get service kube-dns -n kube-system
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   178d

Simple replacing IP for CLUSTER-IP worked like a charm.

Later, after some more doc digging, I find out that I could reference the service by name (which is little bit more elegant and resilient):

resolver kube-dns.kube-system valid=10s;

Answer 4

My problem was $request_uri at the end. After adding it at the end of uri and changing the 127.0.0.1 to 127.0.0.11 solved my issue. I hope it will help people to not spend hours on this.

location /products {
            resolver 127.0.0.11;
            proxy_pass http://products:3000$request_uri;
            }

Answer 5

In several cases where I had this error, adding resolver_timeout 1s; to the Nginx config solved the issue. Most of the time I don't have a resolver entry.

Edit: what also worked for containers where I could explicitly define a nameserver: resolver DNS-IP valid=1s;

Answer 6

We hit this with docker containers on windows trying to lookup host.docker.internal using the docker internal resolver at 127.0.0.11. All queries would resolve correctly except host.docker.internal. Fix was to add the ipv6=off flag to the resolver line in nginx.conf.

Answer 7

I solved this problem with the following way:

docker run --rm -d --network host --name "my_domain" nginx

https://docs.docker.com/network/network-tutorial-host/

Answer 8

You need a local dns server like dnsmasq to resolve using 127.0.0.1. Try installing it using apk add --update dnsmasq and set it up if you're using an alpine (nginx:alpine) variant.