76

I am running docker-nginx on ECS server. My nginx service is suddenly stopped because the proxy_pass of one of the servers got unreachable. The error is as follows:

[emerg] 1#1: host not found in upstream "dev-example.io" in /etc/nginx/conf.d/default.conf:988

My config file is as below:

 server {
       listen      80;
       server_name     test.com;
       location / {
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;

          ##proxy_set_header   Host             $host;
          proxy_set_header   X-Real-IP        $remote_addr;
          proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

          client_max_body_size       10m;
          client_body_buffer_size    128k;

          proxy_connect_timeout      90;
          proxy_send_timeout         90;
          proxy_read_timeout         90;

          proxy_buffer_size          4k;
          proxy_buffers              4 32k;
          proxy_busy_buffers_size    64k;
          proxy_temp_file_write_size 64k;
       }
}

server {
   listen       80 default_server;
   server_name  localhost;

   #charset koi8-r;
   #access_log  /var/log/nginx/log/host.access.log  main;

   location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
   }

   #error_page  404              /404.html;

   # redirect server error pages to the static page /50x.html
   #
   error_page   500 502 503 504  /50x.html;
   location = /50x.html {
      root   /usr/share/nginx/html;
   }
}

I have many servers in the config file, even if one server was down, I need to have running nginx. Is there any way to fix it?

Any suggestion to fix this issue would be appreciated.

Muhammad Tariq
  • 3,318
  • 5
  • 38
  • 42
Matrix
  • 2,399
  • 5
  • 28
  • 53
  • typically this can be for two reasons: either the service in http://dev-exapmle.io:5016/ is down or not responding to requests; or either you have some selinux configuration which is preventing nginx from proxying the requests. I'd look into the logs for for the app serving dev-example.io. If you don't see anything coming in, it's probably a selinux issue, in which case you may want to have a look at this answer: http://stackoverflow.com/questions/27435655/nginx-proxy-pass-not-working-in-selinux – Jose Haro Peralta Mar 10 '17 at 14:37
  • 2
    @JoséAntonioHaroPeralta thanks for the comment. yes that service, dev-exapmle.io:5016, is down. But I have many other services in my nginx config file that works but because of this service, the nginx stopped. Is there any way to fix it? – Matrix Mar 10 '17 at 14:39
  • include a resolver directive (pointing for example to google, 8.8.8.8). Nginx will resolve to that if it can't reach your site, as indicated in answers to the following question: http://stackoverflow.com/questions/32845674/setup-nginx-not-to-crash-if-host-in-upstream-is-not-found – Jose Haro Peralta Mar 10 '17 at 14:48

4 Answers4

74

Just adding a resolver did not resolve the issue in my case. But I was able to work around it by using a variable for the host. Also, I guess it makes more sense to use Docker's DNS at 127.0.0.11 (this is a fixed IP).

Example:

server {
  listen 80;
  server_name test.com;
  
  location / {
    resolver 127.0.0.11;
    set $example dev-example.io:5016;
    proxy_pass http://$example;
  }
}

I found the variable workaround on this page.

random-forest-cat
  • 33,652
  • 11
  • 120
  • 99
Damien
  • 3,060
  • 1
  • 25
  • 29
17

Include to prevent Nginx from crashing if your site is down, include a resolver directive, as follows:

 server {
       listen                80;
       server_name           test.com;
       location / {
          resolver           8.8.8.8;
          proxy_pass         http://dev-exapmle.io:5016/;
          proxy_redirect     off;
 ...

WARNING! Using a public DNS create a security risk in your backend since your DNS requests can be spoofed. If this is an issue, you should point the resolver to a secure DNS server.

Amin
  • 2,605
  • 2
  • 7
  • 15
Jose Haro Peralta
  • 969
  • 7
  • 14
  • 1
    Thanks. what if I use resolver 127.0.0.1 ? and if the server gets reachable again, would it routes the correct path? – Matrix Mar 10 '17 at 15:42
  • 5
    Hi, presumably this just adds another DNS resolution endpoint. In my case, the DNS address had been deleted and would not be resolvable anywhere. Is there not a way to have nginx simply treat the page as down rather than crashing the entire instance? – duct_tape_coder Sep 03 '20 at 18:19
  • 5
    For all copypasters like me ther is missing semicolon resolver 8.8.8.8; – Sultan Zhumatayev Mar 11 '21 at 16:16
2

This usually means that the dns name you provided as upstream server cannot be resolved. To test it, log on nginx server and try pinging upstream server provided and see if the name resolution completes correctly, If its a docker container try docker exec -it to get a shell, then try pinging the upstream to test the name resolution. If the contianer is stopped try to use IP address instead of dns name in your server block.

proxy_pass         http://<IP ADDRESS>:5016/;

You can also use the resolver directive if you want to use different dns server for this location than the host system:

resolver 8.8.8.8;
Farhad Farahi
  • 35,528
  • 7
  • 73
  • 70
  • There is no IP address for that, I have dns which is end-url of Docker-Cloud services – Matrix Mar 10 '17 at 14:51
  • Bring up a default config nginx container, then try pinging the dns to see if the resolution happens correctly. – Farhad Farahi Mar 10 '17 at 14:54
  • Thanks. Then I use resolver instead. but if I use resolver 127.0.0.1; would it work fine as well? What will happen if the DNS gets to work later? Also doesn't it need to have valid=30s; ? – Matrix Mar 11 '17 at 08:57
  • 2
    resolver 127.0.0.1 means you are running a dns server on localhost, are you? when the dns server is fixed remove the resolver line. valid 30s means do not requery for 30s use the existing value. – Farhad Farahi Mar 11 '17 at 09:46
  • Isn't there any way to do it automatically? Use resolver when it's down, use the DNS when it's up again. It could be done automatically only using a script? – Matrix Mar 11 '17 at 10:39
  • You can script it to check every 5 seconds for example and change the config accordingly – Farhad Farahi Mar 11 '17 at 11:48
  • @Farhad Farahi, will setting "resolver 8.8.8.8;" work if I already have "nameserver 8.8.8.8" in my /etc/resolve.conf? – Rico Chen Jul 13 '17 at 15:59
  • @RicoChen this should override your `/etc/resolvce.conf` for nginx process and workers. – Farhad Farahi Jul 13 '17 at 17:38
0

When using nginx plus, you can get around this as well by adding a zone to your upstream with resolve. When use this test in your proxypass. When the server some-server starts resolving, it will starting pass traffic to it.

Make sure to as stated above, put a resolver in other parts of your config. For docker, I use resolver 127.0.0.11 valid=1s;

upstream test {
        zone test-zone 64k;
        server some-server:1234 resolve;
}
Mark Cunningham
  • 19
  • 2