Docker based Consul Service Discovery with two network fails

Question

I work on consul cluster configuration with docker swarm. My service discovery with only one network works properly. But with second network added I get following info "[WARN] memberlist: Was able to connect to 69eca29632dc but other probes failed, network may be misconfigured". How to properly configure this network to overcome this problem?

version: '3'

services:
  consul:
    image: consul:latest
    deploy:
      replicas: 3
    environment:
      - CONSUL_LOCAL_CONFIG={\"disable_update_check\":true}
      - CONSUL_BIND_INTERFACE=eth0
      - CONSUL_HTTP_ADDR=0.0.0.0
    entrypoint:
      - consul
      - agent
      - -server
      - -bootstrap-expect=3
      - -data-dir=/consul/data
      - -bind={{ GetInterfaceIP "eth2" }}
      - -client=0.0.0.0
      - -retry-join=172.177.0.3
      - -retry-join=172.177.0.4
      - -retry-join=172.177.0.5
      - -ui
    networks:
      - backend #works properly without this line 
      - consul
    ports:
      - 8500:8500
      - 8600:8600

networks:
  consul:
    driver: overlay
    ipam:
      config:
        - subnet: 172.177.0.0/16
  backend:
    driver: overlay
    ipam:
      config:
        - subnet: 173.177.0.0/16

score 0 · Answer 1 · answered Aug 26 '19 at 13:56

In my case I got the following warning entries in syslog:

Aug 26 14:43:40 onl-vault01-poc consul[15046]: memberlist: Failed ping: onl-vault03-poc.dc1 (timeout reached)
Aug 26 14:43:42 onl-vault01-poc consul[15046]:     2019/08/26 14:43:42 [WARN] memberlist: Was able to connect to onl-vault03-poc.dc1 but other probes failed, network may be misconfigured
Aug 26 14:43:42 onl-vault01-poc consul[15046]: memberlist: Was able to connect to onl-vault03-poc.dc1 but other probes failed, network may be misconfigured

Granted, the message itself is neither specific nor very helpful. At the end it turned out that I was missing a firewall rule for udp/8302 between the nodes (although this port is documented as WAN port and my nodes are LAN only). You definitely need to expose ports tcp/8300,8301,8302 and udp/8301,8302 (see How do I publish a UDP Port on Docker? how to expose UDP ports in Docker). These are the default listening ports as this writing with Consul 1.5.x (see https://www.consul.io/docs/internals/architecture.html).

Docker based Consul Service Discovery with two network fails

1 Answers1