Server-DB Communication Issue at Built

Question

Setup

Consider this relatively simple docker-app using fastapi as server and couchdb as db.

Here is the project structure

├── app.py
├── compose
│   ├── couchdb.Dockerfile
│   └── server.Dockerfile
├── docker-compose.yml
├── .env  # Global vars e.g. DB credentials
├── Pipfile  # For local dev
├── Pipfile.lock  # Same
└── requirements.txt

Here are is my app.py

import couchdb
from fastapi import FastAPI, HTTPException
import os


def _load_db_client():
    _user = os.environ["COUCHDB_USER"]
    _password = os.environ["COUCHDB_PASSWORD"]
    _host = os.environ["COUCHDB_HOST"]
    _port = os.environ["COUCHDB_PORT"]
    _client = couchdb.Server(f"http://{_user}:{_password}@{_host}:{_port}")
    del _user, _password, _host, _port
    return _client


def _get_or_create_db(_client: couchdb.Server, db_name: str) -> couchdb.Database:
    if db_name in _client:
        return _client[db_name]
    print("Creating DB", db_name)
    return _client.create(db_name)


couch: couchdb.Server = _load_db_client()
db_user = _get_or_create_db(_client=couch, db_name="user")

app = FastAPI()

And here are my docker files:

### docker-compose ###
version: "3.4"

# This help to avoid routing conflict within virtual machines:
networks:
  default:
    ipam:
      driver: default
      config:
        - subnet: 192.168.112.0/24

services:

  couchdb:
    restart: unless-stopped
    build:
      context: .
      dockerfile: compose/couchdb.Dockerfile
    expose:
      - 5984
    ports:
      - "59840:5984"
    env_file:
      - .env

  server:
    restart: unless-stopped
    build:
      context: .
      dockerfile: compose/server.Dockerfile
    expose:
      - 8080
    ports:
      - "8080:8080"
    env_file:
      - .env

### couchdb.Dockerfile ###
FROM couchdb:latest

### server.Dockerfile ###
FROM python:3.8

RUN apt-get update && apt-get -y install tmux nano

ADD ./requirements.txt /srv
WORKDIR /srv
RUN pip install --upgrade pip
RUN pip install -r ./requirements.txt
ADD . /srv

CMD uvicorn app:app --host 0.0.0.0 --port 8080 --reload

Problem

When building docker, the command starting the server inside docker (`uvicorn app:app ...`) fails because of the line `db_user = ...`.

Here is the full error:

INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [6] using statreload
Creating DB user
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/site-packages/uvicorn/subprocess.py", line 62, in subprocess_started
    target(sockets=sockets)
File "/usr/local/lib/python3.8/site-packages/uvicorn/main.py", line 390, in run
    loop.run_until_complete(self.serve(sockets=sockets))
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.8/site-packages/uvicorn/main.py", line 397, in serve
    config.load()
File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 278, in load
    self.loaded_app = import_from_string(self.app)
File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
    module = importlib.import_module(module_str)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import  File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 783, in exec_module
File "", line 219, in _call_with_frames_removed
File "./app.py", line 24, in
     db_user = _get_or_create_db(_client=couch, db_name="user")
 File "./app.py", line 20, in _get_or_create_db
     return _client.create(db_name)
 File "/usr/local/lib/python3.8/site-packages/couchdb/client.py", line 221, in create
     self.resource.put_json(name)
 File "/usr/local/lib/python3.8/site-packages/couchdb/http.py", line 577, in put_json
     return self._request_json('PUT', path, body=body, headers=headers,
File "/usr/local/lib/python3.8/site-packages/couchdb/http.py", line 595, in _request_json
    status, headers, data = self._request(method, path, body=body,
File "/usr/local/lib/python3.8/site-packages/couchdb/http.py", line 590, in _request
    return self.session.request(method, url, body=body,
File "/usr/local/lib/python3.8/site-packages/couchdb/http.py", line 295, in request
    conn = self.connection_pool.get(url)
File "/usr/local/lib/python3.8/site-packages/couchdb/http.py", line 515, in get
    conn.connect()
File "/usr/local/lib/python3.8/http/client.py", line 921, in connect
    self.sock = self._create_connection(
File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
    raise err
File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)ConnectionRefusedError:
[Errno 111] Connection refused

My Guess

This error only occurs when building (`docker-compose up --build -d`): Once built, I can manually enter my server container execute the same command without any error. The user db is then created and my server up and running.

Considering this, I suspect it might be the case that my couchdb container is not fully created when the server container tries to create the new db which yields the error.

Any ideas as to how solve this problem?

EDIT

# .env
COUCHDB_USER=admin
COUCHDB_PASSWORD=superSECRET!
COUCHDB_HOST=couchdb
COUCHDB_PORT=5984

Answer 1

@CyrilG.'s answer, while unfortunately not quite solving the issue, pointed me in the right direction:

There used to be a condition parameter within depends_on where one could pass service_healthy. Unfortunately, this has been deprecated as of v.3.

The possible solutions are:

• Restart the service a couple of times until it works as explained here
• Modify the service itself to handle such problems as explained here

I've opted for option number two. This slight modification of app.py solved the problem:

...
import time
...

def _get_or_create_db(
        _client: couchdb.Server, 
        db_name: str, 
        max_retries: int = 3, 
        retry_wait: int = 10

) -> couchdb.Database:
    retry = 0
    while retry <= max_retries:
        try:
            if db_name in _client:
                return _client[db_name]
            print("Creating DB", db_name)
            return _client.create(db_name)
        except ConnectionRefusedError:
            retry += 1
            time.sleep(retry_wait)

Answer 2

Your database may not be started yet when Python try to connect. Add the dependency in the server like this:

depends_on: 
  - couchdb