1

I'm running a number of docker containers on my Ubuntu server. They are controlled by systemd.

If I run systemctl stop docker-startup@mw-salcom-app.service the container shuts down gracefully. The output from journalctl is as follows (output in reverse time, newest on top):

Nov 05 10:58:11 ubox0 systemd[1]: Stopped Docker container startup for mw/salcom/app.
Nov 05 10:58:11 ubox0 docker[3622]: mw-salcom-app
Nov 05 10:58:09 ubox0 docker[3470]: [Thu Nov 05 10:58:09.921033 2020] [mpm_prefork:notice] [pid 1] AH00169: caught SIGTERM, shutting down
Nov 05 10:58:09 ubox0 systemd[1]: Stopping Docker container startup for mw/salcom/app...

If I however execute shutdown -h now, I get:

Nov 05 10:51:04 ubox0 systemd[1]: Stopped Docker container startup for mw/salcom/app.
Nov 05 10:51:04 ubox0 systemd[1]: docker-startup@mw-salcom-app.service: Failed with result 'timeout'.
Nov 05 10:51:04 ubox0 systemd[1]: docker-startup@mw-salcom-app.service: Main process exited, code=killed, status=9/KILL
Nov 05 10:51:04 ubox0 systemd[1]: docker-startup@mw-salcom-app.service: Killing process 2077 (docker) with signal SIGKILL.
Nov 05 10:51:04 ubox0 systemd[1]: docker-startup@mw-salcom-app.service: State 'stop-sigterm' timed out. Killing.
Nov 05 10:50:04 ubox0 systemd[1]: docker-startup@mw-salcom-app.service: Stopping timed out. Terminating.
Nov 05 10:49:04 ubox0 systemd[1]: Stopping Docker container startup for mw/salcom/app...

The container isn't stopped and the process eventually times out after 2 minutes and the container is aborted.

What is the reason for this?

I'd like my containers to shut down gracefully so that the services running in them are stopped without loss of data.

My service file is:

[Unit]
Description=Docker container startup for %I
Requires=docker.service
After=docker.service

[Service]
TimeoutStartSec=40
Restart=always


wait a while before restarting in case we are doing a restore;


this includes a 'docker stop' command after we restore the data


RestartSec=60
ExecStartPre=-/usr/bin/docker stop %i
ExecStart=/usr/bin/docker start -a %i
ExecStop=/usr/bin/docker stop %i
TimeoutStopSec=60


[Install]
WantedBy=multi-user.target

Update After I searched further, I found that post How to properly handle a Docker container as a systemd service? describes the same problem, but there is no answer there.

NZD
  • 2,818

1 Answers1

0

It seems that docker and systemd don't always get along very well, see:

I have the exact same set-up on a different machine (my backup server) and here everything works fine. I can execute the shutdown command and all docker containers are shut down gracefully.

So the problem might be a dependency problem. Maybe systemd has shut down some service that is needed by the docker containers so they can be shut down properly.

The docker version on the system that 'fails' is 18.09.2, on the system that 'works' it is 18.06.1-ce.

The release notes for docker version 18.09 state:

In Docker versions prior to 18.09, containerd was managed
by the Docker engine daemon. In Docker Engine 18.09, 
containerd is managed by systemd.

When I added the following lines to my unit files:

[Unit]
...
Requires=containerd.service
After=containerd.service

and reloaded the unit files (systemctl daemon-reload), things started working again. My containers were not terminated after the 1 minute time-out that I set, but shut down after a few seconds.

I tried upgrading docker to version 20.10.1 first, to no avail.

I got the idea for this 'solution' when I came across: https://github.com/sous-chefs/docker/issues/1062

NZD
  • 2,818