Zombie docker container that can't be killed

Question

I use docker-compose to create a bunch of containers and link them together. For some of the container definitions, I might have restart: always as the restart policy.

Now I have a postgres container that respawns back to life if stopped.

$docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
a8bb2b781630        postgres:latest     "docker-entrypoint.s…"   About an hour ago   Up About an hour    5432/tcp            dcat_postgres.1.z3pyl24kiq2n4clt0ua77nfx5
docker stop a8bb2b781630
a8bb2b781630
$ docker rm -f a8bb2b781630
a8bb2b781630
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
93fa7b72c2ca        postgres:latest     "docker-entrypoint.s…"   12 seconds ago      Up 4 seconds        5432/tcp            dcat_postgres.1.oucuo5zg3y9ws3p7jvlfztflb

Using docker-compose down in the dir that started the service doesn't work either.

$ docker-compose down
Removing dcat_postgres_1 ... done
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7ee7fb0e98cd        postgres:latest     "docker-entrypoint.s…"   13 seconds ago      Up 5 seconds        5432/tcp            dcat_postgres.1.jhv1q6vns1avakqjxdusmbb78

How can I kill a container and keep it from coming back to life?

EDIT: The container respawns even after restarting the Docker service.

Docker - 18.06.1-ce-mac73 (26764)

macOS High-Sierra, (10.13.6)

If the person downvoted could tell me what's wrong with the question maybe I could improve it. — yam, Sep 23 '18 at 06:09
Your containers are different in a couple of interesting ways (the hex ID, the generated name) and I don't think a Docker restart policy is recreating them. I don't think there's enough information here to guess what is actually doing it. — David Maze, Sep 23 '18 at 10:36
@DavidMaze, the generated name sure looks weird. But I thought the hex ID is normal, no? What info can I get that you think would be helpful? — yam, Sep 24 '18 at 17:17
@AlexKarshin, yes, you can see it in the code snippet above. — yam, Sep 24 '18 at 17:17
More specifically the name `dcat_postgres.1.z3pyl24kiq2n4clt0ua77nfx5` isn’t a format that Docker wil generate by default, and when the container is recreated, the container name is different. That’s why I think it isn’t Docker or your container doing the restart on its own, something else is noticing the container is gone and creating a new one. — David Maze, Sep 24 '18 at 17:50
@DavidMaze, docker-compose tends create it's own names but usually it's just `(directory of the docker-compose.yml)_(container name defined in docker-compose.yml)_(integer)`, but this doesn't fit that description. FYI, it restarts even after restarting the docker service. — yam, Sep 24 '18 at 18:12

score 14 · Accepted Answer · answered Sep 25 '18 at 07:43

14

I figured it out. Turns out it was related to docker swarm. I had experimented with it at some point without fully understanding what it is and what it does and apparently it just stayed there.

All I had to do was:

docker swarm leave --force

and it worked like a head-shot to an actual zombie.

answered Sep 25 '18 at 07:43

yam

1,383
3
15
34

1

This makes sense. Joining a swarm cluster allows the master nodes to control the docker daemon on the host machine. Glad you figured it out! – JackHacks Jun 25 '21 at 22:46

VonC · Answer 2 · 2021-03-24T12:07:48.700

11

Can you try an option like moby/moby issue 10032:

docker stop $(docker ps -a -q) &
docker update --restart=no $(docker ps -a -q) &
systemctl restart docker

(this assume here you have only one running container: the one you cannot prevent to start)

A docker rm -f should be enough though, unless you are using docker with a provision tool like Puppet.

As it turned out, another process (other than docker itself) was responsible for the container to restart (here docker swarm)

Update 2020/2021: For multiple containers, possibly without having to restart the docker daemon

docker ps -a --format="{{.ID}}" | \
 xargs docker update --restart=no | \
 xargs docker stop

Check if you need, as in the issue, remove the images as well ( | xargs docker rmi $(docker images -qa) --force)

edited Mar 24 '21 at 12:07

answered Sep 25 '18 at 06:32

VonC

1,262,500
529
4,410
5,250

There is no equivalent of `systemctl restart docker` in macOS so I had to manually restart the docker engine from the GUI. But the zombie container came back to life again. – yam Sep 25 '18 at 07:18
@yam That means you have another process/provision tool (or setting, like a .bashrc) which trigger the restart of the container. – VonC Sep 25 '18 at 07:41
turns out it was `docker swarm` that I had experimented with (see my answer above). Thanks for taking the time to answer though. – yam Sep 25 '18 at 07:44
@yam so I was right: another process was responsible for the container to restart? – VonC Sep 25 '18 at 07:48
Yes, it was another process but not a 3rd party app. – yam Sep 25 '18 at 07:53
@yam same general idea though. – VonC Sep 25 '18 at 07:56
*this assume here you have only one running container* - and what if you have more? – Simon Forsberg Mar 24 '21 at 11:38
@SimonForsberg You would need to filter your `docker ps`on the container name: `docker ps -f name='^$`, if you want to target one specific container. – VonC Mar 24 '21 at 11:40
@VonC Do I still have to perform the `systemctl restart docker` ? Won't that terminate all my other (perfectly healthy) docker containers? – Simon Forsberg Mar 24 '21 at 11:54
@SimonForsberg Apparently not: https://github.com/moby/moby/issues/10032#issuecomment-662515567 – VonC Mar 24 '21 at 12:05
@VonC I went for a full reboot. Not the ideal solution, but my patience with Docker ran out. – Simon Forsberg Mar 24 '21 at 12:10
@SimonForsberg I understand, that would work too ;) – VonC Mar 24 '21 at 12:11

score 0 · Answer 3 · answered Jul 04 '23 at 02:20

I ran into this issue when playing around with an Ansible deployment of Matrix. I just could not kill it until I did this

CONTAINERS=$(echo $(sudo docker ps | grep matrix | awk '{print $1;}'))
sudo docker container stop $CONTAINERS && sudo docker system prune -a

Here' I'm grepping on matrix but you could use as you like. It's really the prune right after the stop that does the zombie removal.

Zombie docker container that can't be killed

3 Answers3