Docker process not running, and any interaction with docker fails

Question

To preface this, I am running docker in a Ubuntu 20.04 VM on virtualbox.

I created a simple shell script to kill any process running on port 9042, then start my docker-compose file. Here is the script in question:

#!/bin/bash

# Check for and kill any processes running on port 9042
sudo kill -9 $(sudo lsof -t -i:9042)

# start docker-compose
docker-compose -f ./docker/docker-compose.yml up

Since running that, however, it has made my docker installation completely unresponsive to any sort of interaction. Any docker commands will hang indefinitely until cancelled with Ctrl+C, and any other system commands that use docker (such as sudo service docker start) will also hang indefinitely.

If I try to run dockerd, it fails with the message failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid. As my system reports that docker is not running, I go ahead and delete var/run/docker.pid. If I then try to run dockerd again, I get a different error message: failed to start daemon: error while opening volume store metadata database: timeout.

At this stage, some of the docker commands start working again. docker version and docker help both work, but it is still reported that the docker daemon is not running. Attempting to run docker-compose up on a docker-compose file produces this output:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/david/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/david/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.local/lib/python3.8/site-packages/docker/api/client.py", line 205, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/david/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/david/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/david/.local/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.local/bin/docker-compose", line 8, in <module>
    sys.exit(main())
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/main.py", line 67, in main
    command()
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/main.py", line 123, in perform_command
    project = project_from_options('.', options)
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/command.py", line 60, in project_from_options
    return get_project(
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/command.py", line 131, in get_project
    client = get_client(
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/docker_client.py", line 41, in get_client
    client = docker_client(
  File "/home/david/.local/lib/python3.8/site-packages/compose/cli/docker_client.py", line 170, in docker_client
    client = APIClient(**kwargs)
  File "/home/david/.local/lib/python3.8/site-packages/docker/api/client.py", line 188, in __init__
    self._version = self._retrieve_server_version()
  File "/home/david/.local/lib/python3.8/site-packages/docker/api/client.py", line 212, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Other system commands such as sudo service docker start still hang indefinitely until killed.

I have tried every single solution in this thread (Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?) and this one (Docker commands do not respond anymore), but none of them work.

Does anyone know what could be the issue here?

EDIT: A few more points -

The docker.pid file reappears again when I restart my VM
Restarting my VM does not do anything to remedy the problem
Executing commands as the root user likewise doesn't do anything
Trying to reinstall docker using sudo apt-get install --reinstall docker-ce also hangs at the stage Preparing to unpack .../docker-ce_5%3a20.10.0~1.1.beta1-0~ubuntu-focal_amd64.deb ...

@Stefano An instance of Java from my local cassandra installation. Because my containers run something else on the same port, I need to free that up. — AgentNo, Oct 28 '20 at 10:37
I suppose Cassandra is running as a service, why not simply stopping it instead of brutally murdering it? Moreover this approach might work the first time but the second time it's going to kill the `docker-proxy` process which is used to map the ports. I can't really say what's going to happen then — Stefano, Oct 28 '20 at 10:41
That is a fair point - and I've recognised this is an issue, and removed this script from my project. I don't plan on using this script again. — AgentNo, Oct 28 '20 at 10:43

score 0 · Accepted Answer · answered Dec 07 '21 at 02:33

I know this is super late, but I found an answer from another similar question I asked.

Docker containers are stored in the default location at /var/lib/docker/ on Linux. I was able to identify the container that was causing the issue and deleted the actual container files. I then used the CLI to remove all other traces of the container, and docker was able to start running as normal.

Obviously doing this is risky, so make sure you take adequate steps to back up your machine first.

Docker process not running, and any interaction with docker fails

1 Answers1