Restart dockerd in Gitlab CI (docker-in-docker, dind)

Question

I want to setup Gitlab CI/CD on my project which involves:

• 1 x docker-compose.yml for the whole project
• n x Dockerfile for each micro-services.

The thing is, I have one service that uses GPU computing with CUDA which means:

• One dockerfile uses: FROM nvidia/cuda:10.1-devel-ubuntu18.04
• The docker-compose.yml involves runtime: nvidia :

version: '3'
services:

  [other services...]

  my_service:
    runtime: nvidia
    build:
      context: ...
    environment:
      - ...

It works well on machines that have nivida-docker. However, I do not plan on using GPU during Gitlab CI so I looked for a workaround to keep the same docker-compose.yml but not raising error on runtime: nvidia and found this /etc/docker/daemon.json trick to define a fake nvidia runtime that is actually runc.

The trick actually works on my ubuntu machine i.e. I can see the fake runtime after sudo service docker restart; docker info

However I do not manage to restart docker so that it takes the new daemon.json into account.

I tried to start dockerd manually with dockerd but it errors, saying that dockerd isn't define. When I'm just commenting runtime: nvidia it all works like a charm which I dont understand how considering dockerd is missing?

Could you point out a wait to take daemon.json into account on gitlab-ci (alpine linux) ?

Answer 1

As pointed in the documentation, you can mount a daemon.json in every dind instances, using the config.toml configuration file.

They propose you yo host your custom daemon.json in /opt/docker/daemon.json, this would be the one containing your runc runtime.

Then, you have to mount that in each container's /etc/docker/daemon.json, in the config.toml:

[[runners]]
  ...
  executor = "docker"
  [runners.docker]
    image = "alpine:3.12"
    privileged = true
    volumes = ["/opt/docker/daemon.json:/etc/docker/daemon.json:ro"]

As for the location of this file:

You can find the config.toml file in:

• /etc/gitlab-runner/ on *nix systems when GitLab Runner is executed as root (this is also the path for service configuration)
• ~/.gitlab-runner/ on *nix systems when GitLab Runner is executed as non-root
• ./ on other systems

^{Source: https://docs.gitlab.com/runner/configuration/advanced-configuration.html}