How do I debug a "docker build ..." session that errs on the github key not being available?

Question

I'm having an issue with a docker build ... command which breaks when it tries to clone a repository from a private git. The error message says I don't have permissions:

#17 4.712 go: github.com/private/project@v1.0.0: reading github.com/private/project/go.mod at revision v1.0.0: git ls-remote -q origin in /go/pkg/mod/cache/vcs/<big-id>: exit status 128:
#17 4.712 git@github.com: Permission denied (publickey).
#17 4.712 fatal: Could not read from remote repository.
#17 4.712
#17 4.712 Please make sure you have the correct access rights
#17 4.712 and the repository exists.
------
executor failed running [/bin/sh -c cd cmd/service/ && go build -o service]: exit code: 1

I have all the points which I think are important

# syntax=docker/dockerfile:experimental
FROM golang:alpine AS build-env

RUN apk add build-base git openssh-client

RUN mkdir -p -m 700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

RUN --mount=type=ssh git clone git@github.com:myorg/myproject.git myproject

...

Then I run the build with a command that include --ssh

export DOCKER_BUILDKIT=1
docker build --rm=false --no-cache --ssh default --file ./Dockerfile \
       --build-arg GO_VERSION=1.16.3 --tag app:main .

The git clone ... fails with the error above.

Also the --rm=false has no effect because of the export DOCKER_BUILDKIT=1. So I don't know how to debug this. I can't see any containers, they get destroyed as soon as the docker command returns...

I tried varying the --ssh, just in case:

# The following gives me a permission error (as I expected)
... --ssh default=/home/alexis/.ssh/id_rsa ...

# The following is, I think, what happens by default, same git error
# (I verified the socket is under /run/user/1000/keyring/ssh
# as defined in the variable)
... --ssh github=$SSH_AUTH_SOCK ...

Also in the file, the mkdir was 600. 700 or 600 same difference.

So my question is: How do I debug that SSH issue knowing that the containers disappear and the setup doesn't want to work at all?

I've found a lot of info on many answers and in the docs, but it still doesn't work and there isn't much at all on how to really debug such a setup when it fails.

$ docker version
Client:
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d28e
 Built:             Wed Aug  4 21:24:10 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Wed Aug  4 21:26:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Alexis Wilke · Accepted Answer · 2021-11-10T22:32:19.830

Okay, I finally found one option that helps dramatically. The default docker output is neat and all, but it gives you one line of info in color and no details at all of what happened in your RUN command unless an error occurs (error which gets printed at the end).

In order to debug, you can at least get the output of each command using:

docker build --progress=plain ...

The neat output will be replaced with a complete set of lines including errors and standard output. Now I can add things such as:

RUN echo "Hello"

or a test such as:

RUN ssh -A -l git -v github.com

and get the full output.

The intermediate images & containers are still nowhere to be seen... but having output helps greatly.

As a side note, one thing I've discovered is that the --ssh default requires access to your ssh-agent socket and by default this is protected by apparmor. I had to edit the apparmor file and add the following:

/run/user/1000/keyring/ssh rw,

WARNING 1: This line is likely to disappear whenever the docker gets updated.

WARNING 2: You want to run the following command so the changes take effect immediately (or reboot):
sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.docker.docker

I also did a chmod to the /run/user/1000 and /run/user/keyring folders. Once I get things to actually work, I will try to not have those chmod changes... but I'm not holding my hopes up.

Okay, I got things to work!

The next issue was that the name of the key must be id_rsa. This is because the docker creates a root user and that user only has default key names. You may be able to create an .ssh/config file with a different name. But that can be problematic if you work with other people who would not use a key with no such name.

For additional details about the Dockerfile and other setup, see Building Go apps with private gitlab modules in Docker

How do I debug a "docker build ..." session that errs on the github key not being available?

1 Answers1

Linked