Gitlab CI runner not able to expose ports of nested Docker containers

Question

When using GitLab CI, as well as the gitlab-ci-multi-runner, I'm unable to get internally-started Docker containers to expose their ports to the "host", which is the Docker image in which the build is running.

My .gitlab-ci.yml file:

test:
  image: docker
  stage: test
  services:
    - docker:dind
  script:
    - APP_CONTAINER_ID=`docker run -d --privileged -p "9143:9143" appropriate/nc nc -l 9143`
    - netstat -a
    - docker exec $APP_CONTAINER_ID netstat -a
    - nc -v localhost 9143

My command:

gitlab-ci-multi-runner exec docker --docker-privileged test

The output:

$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 runner--project-1-concurrent-0:54664 docker:2375             TIME_WAIT
tcp        0      0 runner--project-1-concurrent-0:54666 docker:2375             TIME_WAIT
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ docker exec $APP_CONTAINER_ID netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:9143            0.0.0.0:*               LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ nc -v localhost 9143
ERROR: Build failed: exit code 1
FATAL: exit code 1

What am I doing wrong here?

Original Question Follows - above is a shorter, easier-to-test example

I have an application image that listens on port 9143. Its startup and config is managed via docker-compose.yml, and works great on my local machine with docker-compose up - I can access localhost:9143 without issue.

However, when running on GitLab CI (the gitlab.com version) via a shared runner, the port doesn't seem to be exposed.

The relevant portion of my .gitlab-ci.yml:

test:
  image: craigotis/buildtools:v1
  stage: test
  script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com/craigotis/myapp
    - docker-compose up -d
    - sleep 60 # a temporary hack to get the logs
    - docker-compose logs
    - docker-machine env
    - docker-compose port app 9143
    - netstat -a
    - docker-compose ps
    - /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60
    - cd mocha
    - npm i
    - npm test
    - docker-compose down

The output is:

$ docker-compose logs
...
app_1  | [Thread-1] INFO spark.webserver.SparkServer - == Spark has ignited ...
app_1  | [Thread-1] INFO spark.webserver.SparkServer - >> Listening on 0.0.0.0:9143
app_1  | [Thread-1] INFO org.eclipse.jetty.server.Server - jetty-9.0.z-SNAPSHOT
app_1  | [Thread-1] INFO org.eclipse.jetty.server.ServerConnector - Started ServerConnector@6919dc5{HTTP/1.1}{0.0.0.0:9143}
...

$ docker-compose port app 9143
0.0.0.0:9143

$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53646 docker:2375             TIME_WAIT   
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53644 docker:2375             TIME_WAIT   
tcp        0      0 runner-e11ae361-project-1925166-concurrent-0:53642 docker:2375             TIME_WAIT   
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

$ docker-compose ps
stty: standard input: Not a tty
    Name                  Command               State                Ports               
----------------------------------------------------------------------------------------
my_app_1   wait-for-it.sh mysql_serve ...   Up      8080/tcp, 0.0.0.0:9143->9143/tcp 
mysql_server   docker-entrypoint.sh --cha ...   Up      3306/tcp     

$ /usr/local/bin/wait-for-it.sh -h localhost -p 9143 -t 60
wait-for-it.sh: waiting 60 seconds for localhost:9143
wait-for-it.sh: timeout occurred after waiting 60 seconds for localhost:9143

The contents of my docker-compose.yml:

version: '2'

networks:
    app_net:
        driver: bridge

services:
    app:
        image: registry.gitlab.com/craigotis/myapp:latest
        depends_on:
        - "db"
        networks:
        - app_net
        command: wait-for-it.sh mysql_server:3306 -t 60 -- java -jar /opt/app*.jar
        ports:
        - "9143:9143"

    db:
        image: mysql:latest
        networks:
        - app_net
        container_name: mysql_server
        environment:
        - MYSQL_ALLOW_EMPTY_PASSWORD=true

It seems like my application container is listening on 9143, and it's properly exposed to the shared GitLab runner, but it doesn't seem to actually be exposed. It works fine on my local machine - is there some special workaround/tweak I need to make this work inside a Docker container running on GitLab?

on your local machine on which host address is it running.. and what is your os on localmachine — Rohit Dhiman, Jan 27 '17 at 11:56
I have produced a minimal working example, for others to use: https://gitlab.com/mslot/dind.sandbox — mslot, Sep 22 '21 at 13:52

score 17 · Answer 1 · answered Jan 16 '18 at 19:32

17

When using docker:dind a container is created and your docker-compose containers get setup within it. It exposes the ports to localhost within the docker:dind container. You cannot access this as localhost from the environment that your code is executing in.

A hostname of docker is setup for you to reference this docker:dind container. You can check by using cat /etc/hosts.

Instead of referencing localhost:9143 you should use docker:9143.

answered Jan 16 '18 at 19:32

Jason Prawn

1,003
11
20

1

I have checked /etc/hosts, and indeed only the special name "docker" was available (no "mysql", etc.). – admirabilis Jan 05 '19 at 00:14
should it also work if I start an internal container with `docker run`? Because nothing with `docker` works for me (inside GitLab CI). – Dmitriy Popov Feb 20 '20 at 17:38
I also checked `cat /etc/hosts`. It does not contain `docker` name. – Dmitriy Popov Feb 20 '20 at 17:55

score 13 · Accepted Answer · edited Jun 20 '20 at 09:12

13

The offical gitab-ci on gitlab.com documentation refers to the example of PostgreSQL

Its working CI does not try to connect to localhost, but rather to the service name

The services keyword defines just another docker image that is run during your build and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

The service container for MySQL will be accessible under the hostname mysql.
So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost.

You could check if this applies in your case, and try accessing your application service in app:9143 instead of localhost:9143.

edited Jun 20 '20 at 09:12

Community

1
1

answered Jan 12 '17 at 05:27

VonC

1,262,500
529
4,410
5,250

4

I'm actually using `services` in my `.gitlab-ci.yml` for other things like `docker:dind`. The reason I want to use `docker-compose` for the testing phase however, is so that the orchestration of my MySQL, application, and other related images (in a test environment) doesn't need to be duplicated in `docker-compose.yml` and `.gitlab-ci.yml`. I already have a working Compose file that I've spent some time perfecting - and that works on developer machines (whereas `.gitlab-ci.yml` is of no use), so I'd really like to reuse it if possible. – Craig Otis Jan 12 '17 at 12:57
@CraigOtis I agree, but, just for testing, could you check if using the service name works better? – VonC Jan 12 '17 at 13:06
Unfortunately after a closer look, I can't run my images as services this way - they require specific environment variables to be set for test-related config, and my app image (as an example) needs to wait for MySQL to be up and running before it in turn launches. (And it seems the GitLab CI config requires that services just be an array of strings...) – Craig Otis Jan 12 '17 at 14:33
@CraigOtis Could your app start right way, and as a CMD, use a wait-for-it script in order to wait for SQL to start? (as in https://docs.docker.com/compose/startup-order/) – VonC Jan 12 '17 at 14:42
My concern with that is that the app image is now tightly coupled to MySQL, whereas it's currently driven by an environment variable, `STORAGE_TYPE` which can be set to `memory`, `mysql`, or some other database. But I could definitely update the application _itself_ to poll on `3306` and not connect until that's reachable. That's probably a good improvement to make in any case - I will give that a try. – Craig Otis Jan 12 '17 at 15:21
@CraigOtis you could make your app dependent on https://github.com/zooniverse/docker-status, which would be in charge of the polling. Also, port polling is mentioned in https://donagh.io/docker-startup-order/ – VonC Jan 12 '17 at 16:58
Nope, I can't seem to get this to work for the life of me. Whether it's running as a service, or I try to start it via docker or docker-compose, I can just never connect to the service, _only_ when running on GitLab CI. Running locally it always works perfectly. – Craig Otis Jan 16 '17 at 21:56
Note that I *can* open ports on this machine - if I just run `java -jar myapp.jar`, I can see _and connect to_ port 9143 on `localhost`. I just can't seem to open any ports when I'm running the image via Docker or Docker Compose. – Craig Otis Jan 18 '17 at 12:02
@CraigOtis Is it a port-forwarding issue between your host and your Linux VM? (as in http://stackoverflow.com/a/37771161/6309 or http://stackoverflow.com/questions/35642821/not-able-to-access-tomcat-application-on-docker-vm-with-hostwindows-ip-while-u/35646587#35646587) – VonC Jan 18 '17 at 12:04
The thing is, `localhost` is already a Docker container. The `.gitlab-ci.yml` file specifies a Docker image within which all my commands are run. So my application image is **two** levels deep. I just want to access it from the **first** level container, which is the one created in GitLab CI for the purpose of running my build. – Craig Otis Jan 18 '17 at 12:10
@CraigOtis still what is your host OS and docker flavor? – VonC Jan 18 '17 at 12:11
@CraigOtis maybe you need some kind of iptable setting (as in http://stackoverflow.com/a/14637461/6309, for ssh but could apply for other ports) – VonC Jan 18 '17 at 12:12
I don't know what the host OS is - it's running on GitLab's infrastructure. My build is running inside an `ubuntu:latest` image. – Craig Otis Jan 18 '17 at 12:21
@CraigOtis Would https://gitlab.com/gitlab-org/gitlab-ce/issues/22707 help? – VonC Jan 18 '17 at 12:40
That seems to be related to the external port on which the GitLab service itself (self-hosted) listens, not to the internal builder images and their own Docker/port config. – Craig Otis Jan 18 '17 at 13:19
Did you run your containers binding the docker.sock? (https://nathanleclaire.com/blog/2014/07/12/10-docker-tips-and-tricks-that-will-make-you-sing-a-whale-song-of-joy/) – VonC Jan 18 '17 at 13:27
Thanks VonC, my question is updated with a more simple example. – Craig Otis Jan 20 '17 at 20:06
@CraigOtis OK I'll have a look – VonC Jan 20 '17 at 20:08

score 2 · Answer 3 · answered Jul 20 '21 at 08:57

In case you GitLab CI Runner run with Docker-executor via socket binding use host.docker.internal-host, because your app is running on the host machine but not on the guest.

# Run and ssh into dind-container
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock docker:20.10-dind /bin/sh

# Next commands run in dind-container

docker run --rm -d -p 8888:80 nginx:alpine
...

wget -qO- "http://localhost:8888/"
wget: can't connect to remote host (127.0.0.1): Connection refused

wget -qO- "http://127.0.0.1:8888/"
wget: can't connect to remote host (127.0.0.1): Connection refused

wget -qO- "http://docker:8888/"
wget: bad address 'docker:8888'

wget -qO- "http://host.docker.internal:8888/"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Rohit Dhiman · Answer 4 · 2017-01-27T11:48:11.840

Your docker-compose.yml seems to be ok.

But I think there is error with your ip or port routing. As I can see from your shared info your app is running on port 9143 on ip 0.0.0.0 as 0.0.0.0:9143.

and you are accessing it as localhost:9143, which can be interpreted as 127.0.0.1:9143.

According to this.

127.0.0.1 is the loopback address (also known as localhost).
0.0.0.0 is a non-routable meta-address used to designate an invalid, unknown, or non-applicable target (a ‘no particular address’ place holder).

Can you try to run your app on 127.0.0.1:9143 then share the result.

UPDATE

or you can use service to run it by service name as documentation suggest:

The services keyword defines just another docker image that is run during your build and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

The service container for MySQL will be accessible under the hostname mysql. So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost.

I believe your understanding regarding 0.0.0.0 is wrong, continue reading the article you linked, and you will find "In the context of servers, 0.0.0.0 means all IPv4 addresses on the local machine." In this context it just means "listen to all interfaces" (a server that binds to 127.0.0.1 not reachable from outside the machine). — Kjell Andreassen, Mar 25 '21 at 22:28

score 0 · Answer 5 · answered Jan 16 '17 at 22:57

0

Usually a docker machine won't run on localhost, but on a docker host with some other ip address. Try using docker-machine ip to get your docker host ip.

answered Jan 16 '17 at 22:57

Yoav Aharoni

2,672
13
18

The issue isn't really that I determine the hostname it's running on - it's that whether it's running on `localhost` through the use of Docker Compose or regular Docker with `-p "...:..."`, or whether I configure my application image to run as a `service:` in the GitLab config, when it's actually running on GitLab CI, the application image is not reachable - even though it works fine on my local machine, and seems to start fine (and remain running) according to the application logs. – Craig Otis Jan 17 '17 at 00:56
Maybe I'm not following you (never used GitLab CI), but I'm just saying that since any docker client can be configured to work against a VM or any other remote docker host (docker-machine env is used to set the DOCKER_HOST), it could be that GitLab's docker client is configured to run against a host other then localhost, in which case, the container is created and runs, but on a different host, with a different ip (and won't be available on localhost, which you are waiting for). So, just wondering if you tried checking that. – Yoav Aharoni Jan 17 '17 at 01:25
Thanks Yoav, my question is updated with a more simple example. – Craig Otis Jan 20 '17 at 20:06

Gitlab CI runner not able to expose ports of nested Docker containers

5 Answers5

Linked