Docker login working only with daemon but not with Docker active service

Question

I have been trying to set up nexus3 as private registry for the Docker images with nginx serving as a reverse proxy for nexus3.

I have been encountering all types of error ranging from Forbidden to Connection Refused and have tried all reference steps.

Below are my configurations.

I have set up a self-signed certificate (orgnexus.crt) with key (orgnexus.key) and used the same as reverse proxy with nginx.

Nexus is configured with port 4444 and the docker hosted repo configured with https port 6666. Nexus is configured to run on "server908" while docker is running on "server446"

Below is the configuration for nginx.

server {

    listen 6666;
    server_name server908.int.org.com;
    keepalive_timeout 60;

    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;

    client_max_body_size 1G;
    chunked_transfer_encoding on;

    location /v2/ {

            access_log              /var/log/nginx/docker.log;
            proxy_set_header        Host $http_host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto "https";
            proxy_pass              http://server908.int.org.com:4444/;
            proxy_read_timeout      90;
}

    location / {

            access_log              /var/log/nginx/docker.log;
            proxy_set_header        Host $http_host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto "https";
            proxy_pass              http://server908.int.org.com:4444/;
            proxy_read_timeout      90;
    }
}

The same cert (orgnexus.crt) has been imported to the docker server at location /etc/docker/certs.d/server.int.org.com:6666.

[root@server446 server908.int.org.com:6666]# ls -ltr
-rwxr-xr-x. 1 root root 2139 May 31 12:07 ca.crt

Copied the nexus cert file (orgnexus.crt) to "server446" at location /etc/pki/ca-trust/source/anchors/server.int.org.com.crt

We are running docker version 1.12.6 and nexus 3.2.1-01.

The issue here comes when the docker service is active.

Active: active (running) since Thu 2017-06-01 20:20:36 HKT; 15h ago
[root@server446 ~]# docker login server908.int.org.com:6666
Username (admin): admin
Password:
Error response from daemon: Get https://server908.int.org.com/v2/: Forbidden

But when we run docker daemon manually with the docker service down, the same command goes through successfully.

[root@server446 ~]# docker login server908.int.org.com:6666
Username (admin): admin
Password:
Login Succeeded

"dockertest" is the username at "server446" .

Below is the docker info logs.

When Daemon run manually with Docker client service down

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-253:8-131425-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 24.77 MB
 Data Space Total: 107.4 GB
 Data Space Available: 8.877 GB
 Metadata Space Used: 602.1 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 7.64 GiB
Name: server446.int.org.com
ID: 3EMU:FC6L:454V:5WN4:BUXQ:AUVM:3E7D:T2TD:7G4M:AHS4:NAJ6:42OS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 16
 Goroutines: 23
 System Time: 2017-06-05T12:04:47.223159468+08:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 server908.int.org.com:6666,x.y.z.232:6666
 127.0.0.0/8
Registries: docker.io (secure)

With Docker client service up.

Containers: 26
 Running: 1
 Paused: 0
 Stopped: 25
Images: 12
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: vg00-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 4.749 GB
 Data Space Total: 8.623 GB
 Data Space Available: 3.874 GB
 Metadata Space Used: 2.019 MB
 Metadata Space Total: 134.2 MB
 Metadata Space Available: 132.2 MB
 Thin Pool Minimum Free Space: 861.9 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge overlay null host
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 7.64 GiB
Name: server446.int.org.com
ID: 3EMU:FC6L:454V:5WN4:BUXQ:AUVM:3E7D:T2TD:7G4M:AHS4:NAJ6:42OS
Docker Root Dir: /app/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 24
 Goroutines: 33
 System Time: 2017-06-05T12:02:26.153754978+08:00
 EventsListeners: 0
Http Proxy: http://10.10.120.98:3128
Https Proxy: http://10.10.120.98:3128
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 server908.int.org.com:6666,x.y.z.232:6666
 127.0.0.0/8
Registries: docker.io (secure)

Within daemon logs(/var/log/daemon.log) I notice this .

Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.837496046+08:00" level=debug msg="Calling POST /v1.24/auth"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.838840542+08:00" level=debug msg="form data: {\"password\":\"*****\",\"serveraddress\":\"server908.int.org.com:6666\",\"username\":\"admin\"}"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.842033018+08:00" level=info msg="{Action=auth, Username=dockertest, LoginUID=1960, PID=31397}"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.843339118+08:00" level=debug msg="AuthZ request using plugin rhel-push-plugin"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.845046779+08:00" level=debug msg="hostDir: /etc/docker/certs.d/server908.int.org.com:6666"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.875205677+08:00" level=debug msg="crt: /etc/docker/certs.d/server908.int.org.com:6666/ca.crt"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.875890502+08:00" level=debug msg="attempting v2 login to registry endpoint https://server908.int.org.com:6666/v2/"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.879260054+08:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://server908.int.org.com:6666/v2/: Forbidden"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.879318527+08:00" level=error msg="Handler for POST /v1.24/auth returned error: Get https://server908.int.org.com:6666/v2/: Forbidden"
Jun  5 13:37:41 server446 dockerd-current[31151]: time="2017-06-05T13:37:41.879342144+08:00" level=error msg="Handler for POST /v1.24/auth returned error: Get https://server908.int.org.com:6666/v2/: Forbidden"

Where I am going wrong? Is it something related to docker group permissions or nginx or the configurations? I suspect the request (from docker client machine) is not reaching to nginx (server908) and is revolving between docker client and daemon.

Check `docker info` and compare between the two ways you run the daemon. — BMitch, Jun 03 '17 at 12:08
@BMitch : does the daemon behave differently when run manually and with docker client? — Alim Azad, Jun 03 '17 at 14:18
It should not, you likely have different options being passed in the two scenarios. Please check `docker info` and post the results as edits to your question. — BMitch, Jun 03 '17 at 14:20
I think you should add the HTTP Basic Authentication that you want: ```auth_basic "Restricted"; auth_basic_user_file docker-registry.htpasswd; ``` — Ricardo Branco, Jun 03 '17 at 15:04
@RicardoBranco : doesn't the docker daemon look for http authentication...!! — Alim Azad, Jun 03 '17 at 16:05
@RicardoBranco : right..will try your solution...hoping it works — Alim Azad, Jun 03 '17 at 16:09
@BMitch : I have added the `docker info` for both daemon and client . I see a difference in the authorization section . Is this causing the issue? — Alim Azad, Jun 05 '17 at 04:23
First guess is the rhel-push-plugin plugin. My second guess is that you've defined the registry in the insecure section while configuring TLS certificates. The insecure registry definition is used for registries without TLS. — BMitch, Jun 05 '17 at 12:56
@BMitch I removed the plugin section but it didn't resolve the issue. For the insecure registry, I will remove it and get back with findings. — Alim Azad, Jun 05 '17 at 13:06
@BMitch : Thanks for you help . I am able to login now . Had to comment out the internet proxy within "/etc/sysconfig/docker". But now facing error while pushing . `error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: ` . Is it something related to nginx configuration ? — Alim Azad, Jun 06 '17 at 06:36
@RicardoBranco : Thanks . With your solution I was able to authenticate users for login to registry . But now I am encountering above mentioned `error parsing HTTP 400 response body: invalid character '<' looking for beginning of value` . Are my nginx configuration correct ? — Alim Azad, Jun 06 '17 at 06:38

score 0 · Accepted Answer · edited Apr 15 '19 at 01:44

I was able to solve the issue and was able to login after configuring nginx correctly referring docker login not working with nexus 3 private registry

Nexus HTTP Port : 8082

Docker Hosted Repo HTTP Port : 4444

Below is my nginx configuration .

    server {

    proxy_send_timeout 120;
    proxy_read_timeout 300;
    proxy_buffering    off;
    tcp_nodelay        on;

    server_tokens off;
    client_max_body_size 1G;

    listen 80;
    server_name server908.int.org.com;
    location / {
          rewrite ^(.*) https://server908.int.org.com$1 permanent;
    }
}

server {

    listen 443;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;

    location / {

      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:8082;
      proxy_read_timeout      90;

    }
}

# correlates to your nexus http connector
server {

    listen 6666;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1G;
    chunked_transfer_encoding on;

    location / {

      access_log              /var/log/nginx/docker.log;
      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:4444;
      proxy_read_timeout      90;

    }
}

Docker login working only with daemon but not with Docker active service

1 Answers1