Prometheus target down at localhost:9323 for docker matrices

Question

I am running Prometheus inside docker container. I want to track metrics of docker. My docker command to run Prometheus is,

docker run -d --name prometheus \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml  -p 9090:9090\
    prom/prometheus

My prometheus.yml file contains.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  external_labels:
      monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'


    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'

    static_configs:
      - targets: ['localhost:9323']

And I have updated daemon.json for docker too. It contains,

{
  "data-root": "/docker-root2/docker",
  "metrics-addr" : "127.0.0.1:9323",
  "experimental": true
}

And I also tried to change it to,

{
  "data-root": "/docker-root2/docker",
  "metrics-addr" : "0.0.0.0:9323",
  "experimental": true
}

I have changed docker root directory to "data-root": "/docker-root2/docker", This doesn't have any connection with question.

Followed docker official tutorial link but this is not working for me. But still I am getting status Down for target docker

Get "http://localhost:9323/metrics": dial tcp 127.0.0.1:9323: connect: connection refused

I want to get the expected result of target status to UP. Further I want to get docker matrices.

`localhost` inside of container refers to container, and not docker host. Read more [here](https://stackoverflow.com/a/24326540/21363224) — markalex, Apr 05 '23 at 10:11
Does this answer your question? [From inside of a Docker container, how do I connect to the localhost of the machine?](https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach) — markalex, Apr 05 '23 at 10:12
I am understand this and I have tried this solution too. But there is no extra_host thing for prometheus. Can you please tell me where to add this extra_hosts ? — mhassaankhokhar, Apr 05 '23 at 10:14
`extra_hosts` mentioned in linked answer is a part of `docker-compose.yml`. Since you don't use one either recreate container with `--add-host host.docker.internal:host-gateway` or use hosts outer address (like 192.168.1.100), or use hosts inner docker address (like 172.17.0.1). Read linked answer thoroughly, and you'll find best variant for you. — markalex, Apr 05 '23 at 10:19
I have changed my command accordingly, like `docker run -d --name prometheus \ --add-host host.docker.internal:host-gateway -v ./prometheus.yml:/etc/prometheus/prometheus.yml -p 9090:9090 prom/prometheus` but still status is down. — mhassaankhokhar, Apr 05 '23 at 10:22
Yes, I tried my system IP as target address too. But that thing also doesn't work. — mhassaankhokhar, Apr 05 '23 at 10:24
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/252986/discussion-between-mhassaankhokhar-and-markalex). — mhassaankhokhar, Apr 05 '23 at 10:25

score 0 · Accepted Answer · edited Apr 05 '23 at 10:39

I have solved this problem by changing two things. I was running Prometheus inside container, so it doesn't have access to the local machine through localhost. For that I updated my prometheus.yml file and changed,

static_configs:
      - targets: ['localhost:9323']

to my host(system) IP address.

static_configs:
      - targets: ['192.168.8.123:9323']

Second thing, I recreated container with an extra option --add-host:

docker run -d --name prometheus \
    --add-host host.docker.internal:host-gateway \
    -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
    -p 9090:9090 \
    prom/prometheus

Prometheus target down at localhost:9323 for docker matrices

1 Answers1