Promtail - service discovery based on label with docker-compose and label in Grafana log explorer

Question

I am playing with Grafana and Promtail. I have the following setup:

version: "3.3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:k88-c660a7e
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:k88-c660a7e
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /var/run/docker.sock:/var/run/docker.sock
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-config.yaml:/etc/grafana/provisioning/datasources/default.yaml
    networks:
      - loki

  mycontainer: # Prints debug output to stdout
    build: .
    labels:
      - "mylabel=true"
    networks:
      - loki

What I want to achieve:

Have Promtail ignore all other containers except ones that have the mylabel=true
Have Grafana display a label with the container name as label in the log explorer

My current promtail-config.yaml scrape-configs:

scrape_configs:
- job_name: containers
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
  relabel_configs: # For some reason this drops all logs
    - source_labels: [__meta_docker_container_label_mylabel]
      regex: "true"
      action: keep
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__:  /var/lib/docker/containers/*/*-json.log

  pipeline_stages:
    - docker:

How do I change it to achieve that?

EDIT: This is probably going to come up - support for docker_sd_configs option is not officially released yet, but I have confirmed with a contributor to the project that it is available and in fact running in their cloud.

score 1 · Answer 1 · answered Jul 21 '22 at 10:19

Use filters for docker labels selection. Use relabel_configs for setting additional labels based on meta values

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /var/run/promtail/positions.yaml

clients:
  - url: https://[YOUR_LOKI_SERVER_NAME]/loki/api/v1/push
    tenant_id: [YOUR_TENANT_ID]

scrape_configs:
- job_name: containers
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 15s
      filters:
        - name: label
          values: ["mylabel=true"]

  pipeline_stages:
    - docker: {}
    - static_labels:
        job: "promtail"
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)'
      target_label: 'container'

Thanks, Tuan. Is there any way we can add filters as wild characters? like if I want to search for anything with `values: ["mylabel=true*"]` — Kermit the Frog, Mar 23 '23 at 22:52

Promtail - service discovery based on label with docker-compose and label in Grafana log explorer

What I want to achieve:

1 Answers1