Why is Prometheus stack operator is not scraping metrics from an app?

Question

I have cluster for small demo. I need to collect metrics from cluster itself and application, that ran in.

I installed helm kube-prometheus-stack and can see cluster metrics.

Now I want to get metrics from application.

This is my setup:

Prometheus stack installed in monitoring namespace.
In Prometheus object I changed serviceMonitorNamespaceSelector and serviceMonitorSelector to {}.

In monitoring namespace I create ServiceMonitor object:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    serviceMonitorSelector: prometheus
    release: kube-prometheus-stack
  name: app-monitoring
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    targetPort: 8080
    path: /metrics
  selector:
    matchLabels:
      app: 'demo'
  namespaceSelector:
    matchNames:
    - app

My app service definition is:

apiVersion: v1
kind: Service
metadata:
  name: app-service
  namespace: app
  labels:
    app: 'demo'
spec:
  selector:
    app: demo
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

Also I have ingress, deployment and so on.

First of all I check that ingres -> service -> pod works - I can successfully access to my public address and get data from service and /metrics response like that:

# HELP dotnet_collection_count_total GC collection count
# TYPE dotnet_collection_count_total counter
dotnet_collection_count_total{generation="0"} 197
dotnet_collection_count_total{generation="1"} 17
dotnet_collection_count_total{generation="2"} 2
...

Then I check that service work with port forwarding — forward my local 8080 to 8080 of service — and it also work fine.

But in Prometheus UI in targets I don't see my target.

In configuration of Prometheus I see that job for my service exists:

rule_files:
- /etc/prometheus/rules/prometheus-kube-prometheus-stack-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: serviceMonitor/monitoring/app-monitoring/0
  honor_timestamps: true
  track_timestamps_staleness: false
  scrape_interval: 30s
  scrape_timeout: 10s
  scrape_protocols:
  - OpenMetricsText1.0.0
  - OpenMetricsText0.0.1
  - PrometheusText0.0.4
  metrics_path: /metrics
  scheme: http
  enable_compression: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (app);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_container_port_number]
    separator: ;
    regex: "8080"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: "8080"
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - app

I don't see any other requirements in documentation. So my question is: what have I missed?

Update: I also checked service discovery status in Prometheus UI:

Discovered Labels   Target Labels
__address__="10.112.129.15:8080"
__meta_kubernetes_endpoint_address_target_kind="Pod"
__meta_kubernetes_endpoint_address_target_name="app-deployment-7f8955b777-s4b4z"
__meta_kubernetes_endpoint_node_name="cl1kl0a11gnnk75osdm3-ysaj"
__meta_kubernetes_endpoint_port_protocol="TCP"
__meta_kubernetes_endpoint_ready="true"
__meta_kubernetes_endpoints_label_app="demo"
__meta_kubernetes_endpoints_labelpresent_app="true"
__meta_kubernetes_endpoints_name="app-service"
__meta_kubernetes_namespace="app"
__meta_kubernetes_pod_controller_kind="ReplicaSet"
__meta_kubernetes_pod_controller_name="app-deployment-7f8955b777"
__meta_kubernetes_pod_host_ip="10.129.0.26"
__meta_kubernetes_pod_ip="10.112.129.15"
__meta_kubernetes_pod_label_app="demo"
__meta_kubernetes_pod_label_pod_template_hash="7f8955b777"
__meta_kubernetes_pod_labelpresent_app="true"
__meta_kubernetes_pod_labelpresent_pod_template_hash="true"
__meta_kubernetes_pod_name="app-deployment-7f8955b777-s4b4z"
__meta_kubernetes_pod_node_name="cl1kl0a11gnnk75osdm3-ysaj"
__meta_kubernetes_pod_phase="Running"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="69cfa006-ae78-48f6-8d46-b781bfb702b2"
__meta_kubernetes_service_annotation_kubectl_kubernetes_io_last_applied__meta_kubernetes_service_label_app="demo"
__meta_kubernetes_service_labelpresent_app="true"
__meta_kubernetes_service_name="app-service"
__metrics_path__="/metrics"
__scheme__="http"
__scrape_interval__="30s"
__scrape_timeout__="10s"
job="serviceMonitor/monitoring/app-monitoring/0"

So the IP is discovered correctly.

Why is Prometheus stack operator is not scraping metrics from an app?

0 Answers0