6

I have main() function which spawns two separate sub-processes. These both sub-process shares metrics. How can I share metrics for both process and keep it updating? Here, is my snippet for more understanding.

from multiprocessing import Process
import prometheus_client as prom
from prometheus_client import Counter

# Metrics
c1 = prom.gauge('Counter1', 'Number of Request')
c2 = prom.gauge('Gauge1', 'Processing time in Seconds')

def process_abc():
  while True:
    #Some operations
    c1.set(some_value)
    c2.set(some_value)
    time.sleep(10)

def process_xyz():
   while True:
     #Some operations
     c1.set(some_value)
     c2.set(some_value)
     time.sleep(10)

def main():
  prom.start_http_server(8080)

  Process(target=process_abc).start()
  Process(target=process_xyz).start()

if __name__ == "__main__":
  main()

I was able to see metrics name at endpoint but count is always zero means it's never updated by sub-process.

Sahil Sangani
  • 109
  • 1
  • 2
  • 6

1 Answers1

1

The prometheus_client library documentation addresses this case:

Prometheus client libraries presume a threaded model, where metrics are shared across workers. This doesn't work so well for languages such as Python where it's common to have processes rather than threads to handle large workloads.

I won't copy here the explanation (which is geared toward gunicorn) use case but basically, you need to:

  • define an env variable with directory to use: since you are using Process yourself, you can set it in the code
os.environ["PROMETHEUS_MULTIPROC_DIR"] = "/path/to/writeable/tmp/"
  • each process must have its own collector, register it at start and unregister it at exit:
from prometheus_client import multiprocess

def called_from_process():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(CollectorRegistry())

def process_exit(process):
    if process.pid is not None:
        multiprocess.mark_process_dead(process.pid)

p = Process(target=f)
# f calls called_from_process
p.start()
p.join()
process_exit(process)

See the full documentation about how to handle gauge and any other quirk.

I expect that PROMETHEUS_MULTIPROC_DIRshould be cleaned at startup of your application to handle odd case where the application previous run was not able to do so.

Michael Doubez
  • 5,937
  • 25
  • 39
  • Quick question: If we have two separate registry for each process. How does these processes will update the same metrics? @Michael – Sahil Sangani Oct 15 '21 at 13:29
  • Sorry but still it's not working. It's not able to update metrics from Sub-process. – Sahil Sangani Oct 15 '21 at 14:44
  • 1
    If you have processes updating the same metric, it would not be a pb for counter or assimilated accumulation metrics. For gauge, you can use the `multiprocess_mode` parameter to adjust how it is handled. – Michael Doubez Oct 19 '21 at 07:02
  • For me, I used `threads` at the end and pass created object of metrics and fetched it whenever needed and updated. – Sahil Sangani Oct 19 '21 at 21:55