Prometheus Multiprocess Mode with Gunicorn

Gunicorn's prefork model gives each worker its own copy of every prometheus_client metric, and a scrape reaching one worker sees one worker's numbers. Multiprocess mode fixes that by having each worker write its values to shared files and aggregating them at scrape time. This article is the complete, working configuration: directory, hooks, endpoint, gauge modes, worker recycling and container deployment, with the checks that prove every worker is counted. It is part of metrics in multi-process Python servers in the Python metrics and instrumentation section.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" "gunicorn>=22.0.0,<24.0.0" "flask>=3.0.0,<4.0.0"

A Gunicorn-served application — Flask here, though nothing below is Flask-specific — and a Prometheus server that can reach it.

Implementation steps

Step 1 — Set the directory in the environment. The variable must be present when prometheus_client is first imported, which under Gunicorn means before the application module loads. Setting it in the process environment — the container spec, the systemd unit, the shell that launches Gunicorn — is the reliable way.

export PROMETHEUS_MULTIPROC_DIR=/run/prometheus
gunicorn -c gunicorn.conf.py myservice.app:app

Step 2 — Clean it in on_starting. The hook runs in the master before any worker is forked, so it is the one safe moment to empty the directory.

# gunicorn.conf.py
import os, shutil
from prometheus_client import Counter, multiprocess

workers = 4
max_requests = 5000
max_requests_jitter = 500

def on_starting(server):
    d = os.environ["PROMETHEUS_MULTIPROC_DIR"]
    shutil.rmtree(d, ignore_errors=True)
    os.makedirs(d, exist_ok=True)

Step 3 — Serve an aggregating endpoint. A route in the application, answered by whichever worker receives it.

# myservice/app.py
from flask import Flask, Response
from prometheus_client import CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST, multiprocess

app = Flask(__name__)

@app.get("/metrics")
def metrics():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return Response(generate_latest(registry), mimetype=CONTENT_TYPE_LATEST)

Step 4 — Define metrics, with modes on gauges.

# myservice/metrics.py
from prometheus_client import Counter, Gauge, Histogram

REQUESTS = Counter("http_requests_total", "HTTP requests", ["route", "method", "status"])
LATENCY = Histogram("http_request_duration_seconds", "HTTP latency", ["route"],
                    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0))
IN_FLIGHT = Gauge("http_requests_in_flight", "Requests in flight",
                  multiprocess_mode="livesum")

Step 5 — Handle worker exits. Clean up gauges and count the exit, with the reason where Gunicorn provides one.

# gunicorn.conf.py (continued)
WORKER_EXITS = Counter("gunicorn_worker_exits_total", "Worker exits", ["reason"])

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)
    WORKER_EXITS.labels(reason="exit").inc()

def worker_abort(worker):
    WORKER_EXITS.labels(reason="timeout").inc()

worker_abort runs inside the worker when the master sends it SIGABRT after a timeout, which is exactly the case that produces no application error.

Deploying in containers

In Kubernetes, the directory is best provided by an emptyDir volume backed by memory. It is created empty when the pod starts, which satisfies the clean-start requirement even without the on_starting hook, and it is private to the pod, so two replicas can never share it.

spec:
  containers:
    - name: api
      env:
        - name: PROMETHEUS_MULTIPROC_DIR
          value: /run/prometheus
      volumeMounts:
        - name: prom-multiproc
          mountPath: /run/prometheus
  volumes:
    - name: prom-multiproc
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi

The size limit matters: a memory-backed volume counts against the container's memory, and a server recycling workers for weeks accumulates files. Sixty-four megabytes is generous for most services, and hitting it fails writes rather than the pod, which is a better failure than an out-of-memory kill.

The on_starting hook is still worth keeping. A container restart inside the same pod — after a crash, for instance — keeps the emptyDir and its stale files; only a new pod gets a fresh volume.

Metrics that do not aggregate

Some familiar metrics disappear in multiprocess mode, and knowing which saves a confused hour.

The default process collector — process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds — reports the current process and is registered on the default registry, which is not served. Container-level resource metrics from cAdvisor or the kubelet cover the pod as a whole; per-worker memory, if needed, can be a gauge with multiprocess_mode="all", which keeps one series per worker labelled by process identifier.

The platform and GC collectors have the same limitation. GC metrics recorded per worker, as described in monitoring Python GC and memory usage, need to be recorded as ordinary counters and gauges from a hook rather than relying on the built-in collector.

Custom collectors — classes with a collect method registered on the default registry — are not called by the multiprocess endpoint. They can be registered on the per-scrape registry alongside the MultiProcessCollector, in which case they run in whichever worker answers and report that worker's view, which is correct only for values that are the same everywhere, such as build information.

Summaries aggregate their count and sum across processes but not their quantiles, which are computed per process and cannot be combined. Histograms are the multiprocess-safe choice for latency, and the one this site recommends anyway.

Where to record, and what the middleware looks like

The metrics are only as good as the place they are recorded. A request's count and duration belong at the outermost point the application controls, so that every route — including errors raised by routing itself — is counted once. In Flask that is a pair of request hooks; in Django, a middleware at the top of the stack; under an ASGI framework, a middleware as described in instrumenting FastAPI with Prometheus metrics.

import time
from flask import g, request
from myservice.metrics import REQUESTS, LATENCY, IN_FLIGHT

@app.before_request
def _start():
    g._t0 = time.perf_counter()
    IN_FLIGHT.inc()

@app.after_request
def _count(response):
    route = request.url_rule.rule if request.url_rule else "unmatched"
    REQUESTS.labels(route=route, method=request.method,
                    status=str(response.status_code)).inc()
    return response

@app.teardown_request
def _finish(exc):
    IN_FLIGHT.dec()
    route = request.url_rule.rule if request.url_rule else "unmatched"
    LATENCY.labels(route=route).observe(time.perf_counter() - g._t0)

The route label uses the matched rule — /orders/<int:order_id> — rather than the path, which would put every order identifier into a label and turn one series into millions. Unmatched requests share one value, so a scanner probing random paths cannot inflate cardinality either.

The split between the two hooks is deliberate. after_request sees the response and therefore the status code, but does not run when a request fails with an unhandled exception; teardown_request always runs, so the in-flight gauge and the duration live there and can never leak. A request that raises before a response exists is counted by the error handler that turns it into a 500, which then passes through after_request like any other response.

Two details are easy to miss under Gunicorn. The metrics module must be imported inside the application, not in gunicorn.conf.py before the variable is set, and the /metrics route itself should be excluded from the request counter, or every scrape inflates the traffic it measures — a small effect, but one that makes the verification arithmetic below fail by exactly the number of scrapes. Checking request.path == "/metrics" at the top of each hook and returning early is enough.

Configuration options

Verification

Send a known number of requests and compare the counter before and after:

before=$(curl -s localhost:8000/metrics | awk '/^http_requests_total\{route="\/health"/ {s+=$2} END {print s+0}')
for i in $(seq 1000); do curl -s -o /dev/null localhost:8000/health; done
after=$(curl -s localhost:8000/metrics | awk '/^http_requests_total\{route="\/health"/ {s+=$2} END {print s+0}')
echo $((after - before))

Expected Output: exactly the number sent — the health check's own scrapes excluded by route.

1000

Then force recycling with kill -HUP <master pid>, which replaces every worker, and repeat. The difference is still exactly a thousand and the counter never decreased, which confirms dead workers' files are counted. Finally, with no traffic in flight, http_requests_in_flight reads zero, which confirms mark_process_dead removed the old workers' gauge values.

Common mistakes

start_http_server in each worker. Error signature: OSError: [Errno 98] Address already in use in every worker but one. Root cause: one port, many workers. Remediation: an application route, or one server started from the master.

The variable set in gunicorn.conf.py after importing prometheus_client. Error signature: values per worker despite the configuration. Root cause: the backend was chosen at import. Remediation: set it in the environment, or set it at the very top of the config before any import.

No sizeLimit on the memory volume. Error signature: pod memory creeping upward over weeks. Root cause: file growth from recycling counted against the container. Remediation: a size limit and a restart policy that renews pods occasionally.

Summaries for latency. Error signature: quantiles that differ wildly between scrapes. Root cause: quantiles computed per process and not combinable. Remediation: histograms.

Expecting process_* metrics. Error signature: dashboards that go empty after enabling multiprocess mode. Root cause: the process collector is not part of the aggregated registry. Remediation: container metrics, or per-worker gauges with the all mode.

Frequently Asked Questions