Metrics in Multi-Process Python Servers

A Python web service under Gunicorn, a Celery worker pool and a job using a process pool all have the same property: several processes, each with its own memory and therefore its own metric values. A counter incremented in one worker is invisible to the others, and a scrape that happens to reach one worker sees only that worker's share of the traffic. Without aggregation the numbers are wrong in a way that looks plausible — which is worse than obviously broken. This guide covers why, and the two correct approaches: prometheus_client multiprocess mode, and per-process export with OpenTelemetry. It is part of the Python metrics and instrumentation section, with detail in Prometheus multiprocess mode with Gunicorn and collecting metrics from Celery workers.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "gunicorn>=22.0.0,<24.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Concept and architecture

Metrics libraries hold values in process memory. That is fast — an increment is an addition behind a lock — and it means a value exists only in the process that recorded it. Prefork servers are designed so that workers share nothing, and metrics inherit that isolation.

There are two ways to get a correct total, and they differ in where the aggregation happens.

Aggregate in the process, before exposition. prometheus_client's multiprocess mode has every process write its values to memory-mapped files in a shared directory. When the metrics endpoint is scraped — by whichever worker receives the request — a special collector reads every process's files and combines them. Counters and histograms are summed; gauges are combined according to a mode declared on each gauge. The scrape sees one coherent set of values for the whole server.

Aggregate downstream, after export. With the OpenTelemetry SDK exporting over OTLP, each process pushes its own metrics, labelled with its own instance identity. The collector or the backend sums them. Nothing is shared between processes, and the only requirement is that each worker initialises its meter provider after the fork, so that it has its own export thread.

The choice usually follows the metrics path already in place, which is the subject of OpenTelemetry vs Prometheus for Python metrics. Scraped Prometheus endpoints need multiprocess mode; OTLP push does not, but has its own post-fork requirement and produces more series, one set per process.

Gauges deserve particular attention in both approaches, because the correct way to combine them depends on what they measure. Summing the number of in-flight requests across workers gives the server's total, which is right. Summing each worker's memory usage gives the server's total, which may be right. Summing each worker's "last successful job timestamp" gives nonsense. Multiprocess mode forces the decision by requiring a mode per gauge; OTLP defers it to query time, where it is easy to forget.

Step-by-step implementation

Step 1 — Recognise the symptom. A counter whose rate is suspiciously low and whose raw value jumps up and down between scrapes is the signature. Adding the process identifier temporarily to a scrape — or curling the metrics endpoint several times in a row — shows different values from different workers.

for i in 1 2 3 4 5; do curl -s localhost:8000/metrics | grep '^http_requests_total{' | head -1; done

Expected Output: a value that goes backwards, which a counter never legitimately does.

http_requests_total{route="/orders"} 310.0
http_requests_total{route="/orders"} 290.0
http_requests_total{route="/orders"} 305.0

Step 2 — Configure multiprocess mode. A directory, writable by every worker and emptied at every server start, and an environment variable pointing at it. The variable must be set before prometheus_client is imported, because the library chooses its storage backend at import.

export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc
rm -rf "$PROMETHEUS_MULTIPROC_DIR" && mkdir -p "$PROMETHEUS_MULTIPROC_DIR"

Step 3 — Serve metrics through a MultiProcessCollector. The endpoint builds a fresh registry per scrape and adds the collector, which reads every process's files. The default registry must not be served in multiprocess mode — it contains only the current process's values.

from prometheus_client import CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST
from prometheus_client import multiprocess

def metrics_endpoint():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return generate_latest(registry), CONTENT_TYPE_LATEST

Step 4 — Declare a mode for every gauge. The mode tells the collector how to combine each process's value. livesum sums values from live processes only, which suits in-flight counts. max or min suits values where the extreme matters. mostrecent suits timestamps. all keeps each process's value as a separate series labelled with the process identifier.

from prometheus_client import Gauge

IN_FLIGHT = Gauge("http_requests_in_flight", "Requests being handled",
                  multiprocess_mode="livesum")
LAST_SUCCESS = Gauge("job_last_success_timestamp_seconds", "Last job success",
                     multiprocess_mode="mostrecent")

Step 5 — Clean up after dead workers. When a worker exits, its files remain. Counters and histograms should keep contributing — their values are cumulative and the requests really happened — but live gauges should not. The mark_process_dead hook removes a dead process's gauge contributions, and Gunicorn's child_exit hook is where to call it.

# gunicorn.conf.py
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

Step 6 — For OTLP, initialise after fork. With the OpenTelemetry SDK, the meter provider and its periodic reader must be created in each worker. A provider created in the master before forking has a reader thread only in the master; workers record into an aggregator nothing ever exports.

# gunicorn.conf.py
def post_fork(server, worker):
    from myservice.telemetry import init_metrics
    init_metrics(instance_id=f"{worker.pid}")      # a provider per worker

Configuration reference

Async and concurrency considerations

Multiprocess mode changes the cost of recording a metric. In single-process mode an increment is an in-memory addition; in multiprocess mode it writes to a memory-mapped file. That is still fast — microseconds — and it is measurably slower, particularly for histograms with many buckets, which write several values per observation. On extremely hot paths, recording fewer observations or aggregating locally before recording is worth considering.

The shared directory grows with the number of distinct processes over the server's lifetime, not the number alive at once. A server that recycles workers every few thousand requests creates a new set of files for each new worker, and the collector reads all of them on every scrape. Over days, scrape latency rises with the file count. Clearing the directory on each full server restart bounds it; for servers that run for weeks with frequent recycling, the file count is worth monitoring.

For asyncio workers — Uvicorn workers under Gunicorn — nothing changes about multiprocess mode itself. Each worker process has its own files regardless of how it handles concurrency internally, and the collector aggregates them the same way.

Process pools inside a service are a separate case. Work sent to a ProcessPoolExecutor runs in child processes that inherit PROMETHEUS_MULTIPROC_DIR if it is set, so their metrics are aggregated too. Without it, metrics recorded in pool processes are simply lost when the process exits. Setting the directory for the whole service, rather than only for the web server, covers both.

Production code examples

A complete Gunicorn configuration and metrics endpoint for multiprocess mode:

# gunicorn.conf.py
import os
import shutil
from prometheus_client import multiprocess

MULTIPROC_DIR = os.environ.setdefault("PROMETHEUS_MULTIPROC_DIR", "/tmp/prometheus_multiproc")

def on_starting(server):
    # 1. A clean directory for every server start.
    shutil.rmtree(MULTIPROC_DIR, ignore_errors=True)
    os.makedirs(MULTIPROC_DIR, exist_ok=True)

def child_exit(server, worker):
    # 2. Dead workers stop contributing live gauges.
    multiprocess.mark_process_dead(worker.pid)

# myservice/metrics.py
from prometheus_client import (CollectorRegistry, Counter, Gauge, Histogram,
                               generate_latest, CONTENT_TYPE_LATEST, multiprocess)

REQUESTS = Counter("http_requests_total", "Requests", ["route", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Latency", ["route"],
                    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5))
IN_FLIGHT = Gauge("http_requests_in_flight", "In flight", multiprocess_mode="livesum")

def render_metrics() -> tuple[bytes, str]:
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)   # 3. every worker's files
    return generate_latest(registry), CONTENT_TYPE_LATEST

Expected Output: stable, aggregated values on every scrape, whichever worker answers.

http_requests_total{route="/orders",status="201"} 1200.0
http_requests_in_flight 10.0

Choosing between the two approaches

Both approaches produce correct numbers. They differ in operational properties that often decide the choice.

Series count. Multiprocess mode exposes one set of series for the whole server. Per-process OTLP export produces one set per worker, distinguished by instance identity, and the backend holds all of them unless the collector aggregates them away. For a service with thirty-two workers across ten replicas, that is the difference between ten sets of series and three hundred and twenty. Aggregating at the collector, as in dropping and aggregating metrics in the collector, brings it back down.

Per-worker visibility. Per-process export keeps each worker's values separately available, which is useful when one worker misbehaves. Multiprocess mode hides that unless gauges are declared with the all mode.

Worker recycling. Multiprocess mode handles recycling cleanly for counters — a dead worker's final values persist — and needs the cleanup hook for gauges. Per-process export creates new series for each new worker's instance identity, and old ones age out with staleness, producing churn in the series count.

Operational simplicity. Multiprocess mode needs a shared directory, an environment variable set at the right time, a cleanup hook and a special endpoint. Per-process export needs a post-fork hook and nothing else in the process, but moves work to the collector.

For a service already scraped by Prometheus, multiprocess mode is usually the least disruptive fix. For a fleet moving to OTLP, per-process export with collector-side aggregation is the natural end state.

Where the files live and what they contain

Multiprocess mode is simple enough to reason about once the file layout is visible. Each process writes files named after the metric type and its process identifier: counter_4121.db, histogram_4121.db, gauge_livesum_4121.db, and so on. Each file is a memory-mapped key-value store: the key encodes the metric name and label values, and the value is a float. An increment finds the key and updates the float in place.

Two practical consequences follow. First, the directory must be on a filesystem that supports memory mapping and is fast — a local tmpfs or container scratch volume is ideal, and a network filesystem is not. In Kubernetes, an emptyDir volume with medium: Memory gives each pod a fresh, fast directory that disappears with the pod, which also handles the clean-start requirement automatically. Second, the directory is per server instance, never shared between pods or hosts. Two servers writing into the same directory would have their totals merged, and processes with the same identifier on different hosts would overwrite each other's files.

Inspecting the directory directly is a useful diagnostic. A file count far above the worker count indicates heavy recycling; a count of zero while traffic is flowing means the environment variable is not reaching the workers, or was set after import.

Metrics that belong to the master, not the workers

Some useful measurements are not per-request at all: how many workers are alive, how often they are being recycled, how long a worker lived before it exited. These are properties of the process manager, and the natural place to record them is in the Gunicorn master through its server hooks.

The master is a process too, and in multiprocess mode it writes its own files like any worker. A counter incremented in child_exit — worker exits, labelled by exit reason — is aggregated into the same exposition as the request metrics. A gauge set in on_starting for the configured worker count is combined by its mode like any other.

Worker exit counts are among the most useful metrics a prefork server can expose, and among the least commonly present. A worker killed by the timeout watchdog — Gunicorn's WORKER TIMEOUT — does not produce an application error; the request that caused it simply vanishes. A counter of worker exits by reason, alerted on when timeouts appear, catches blocked workers that no request metric can see, because the requests they were handling never recorded a completion. It pairs naturally with the event-loop and thread-pool saturation signals in diagnosing blocked event loops in production.

The same reasoning applies to Celery, whose main process manages a pool of child processes and whose signals — worker process initialisation and shutdown — are the equivalents of Gunicorn's hooks. The specifics are in collecting metrics from Celery workers.

Verifying aggregation end to end

The test that aggregation works is the one from Step 1, reversed: repeated scrapes must return monotonically non-decreasing counters, whichever worker answers. Sending a known number of requests — a thousand, say — and checking the counter rose by exactly a thousand confirms that no worker's share is missing. Doing the same after forcing a worker restart, by sending HUP to the master or waiting for max_requests recycling, confirms that dead workers' counts persist and that gauges return to their true values.

In the metrics backend, the check is the absence of counter resets. Prometheus records resets implicitly when a counter decreases; a query over resets(http_requests_total[1h]) that returns non-zero values for a service with no restarts is the unaggregated symptom, visible after the fact across a whole fleet.

Common mistakes

Serving the default registry. Error signature: counters that jump between workers' totals. Root cause: each scrape sees one process. Remediation: a fresh registry with a MultiProcessCollector.

The directory set after import. Error signature: multiprocess mode configured and still per-process values. Root cause: prometheus_client chose its in-memory backend at import. Remediation: set the variable in the environment before the process starts.

A directory reused across restarts. Error signature: counters that start at the previous run's totals. Root cause: stale files from dead processes of an earlier run. Remediation: empty it in on_starting.

Gauges without a mode. Error signature: gauge values several times too large, or timestamps summed. Root cause: the default combination does not fit the gauge. Remediation: declare a mode for every gauge.

No mark_process_dead. Error signature: in-flight counts that never return to zero after worker recycling. Root cause: dead workers' last gauge values persisting. Remediation: call it in child_exit.

OTLP provider created before fork. Error signature: no metrics from any worker. Root cause: the reader thread exists only in the master. Remediation: create the provider in post_fork.

Metrics recorded at import time. Error signature: a metric present in one worker's files and absent from the others, or a series with a stray process label. Root cause: a value recorded in the master before forking, copied into each child's memory but written to the master's file. Remediation: record nothing at import; initialise values in the worker after fork.

The directory on a network filesystem. Error signature: slow scrapes and occasional corrupted reads. Root cause: memory-mapped files on storage that does not support them well. Remediation: a local tmpfs or an in-memory emptyDir.

One directory shared by two server instances. Error signature: totals roughly double what the load balancer reports. Root cause: two servers' files merged by one collector. Remediation: a directory per instance, never a shared volume.

No worker-exit metric. Error signature: requests that vanish without an error, and no metric that moves. Root cause: timeouts kill workers mid-request, and the request never records completion. Remediation: count worker exits by reason in the master's child_exit hook and alert on timeouts.

Frequently Asked Questions