Prometheus Client Instrumentation in Python

Exposing numeric telemetry from a Python service starts with the official prometheus_client library: it owns the in-process registry, the metric types, and the text exposition format that a Prometheus server scrapes. Backend engineers reach for it because it has no broker, no agent, and no push step in the common case — the process simply holds counters in memory and renders them on demand. This guide is part of the Python Metrics and Instrumentation guide, and it pairs with the deeper material on choosing the right metric type and controlling cardinality and on how the same workload looks under the OpenTelemetry metrics SDK. Two focused walkthroughs build directly on this page: instrumenting Flask with Prometheus metrics and exposing custom metrics with the Prometheus client. If you have not yet settled the pull-versus-push question for your deployment, read OpenTelemetry vs Prometheus for Python metrics first and come back once scraping is the answer.

The hard part is rarely the first counter. It is the lifecycle: where instruments live, which registry owns them, how the /metrics endpoint is wired into a real WSGI or ASGI app, and how prefork servers like gunicorn aggregate values across workers without resetting them on every scrape. Get those four decisions right at the start and the instrumentation stays correct as the service grows; get them wrong and you spend a quarter explaining why the dashboard undercounts.

Prerequisites

The exposition format and the multiprocess API are stable within the 0.x series, but minor releases have changed default collector behavior and added exposition flags. Pin the range so a scrape that worked in staging renders identically in production.

pip install "prometheus-client>=0.20.0,<1.0.0"

For an in-app endpoint you also need a server. Flask and Django speak WSGI; FastAPI and Starlette speak ASGI. The client ships handlers for both, so no extra exposition dependency is required.

# WSGI app (Flask, Django)
pip install "gunicorn>=21.2.0,<23.0.0"
# ASGI app (FastAPI, Starlette)
pip install "uvicorn>=0.29.0,<0.35.0"

The same pins expressed as a pyproject.toml dependency list, which is what belongs in a service repository:

[project]
dependencies = [
  "prometheus-client>=0.20.0,<1.0.0",
  "uvicorn>=0.29.0,<0.35.0",
]

Two environment variables shape runtime behavior, and both belong in your deployment manifest rather than in code, because they differ between a local run and a prefork production server:

# Required only for prefork servers (gunicorn, uvicorn --workers N)
export PROMETHEUS_MULTIPROC_DIR=/var/run/prom_multiproc
# Optional: suppress the *_created timestamp series a Counter/Summary emits
export PROMETHEUS_DISABLE_CREATED_SERIES=true

PROMETHEUS_MULTIPROC_DIR must name a directory that exists, is writable by every worker, and is emptied before the master process forks. On Kubernetes an emptyDir volume is the natural fit; on a bare host use a tmpfs path under /var/run so a reboot cannot resurrect stale files. Older releases read a lowercase prometheus_multiproc_dir; that spelling still works in some versions but is deprecated, so standardize on the uppercase name.

Concept & architecture

A prometheus_client deployment has three moving parts: instruments, a registry, and an exposition endpoint. Everything else — multiprocess files, custom collectors, framework middleware — is a variation on how those three are wired together.

Instruments are the metric types. A Counter only goes up and resets on process restart — use it for totals like requests served or errors raised. A Gauge goes up and down — use it for instantaneous values like in-flight requests, queue depth, or memory. A Histogram buckets observations and exposes _bucket, _sum, and _count series so quantiles can be computed server-side, which aggregates correctly across many instances. A Summary exposes _sum and _count only; the Python client, unlike some other language clients, deliberately implements no client-side quantiles, so a Summary gives you an average and nothing more. Two lesser-known types round out the set: Info emits a constant _info series carrying build or version labels, and Enum emits one series per state with a 1 on the current one — both are useful for exposing what a process is rather than what it did. The trade-offs are worked through in depth under choosing between Counter, Gauge, Histogram, and Summary.

The registry is the container that knows about every instrument. By default each instrument registers itself with the module-level REGISTRY at construction, which is why you almost never touch the registry directly. You pass an explicit CollectorRegistry only for test isolation or for multiprocess aggregation. The registry is also the unit of rendering: generate_latest(registry) walks its collectors, calls collect() on each, and concatenates the result, so whatever a registry contains at scrape time is exactly what the scrape returns.

Default collectors are registered automatically the moment you import the library. ProcessCollector exports process_cpu_seconds_total, process_resident_memory_bytes, and open file descriptors on Linux, while PlatformCollector and GCCollector export interpreter and garbage-collector internals. You get these for free, and they are usually the fastest way to spot a leaking worker without writing any code.

Labels turn one metric name into many time series. Calling .labels(method="GET", status="200") returns a child object you then increment or observe; the parent metric itself cannot be incremented once it declares labels. Each unique label-value combination is a distinct series stored in memory and re-serialized on every scrape, which is why uncontrolled label values are the primary cause of cardinality blowups — read controlling label cardinality in Prometheus before you label anything with a user ID or a raw URL path.

Collectors are the extension point. Anything with a collect() method that yields metric families can be registered, which lets you expose a value that lives outside your process — a queue depth in Redis, a row count in Postgres — by reading it at scrape time instead of maintaining a gauge on every change. That pattern is developed further in exposing custom metrics with the Prometheus client.

The text exposition format

What a scrape returns is the Prometheus text exposition format: a flat, line-oriented document where each line is one sample. A metric carries an optional # HELP line (human description), a # TYPE line (counter, gauge, histogram, or summary), and one or more sample lines of the shape name{label="value",...} number. The format is deliberately dumb — there is no nesting, no timestamps in the common case, and no compression — which is what makes it cheap to generate and trivial to debug with curl.

The composite types expand into several series, and knowing the expansion is what lets you predict cost before you deploy. A Histogram named x emits x_bucket{le="..."} for each bucket boundary plus a synthetic le="+Inf", an x_sum, and an x_count. A Summary named y emits y_sum and y_count. Counters render with a _total suffix appended automatically, so Counter("requests_total", ...) and Counter("requests", ...) both expose requests_total. Counters and summaries additionally emit a _created series carrying the Unix timestamp of instrument creation — harmless in a small service, but a silent doubling of counter series in a large one, which is what PROMETHEUS_DISABLE_CREATED_SERIES (or a call to disable_created_metrics()) is for.

Do the arithmetic before shipping. A histogram with 10 explicit bucket boundaries and one label of 4 values costs 4 * (10 + 1 + 2) = 52 series, not one. Add a second label with 5 values and it is 260. Multiply by the number of replicas, then by the retention period, and the reason cardinality reviews exist becomes obvious.

generate_latest(registry) produces this document as bytes and CONTENT_TYPE_LATEST is the matching Content-Type header (text/plain; version=0.0.4; charset=utf-8). Every exposition path — start_http_server, the WSGI and ASGI apps, a hand-rolled route — ultimately calls generate_latest, so the rendered bytes are identical regardless of how you serve them. Returning the wrong content type is one of the few ways to break a scrape while still returning a perfectly valid body, so always pair the two constants.

Default collectors and what they cost

Importing the library registers three collectors against REGISTRY automatically. ProcessCollector exposes process_cpu_seconds_total, process_resident_memory_bytes, process_virtual_memory_bytes, process_start_time_seconds, and process_open_fds on Linux — invaluable for spotting a leaking worker without any extra code, and unavailable on platforms without /proc. PlatformCollector exposes a static python_info series with version and implementation labels. GCCollector exposes python_gc_collections_total, python_gc_objects_collected_total, and per-generation gauges.

These are nearly free at normal cardinality, but two caveats matter. First, under multiprocess mode they are per-process and are deliberately excluded from the merged scrape, so if you need process memory you expose it per worker on a side channel or scrape the container runtime instead. Second, in a tightly scraped, many-replica fleet the default series add up; you can unregister a collector with REGISTRY.unregister(GC_COLLECTOR) if nothing queries it. Import the singletons (PROCESS_COLLECTOR, PLATFORM_COLLECTOR, GC_COLLECTOR) from prometheus_client to unregister selectively, and do it once at startup before the first scrape rather than conditionally at runtime.

Step-by-step implementation

Step 1 — Define instruments once, at import time. Instruments are stateful objects keyed by name within the registry. Constructing the same metric twice raises ValueError: Duplicated timeseries, so declare them at module scope in a dedicated module and import the objects where needed rather than recreating them per request. Choose the histogram buckets deliberately at this point: the defaults target generic web latency in seconds and will quantize a fast RPC service into a single bucket.

# metrics.py — single source of truth for instrument objects
from prometheus_client import Counter, Gauge, Histogram

# Counter: monotonically increasing total of handled requests
REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests processed",
    ["method", "status"],          # labels: keep values bounded
)

# Gauge: instantaneous count of requests currently being served
IN_PROGRESS = Gauge(
    "http_requests_in_progress",
    "HTTP requests currently in flight",
)

# Histogram: latency distribution with explicit, domain-tuned buckets
LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    ["method"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)

Step 2 — Record observations from your code. Increment counters, set or shift gauges, and observe histogram values. The time() helper on a histogram measures wall-clock duration and observes it on exit; track_inprogress() increments a gauge on entry and decrements on exit; count_exceptions() increments a counter only when the block raises. Using the helpers rather than hand-rolled timing means the metric is still correct when the block exits through an exception.

from metrics import REQUESTS, IN_PROGRESS, LATENCY

def handle(method: str):
    with IN_PROGRESS.track_inprogress():            # +1 on enter, -1 on exit
        with LATENCY.labels(method=method).time():  # observes elapsed seconds
            status = do_work()
            REQUESTS.labels(method=method, status=status).inc()

Step 3 — Expose the endpoint for a standalone process. For a worker with no web server, start_http_server spins up a background thread serving the exposition on a dedicated port. It is the simplest path for batch jobs, Celery workers, and long-running scripts, and it needs exactly one line.

from prometheus_client import start_http_server
import time

start_http_server(8000, addr="0.0.0.0")   # serves the exposition on :8000
while True:
    handle("GET")
    time.sleep(1)

Step 4 — Or mount a scrape handler in an existing app. When you already run a web server, do not open a second port unless you must. The client exposes ready-made WSGI and ASGI apps you can mount on /metrics, so scraping shares the app's port, TLS, and auth middleware. The concrete WSGI wiring for a real framework, including the DispatcherMiddleware pattern and route-safe labels, is walked through in instrumenting Flask with Prometheus metrics.

# ASGI: mount the client's app on a Starlette/FastAPI route
from prometheus_client import make_asgi_app
from fastapi import FastAPI

app = FastAPI()
app.mount("/metrics", make_asgi_app())   # WSGI equivalent: make_wsgi_app()

Step 5 — Enable multiprocess mode for prefork servers. Gunicorn and multi-worker uvicorn fork the process, so each worker holds its own in-memory registry. A single scrape hits one random worker and sees only that worker's data, which looks like a counter that shrinks. The fix is a shared on-disk directory plus a MultiProcessCollector, wired in the configuration section below.

Step 6 — Verify the exposition before you trust the dashboard. Curl the endpoint, confirm the # TYPE lines match what you intended, and count the series with something like curl -s localhost:8000/metrics | grep -vc '^#'. Two things are worth checking on day one: that every label you emit has a bounded value set, and that the total series count is what your arithmetic predicted. A metric that is wrong at this stage is far cheaper to fix than one that has been feeding an alert rule for a month.

Configuration reference

Multiprocess wiring

Under multiprocess mode, instruments write to memory-mapped files in PROMETHEUS_MULTIPROC_DIR instead of an in-memory registry. The scrape endpoint must build a fresh registry and attach a MultiProcessCollector that reads and merges those files. The default ProcessCollector and GCCollector are per-process and do not aggregate, so a clean registry also avoids double-counting them.

# metrics_endpoint.py — multiprocess-aware scrape handler
import os
from prometheus_client import CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST
from prometheus_client import multiprocess

def render_metrics() -> tuple[bytes, str]:
    if os.environ.get("PROMETHEUS_MULTIPROC_DIR"):
        registry = CollectorRegistry()                  # fresh, empty registry
        multiprocess.MultiProcessCollector(registry)    # merges per-worker files
    else:
        from prometheus_client import REGISTRY as registry
    return generate_latest(registry), CONTENT_TYPE_LATEST

A gunicorn child_exit hook must call multiprocess.mark_process_dead(worker.pid) so a dead worker's live-mode gauge files are cleaned up; otherwise a recycled worker leaves an in-flight gauge stuck above zero forever. Counter, summary, and histogram files are intentionally retained after a worker dies, because their totals still belong in the aggregate.

# gunicorn.conf.py
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

Two behaviors surprise people. Gauge aggregation is controlled entirely by multiprocess_mode: all exposes one series per worker with a pid label (correct but cardinality-multiplying), livesum adds up only living workers (what you want for in-flight requests), and min/max reduce across workers. And Info and Enum are not supported in multiprocess mode at all — expose build information from a single sidecar process or as a target label instead.

Async & concurrency considerations

The client is thread-safe: inc(), set(), and observe() take internal locks, so concurrent threads sharing one instrument are correct without extra synchronization. Under asyncio, a single event loop runs instrument mutations on one thread, so there is no contention and no need to offload to an executor — recording a metric is a cheap in-memory operation measured in microseconds. Modern releases detect coroutine functions when you use time() or count_exceptions() as decorators, so decorating an async def handler measures the whole awaited call rather than the time to build the coroutine object.

The endpoint is the subtlety. generate_latest() walks the entire registry and renders text synchronously; under multiprocess mode it also reads and merges every per-worker file. Both make_asgi_app() and a hand-rolled ASGI route perform that work inline on the event loop, so a very large registry can produce a visible latency blip on every scrape. At normal cardinality this is irrelevant; at hundreds of thousands of series it is not, and the answer is to cut the series count rather than to move the work — a registry that takes a hundred milliseconds to render is a registry nobody can query efficiently either. Keep the scrape interval sane (10–30 seconds) and keep label cardinality bounded.

Fork timing is the other trap. Instruments created before a fork are inherited by every child, which is fine, but a background thread started before the fork is not — start_http_server called at import time in a gunicorn master leaves each worker without a serving thread, or with several workers fighting over the same port. Start servers and any scrape-time machinery from a post-fork hook, or mount the exposition as a route so the framework owns the socket.

For per-request latency, prefer a Histogram over a Summary. Beyond the aggregation argument — bucket counts sum cleanly across replicas, in-process quantiles do not — the Python client's Summary simply has no quantiles to offer, so a Summary on request duration buys you an average you could have computed from a histogram's _sum and _count anyway.

Production code examples

This end-to-end example runs an ASGI service under uvicorn with multiprocess mode enabled, records request totals and latency from middleware, and exposes an aggregated scrape endpoint.

# app.py
import os
import random
import time
from fastapi import FastAPI, Request, Response
from prometheus_client import Counter, Histogram, Gauge
from prometheus_client import CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST
from prometheus_client import multiprocess

# 1. Instruments declared once at import time
REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests", ["method", "status"]
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency in seconds", ["method"],
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
IN_PROGRESS = Gauge(
    "http_requests_in_progress", "In-flight requests",
    multiprocess_mode="livesum",          # sum live workers under multiproc
)

app = FastAPI()

# 2. Middleware records every request without per-route boilerplate
@app.middleware("http")
async def record_metrics(request: Request, call_next):
    method = request.method
    status = "500"                         # assume failure until proven otherwise
    IN_PROGRESS.inc()
    start = time.perf_counter()
    try:
        response = await call_next(request)
        status = str(response.status_code)
        return response
    finally:
        IN_PROGRESS.dec()
        LATENCY.labels(method=method).observe(time.perf_counter() - start)
        REQUESTS.labels(method=method, status=status).inc()

# 3. Business route
@app.get("/work")
async def work():
    time.sleep(random.uniform(0.01, 0.2))
    return {"ok": True}

# 4. Multiprocess-aware scrape endpoint
@app.get("/metrics")
async def metrics():
    if os.environ.get("PROMETHEUS_MULTIPROC_DIR"):
        registry = CollectorRegistry()
        multiprocess.MultiProcessCollector(registry)
    else:
        from prometheus_client import REGISTRY as registry
    data = generate_latest(registry)
    return Response(content=data, media_type=CONTENT_TYPE_LATEST)

Note the status = "500" default: initializing the label value before the try means an unhandled exception still produces a counted request instead of a NameError inside the finally block. Run the service with the multiprocess directory set so several uvicorn workers aggregate:

export PROMETHEUS_MULTIPROC_DIR=/tmp/prom_multiproc
mkdir -p "$PROMETHEUS_MULTIPROC_DIR" && rm -f "$PROMETHEUS_MULTIPROC_DIR"/*
uvicorn app:app --workers 4 --port 8080

Expected Output: scraping http://localhost:8080/metrics after a few requests returns the text exposition format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 42.0
# HELP http_request_duration_seconds Request latency in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",le="0.01"} 3.0
http_request_duration_seconds_bucket{method="GET",le="0.05"} 11.0
http_request_duration_seconds_bucket{method="GET",le="0.1"} 24.0
http_request_duration_seconds_bucket{method="GET",le="0.25"} 41.0
http_request_duration_seconds_bucket{method="GET",le="+Inf"} 42.0
http_request_duration_seconds_sum{method="GET"} 4.713
http_request_duration_seconds_count{method="GET"} 42.0
# HELP http_requests_in_progress In-flight requests
# TYPE http_requests_in_progress gauge
http_requests_in_progress 0.0

Under multiprocess mode the default process_* and python_gc_* series are absent from this scrape, because the fresh registry holds only the MultiProcessCollector. Expose those separately per worker if you need them, or take them from the container runtime.

Standalone worker with a custom collector

A background worker with no web server uses start_http_server directly. When a value lives in an external system rather than being mutated by your code — a queue depth in Redis, a row count in a database — write a custom collector that reads it at scrape time instead of maintaining a gauge on every change. This avoids drift between the reported value and reality, and it means the expensive read only happens when Prometheus actually scrapes.

# worker.py
import time
from prometheus_client import start_http_server, Counter
from prometheus_client.core import GaugeMetricFamily, REGISTRY

JOBS = Counter("worker_jobs_total", "Jobs handled", ["result"])

class QueueDepthCollector:
    # collect() runs once per scrape; yield one metric family per call
    def collect(self):
        depth = read_queue_depth()                 # external read, scrape-time
        g = GaugeMetricFamily(
            "worker_queue_depth", "Pending jobs in the broker queue",
            labels=["queue"],
        )
        g.add_metric(["default"], depth)
        yield g

REGISTRY.register(QueueDepthCollector())           # custom collector, no state

def main():
    start_http_server(9000)                         # exposition on :9000
    while True:
        result = run_one_job()
        JOBS.labels(result=result).inc()
        time.sleep(0.5)

Expected Output: a scrape of :9000/metrics interleaves the counter, the scrape-time gauge, and the default process collectors:

# HELP worker_jobs_total Jobs handled
# TYPE worker_jobs_total counter
worker_jobs_total{result="ok"} 137.0
worker_jobs_total{result="retry"} 4.0
# HELP worker_queue_depth Pending jobs in the broker queue
# TYPE worker_queue_depth gauge
worker_queue_depth{queue="default"} 12.0
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.1273728e+07

Two rules keep custom collectors safe. Never let collect() raise or block for long — an exception there fails the whole scrape, and a slow external read turns a scrape timeout into a gap in every metric the process exposes, so wrap the lookup in a short timeout and fall back to the last known value. And never register the same collector twice; guard registration behind module-level execution exactly as you do for instruments. For values your own code changes, a plain Gauge with .inc()/.dec() is simpler and cheaper than a collector.

Common mistakes

Duplicated timeseries on import

Error signature: ValueError: Duplicated timeseries in CollectorRegistry: ... Root cause: the same metric name is constructed twice — typically because a module that defines instruments is imported under two different names, or instruments are created inside a function that runs per request. Remediation: define every instrument exactly once at module scope in a dedicated metrics.py and import the objects. For test suites that reload modules, construct instruments against a dedicated CollectorRegistry() you can discard between cases.

Counters reset on every scrape under gunicorn

Error signature: values jump between scrapes and never accumulate; http_requests_total looks tiny relative to real traffic. Root cause: prefork workers each hold a private in-memory registry, and the scrape lands on a random worker. Remediation: set PROMETHEUS_MULTIPROC_DIR, render through a MultiProcessCollector, and add the child_exit hook calling mark_process_dead. Clear the directory on startup so a redeploy does not resurrect stale files.

Using a Summary where you needed percentiles

Error signature: the dashboard panel for p99 latency is empty, or a service-wide quantile computed from per-pod values is wrong. Root cause: the Python client's Summary exposes only _sum and _count; where other clients compute in-process quantiles, this one does not, and in-process quantiles could not be aggregated across replicas anyway. Remediation: switch to a Histogram with buckets tuned to your latency SLO, and compute quantiles in PromQL with histogram_quantile over rate(..._bucket[5m]).

An in-flight gauge that never returns to zero

Error signature: http_requests_in_progress drifts upward across deploys and never settles, even when the service is idle. Root cause: either a dec() that is skipped when the handler raises, or dead workers' gauge files being merged into the scrape because mark_process_dead was never called. Remediation: use track_inprogress() or a finally block so the decrement always runs, set multiprocess_mode="livesum", and wire the gunicorn child_exit hook.

Unbounded labels from request paths

Error signature: scrape duration climbs, Prometheus memory grows, and the exposition is megabytes long. Root cause: a label carries a per-request value — a raw path containing IDs, a user identifier, a full URL — so every request creates a new permanent series in the process. Remediation: label with the route pattern rather than the concrete path, cap free-form values with an allow-list plus an other bucket, and review the series arithmetic described above. The full technique is in controlling label cardinality in Prometheus.

Related

• Python Metrics and Instrumentation — the parent guide covering instrument choice, cardinality, transport, and cost together.
• Instrumenting Flask with Prometheus metrics — the WSGI wiring, request hooks, and route labels in full.
• Exposing custom metrics with the Prometheus client — scrape-time collectors for values your process does not own.
• Metric types and cardinality control — choosing the right instrument and bounding its label set.
• OpenTelemetry vs Prometheus for Python metrics — the pull-versus-push decision and the exporter bridge between them.
• The OpenTelemetry metrics SDK in Python — the same instrumentation problem solved with a push pipeline.
• Instrumenting Django with Prometheus metrics — middleware, resolver-match route labels, and a scrape that does not count itself.

Frequently Asked Questions