Exposing Custom Metrics with the Prometheus Client

Framework auto-instrumentation gives you HTTP counters, but the numbers that matter to the business — orders placed, payments declined, queue depth, batch rows processed — only you can emit. This page is for backend engineers and SREs who already have a /metrics endpoint and now need to add their own signals to it with prometheus_client, naming and labeling them so they stay queryable and cheap. It builds on the broader Prometheus client instrumentation guide and is part of the Python Metrics and Instrumentation guide.

The whole discipline reduces to three decisions per metric: which instrument type, what name and unit, and which labels. Get those right and the series are cheap, aggregatable, and self-documenting; get the labels wrong and you take down your Prometheus server.

Prerequisites

Only the client is required to define and render metrics; you expose them through whatever server you already run.

pip install "prometheus-client>=0.20.0,<1.0.0"

If you serve metrics from a prefork server (gunicorn, multi-worker uvicorn), set the multiprocess directory before the workers start so custom series aggregate across workers exactly like default ones.

export PROMETHEUS_MULTIPROC_DIR=/tmp/app_prom
mkdir -p "$PROMETHEUS_MULTIPROC_DIR"

Nothing else is needed: there is no exporter to configure and no background thread to start. Custom metrics live in the same process memory as the default collectors and are serialized only when a scrape arrives.

Implementation

Step 1 — Map each signal to an instrument type. A monotonic total uses a Counter. A value that rises and falls uses a Gauge. A distribution of magnitudes or durations uses a Histogram. The decision is rarely ambiguous once you ask "does this only ever go up?" — and the deeper trade-offs, especially Histogram versus Summary and why per-process quantiles cannot be aggregated, are covered in choosing between Counter, Gauge, Histogram, and Summary.

Step 2 — Declare the instruments once, in one module. Constructing an instrument registers it with the default REGISTRY, and registering the same name twice raises Duplicated timeseries in CollectorRegistry. Put every instrument at module level in a single module that business code imports; never construct instruments inside a request handler, a factory function, or a class body that runs more than once.

# business_metrics.py — one module owns the instruments
from prometheus_client import Counter, Gauge, Histogram

# Counter: orders only ever accumulate
ORDERS = Counter(
    "orders_processed_total",
    "Orders processed, by outcome and payment method",
    ["outcome", "payment_method"],          # both are small enumerations
)

# Gauge: queue depth goes up and down
QUEUE_DEPTH = Gauge(
    "order_queue_depth",
    "Orders currently waiting in the processing queue",
)

# Histogram: payment latency distribution, buckets in seconds
PAYMENT_LATENCY = Histogram(
    "payment_gateway_duration_seconds",
    "Payment gateway round-trip latency in seconds",
    ["payment_method"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
)

Buckets are the one histogram argument worth deliberating over. They are cumulative upper bounds in seconds, they are fixed at construction, and every bucket multiplies the series count for that metric — eight buckets plus +Inf, _sum, and _count is eleven series per label combination. Choose boundaries that straddle your SLO threshold so a quantile query has resolution where it matters, rather than the default set tuned for generic sub-second web requests.

Step 3 — Record from business code. Increment counters on outcomes, set or shift gauges as state changes, and time histograms around the operation. The .time() context manager observes elapsed seconds on exit; .count_exceptions() increments only when the block raises; .track_inprogress() on a gauge increments on entry and decrements on exit, which is the correct way to measure concurrency.

from business_metrics import ORDERS, QUEUE_DEPTH, PAYMENT_LATENCY

def process_order(order, method):
    QUEUE_DEPTH.dec()                                   # leaving the queue
    with PAYMENT_LATENCY.labels(method).time():         # observe round trip
        ok = charge(order, method)
    outcome = "success" if ok else "declined"
    ORDERS.labels(outcome=outcome, payment_method=method).inc()

.labels(...) performs a dictionary lookup and returns a child object. In a hot loop, resolve the child once outside the loop and call .inc() on it, rather than re-resolving per iteration.

Step 4 — Read externally owned values at scrape time. For values your code does not naturally push on every change — a queue depth held by Redis, a cache size, a licence-seat count — use a Gauge callback so the value is read when Prometheus scrapes. This avoids drift between the real value and a stale reported one.

QUEUE_DEPTH.set_function(lambda: redis.llen("order_queue"))  # read at scrape

The callback runs inline during exposition, so it must be fast and must not raise: a slow call delays every scrape, and an exception fails the whole endpoint, not just that metric. Wrap anything that touches the network in a timeout and fall back to the last known value. When one source yields several related metrics at once, write a custom collector class with a collect() method instead of stacking callbacks.

Step 5 — Follow naming and unit conventions. Names are the contract every dashboard and alert depends on. Use snake_case, base units, and the conventional suffixes: _total for counters, _seconds for any duration, _bytes for sizes. Prefix related metrics with the subsystem they belong to so they group in autocomplete. Never bake units into the value (no milliseconds, no kilobytes) — store seconds and bytes and let PromQL scale for display. The # TYPE line is emitted automatically from the instrument class, so you do not write it, and the help string is the only documentation most on-call engineers will ever read: make it a sentence, not a repeat of the name.

These conventions are Prometheus-specific. If the same service also emits OpenTelemetry instruments, note that OTel uses dotted names and a separate unit field, and the translation between the two is one of the practical differences discussed in OpenTelemetry vs Prometheus for Python metrics.

Step 6 — Keep labels bounded. Every distinct combination of label values is a separate time series stored in memory and on the wire. Labels must draw from small, enumerable sets — outcome, payment method, region, HTTP status class. Never label by user ID, email, order ID, raw URL, or any free-form string, because each new value adds a permanent series and the total multiplies across labels. A counter with outcome (3 values) and payment_method (4 values) is 12 series; adding user_id makes it unbounded. The full treatment, including how to estimate and cap the series budget, is in controlling label cardinality in Prometheus.

# WRONG: user_id is unbounded -> one series per user, forever
ORDERS.labels(outcome="success", payment_method="card", user_id=order.user).inc()

# RIGHT: bounded labels only; identity belongs in logs or traces
ORDERS.labels(outcome="success", payment_method="card").inc()

The identity you just removed is not lost — it belongs in a log line or a span attribute, where storage is per-event rather than per-series. Emitting the order ID through structured logging with the Python standard library, and correlating with metrics by adding trace IDs to log records, gives you the drill-down without the cardinality bill.

Step 7 — Pre-seed label children so series exist before the first event. A labeled metric does not emit a series for a given label combination until that combination is first observed. This means a dashboard panel for orders_processed_total{outcome="declined"} shows "no data" until the first decline happens, and an alert on rate(...) == 0 never fires because the expression returns an empty vector rather than zero. Pre-create the children you know about by calling .labels(...) once at startup; for a counter this registers a series at value 0.

# Pre-seed known label combinations so the series exist from boot
for outcome in ("success", "declined"):
    for method in ("card", "wallet", "transfer"):
        ORDERS.labels(outcome=outcome, payment_method=method)

This only helps for label sets you can enumerate — which is exactly the bounded labels you should be using. If you cannot enumerate the values to pre-seed, that is a strong signal the label is too high-cardinality and belongs in a log line, not a metric.

Step 8 — Expose the instruments. Custom metrics registered with the default REGISTRY render automatically through the same endpoint as everything else — make_wsgi_app(), make_asgi_app(), start_http_server(), or a manual generate_latest() route. There is no extra registration step; constructing the instrument is the registration. The only requirement is that the module holding the instruments is imported by the time the first scrape lands, which for a start_http_server() process means importing it explicitly.

from prometheus_client import start_http_server
import business_metrics  # importing it registers the instruments

start_http_server(8000)   # custom series now appear at /metrics

Inside a web framework the instruments ride the endpoint you already mounted — the wiring for that, including the WSGI dispatcher and the gunicorn child_exit hook, is covered in instrumenting Flask with Prometheus metrics.

Configuration Options

Verification

Drive a few operations, then scrape and confirm the suffixes, types, and bounded labels.

curl -s localhost:8000/metrics | grep -E "orders_processed|payment_gateway|order_queue"

Expected Output:

# HELP orders_processed_total Orders processed, by outcome and payment method
# TYPE orders_processed_total counter
orders_processed_total{outcome="success",payment_method="card"} 18.0
orders_processed_total{outcome="declined",payment_method="card"} 2.0
# HELP order_queue_depth Orders currently waiting in the processing queue
# TYPE order_queue_depth gauge
order_queue_depth 4.0
# HELP payment_gateway_duration_seconds Payment gateway round-trip latency in seconds
# TYPE payment_gateway_duration_seconds histogram
payment_gateway_duration_seconds_bucket{payment_method="card",le="0.5"} 15.0
payment_gateway_duration_seconds_bucket{payment_method="card",le="1.0"} 20.0
payment_gateway_duration_seconds_bucket{payment_method="card",le="+Inf"} 20.0
payment_gateway_duration_seconds_sum{payment_method="card"} 7.84
payment_gateway_duration_seconds_count{payment_method="card"} 20.0

Three things to check in that output. The # TYPE lines must match the instrument you intended — a gauge where you expected a counter usually means the wrong class or a missing _total. The histogram must carry _bucket, _sum, and _count; if only _sum and _count appear you are looking at a Summary. And the series count must be small and stable:

curl -s localhost:8000/metrics | grep -c "^orders_processed_total{"

A healthy scrape shows a handful of series per metric — one per bounded label combination — not thousands. If that count grows with traffic or user count, a label is unbounded and must be removed.

For a regression test, assert on the registry directly rather than parsing text. get_sample_value reads a single sample by name and exact label set, so it fails loudly if you rename a metric or change its labels.

from prometheus_client import REGISTRY

LABELS = {"outcome": "declined", "payment_method": "card"}

def test_declined_order_increments_counter():
    before = REGISTRY.get_sample_value("orders_processed_total", LABELS) or 0.0
    process_order(make_order(), "card")          # gateway stubbed to decline
    after = REGISTRY.get_sample_value("orders_processed_total", LABELS)
    assert after == before + 1                    # reads the delta, not the total

Comparing a delta rather than an absolute value keeps the test order-independent, which matters because the default registry is process-global and other tests will have incremented the same counter.

Common Mistakes

Units encoded in the name or value

Error signature: PromQL math is off by 1000; a panel labeled "seconds" shows milliseconds. Root cause: the metric stores milliseconds or kilobytes and the name lies, or the name carries a unit the value contradicts. Remediation: always store base units — seconds, bytes — and suffix the name accordingly (_seconds, _bytes). Let Grafana and PromQL scale for display rather than pre-scaling in the application.

A label value set that grows without bound

Error signature: Prometheus memory climbs steadily, prometheus_tsdb_head_series grows linearly with traffic, and queries slow down. Root cause: a label carries a high-cardinality value such as user ID, order ID, or raw path, creating a new permanent series per value. Remediation: remove the offending label and move that identity into logs or trace attributes. Keep labels to small enumerations and validate the series budget per the label cardinality guidance.

Duplicated timeseries on import or reload

Error signature: ValueError: Duplicated timeseries in CollectorRegistry: {'orders_processed_total'} at startup, in tests, or on an autoreload. Root cause: the same metric name was constructed twice against the default registry — instruments defined inside an app factory, a class that is instantiated per request, or a module imported under two paths. Remediation: define instruments once at module level and import that module everywhere you record. In tests, pass registry=CollectorRegistry() so each test builds an isolated registry instead of colliding with the global one.

Custom series disappear or fluctuate under gunicorn

Error signature: counters jump backwards between scrapes and totals look far too small for the real traffic. Root cause: each prefork worker keeps its own registry, so a scrape reports whichever worker answered. Remediation: set PROMETHEUS_MULTIPROC_DIR before workers start, render the scrape through a MultiProcessCollector, give every gauge an explicit multiprocess_mode, and call mark_process_dead in the child_exit hook. The full wiring lives in the Prometheus client instrumentation guide.

Related

• Prometheus client instrumentation in Python — the parent guide: registries, exposition servers, multiprocess mode, and framework hooks.
• Python Metrics and Instrumentation — the wider guide covering instrument choice, cardinality, transport, and cost.
• Instrumenting Flask with Prometheus metrics — mounting the endpoint your custom metrics ride on, with gunicorn aggregation.
• Choosing between Counter, Gauge, Histogram, and Summary — the instrument decision in depth, including why quantiles do not aggregate.
• Controlling label cardinality in Prometheus — estimating and capping the series budget your labels create.
• Recording counters and histograms with OpenTelemetry — the same custom signals expressed through the OpenTelemetry metrics API.

Frequently Asked Questions