Metric Types and Cardinality Control in Python

Choosing the right metric type and bounding label cardinality are the two decisions that determine whether a Python service produces cheap, queryable telemetry or an unmaintainable time-series explosion. This guide covers counter, gauge, histogram, and summary semantics, when each applies, how labels multiply series count, the high-cardinality anti-patterns that destroy Prometheus, histogram bucket design for latency SLOs, the quantile trade-off between summaries and histograms, and recording rules for query-time aggregation. It is part of the Python Metrics and Instrumentation guide. For the library mechanics referenced throughout, see Prometheus client instrumentation and the OpenTelemetry metrics SDK guides, and for two focused deep dives see controlling label cardinality in Prometheus and choosing between counter, gauge, histogram, and summary.

Label cardinality fan-out The counter http_requests_total carries a method label with three values and a status label with four values, so it expands into a grid of twelve independent stored time series, each accumulating its own samples. http_requests_total Counter · 1 metric name method — 3 values × status — 4 values 12 stored time series status: 200 404 500 503 GET POST PUT 3 methods × 4 statuses = 12 series · one user_id label would make it 1.2M
One metric name fans out into one stored time series per unique label-value combination — the product of the label cardinalities, not the count of metric names.

Prerequisites

Install pinned client libraries. The examples use the official Prometheus client and the OpenTelemetry metrics SDK.

pip install "prometheus-client>=0.20.0,<1.0.0"
pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"

Two environment variables matter for the material below. Set PROMETHEUS_MULTIPROC_DIR to a writable, empty-on-boot directory whenever the service runs under gunicorn or uwsgi, and set OTEL_METRIC_EXPORT_INTERVAL if you export OTLP and want the push cadence to match your scrape interval.

export PROMETHEUS_MULTIPROC_DIR=/dev/shm/prom-metrics   # cleared on each boot
export OTEL_METRIC_EXPORT_INTERVAL=15000                # milliseconds

A Prometheus server (>=2.50,<3.0) scraping the exposition endpoint is assumed for the recording-rule and relabeling material. Python 3.10+ is assumed for the type annotations used in the snippets.

Concept and Architecture

A metric in the Prometheus data model is a named numeric measurement plus an optional set of key-value labels. The unit of storage is the time series: one append-only stream of timestamped samples, identified by the metric name and the exact set of label values. The cost of a metric on the backend is therefore not the number of metric names you define but the number of distinct label combinations those metrics emit. This is the single fact that drives every design decision below.

Prometheus distinguishes four metric types. A Counter is monotonic: it only increases and resets to zero when the process restarts, which is why queries always wrap it in rate() or increase(). A Gauge is a free-floating snapshot that rises and falls. A Histogram observes values into a fixed set of cumulative buckets and exposes _bucket, _sum, and _count series so that quantiles can be computed at query time. A Summary computes configurable quantiles inside the client process and exposes them directly alongside _sum and _count. The semantic differences and a full decision table live in choosing between counter, gauge, histogram, and summary.

The four metric types compared Counter, gauge, histogram and summary compared on four rows: the shape the value traces over time, the number of series emitted per label set, where quantiles are computed, and the relative cost inside the client process. Counter and gauge emit one series each at low cost, a histogram emits one series per bucket plus sum and count and computes quantiles on the Prometheus server, and a summary emits one series per configured quantile computed inside the process at the highest client cost. value over time series per label set quantile source client cost Counter Gauge Histogram Summary p50 p99 1 series _total only 1 series current value buckets + 1 plus _sum and _count 1 per quantile plus _sum and _count not applicable not applicable Prometheus server at query time client process at observe time low low moderate high
The four types side by side: what the value looks like over time, how many series each emits per label set, where the quantile is computed, and what it costs the emitting process.

The reason histograms dominate latency monitoring is aggregatability. Because every histogram exports raw bucket counts, the Prometheus server can sum buckets across all replicas of a service and then apply histogram_quantile() to the merged result. Summary quantiles are computed per process and cannot be averaged or summed without statistical nonsense, so they describe one instance only. For service-level objectives spanning a fleet, that distinction is decisive.

The four types also differ in what they cost the process that emits them. A Counter or Gauge is a single number guarded by a lock, so observing it is a dictionary lookup plus a short critical section, and the exposition footprint is one line per series. A Histogram allocates one counter per bucket plus a sum and a count, so its observe path is a bucket lookup and an increment, still cheap, but its exposition footprint is the bucket count plus two extra series per label combination. A Summary is the heaviest: it maintains a streaming quantile estimator per series, which costs more CPU per observation and more memory to hold the sketch. These costs scale with cardinality, so a histogram with many buckets attached to a high-cardinality label is doubly expensive — once for the buckets and once for the label fan-out. The right type is therefore a joint decision about semantics and cost, never semantics alone.

The server-side cost is easier to reason about than most teams assume. Each active series carries an entry in the inverted index plus an in-memory chunk that accumulates samples until it is flushed, and the practical figure most operators converge on is a few kilobytes of head memory per active series. A million active series is therefore measured in gigabytes of resident memory before you have run a single query, and queries themselves allocate proportionally to the number of series they touch. Sample rate barely moves this number; series count dominates it. That is why a service scraped every 15 seconds with 200 well-chosen series is free, and the same service with a user_id label is a capacity incident.

Two more model details matter in production. First, the OpenTelemetry data model maps onto these same shapes — a Counter maps to a monotonic Sum, a Gauge to a Gauge, and a Histogram to an explicit-bucket Histogram — but OpenTelemetry adds the notion of temporality (cumulative versus delta) that the Prometheus exposition format does not expose directly; the OpenTelemetry metrics SDK guide covers that mapping, and the trade-offs between the two pipelines are laid out in OpenTelemetry versus Prometheus for Python metrics. Second, none of these types tolerates an unbounded label, because every type multiplies its base series count by the cardinality of its labels, and for histograms that multiplier is applied to every bucket.

Step-by-Step Implementation

One request through the instruments A vertical timeline of four instrumentation points in a single request: the in-flight gauge is incremented on entry, a timer starts around the handler, the histogram observes the elapsed time, and the counter is incremented while the gauge is decremented in the finally block. Dashed connectors link steps one, three and four to the exposition lines they produce; step two produces no line because timing alone emits nothing. one request through the instruments IN_FLIGHT.labels(route).inc() gauge +1 — request enters the handler 1 start = time.perf_counter() timed region: the handler runs 2 LATENCY.labels(route).observe(dt) one bucket, _sum and _count updated 3 REQUESTS.inc() ; IN_FLIGHT.dec() in finally — failures counted too 4 resulting exposition lines GET /checkout → 200 in 31 ms 1 http_requests_in_flight{route="/checkout"} 1.0 3 http_request_duration_seconds_bucket{le="0.05"} 1.0 http_request_duration_seconds_sum 0.031 http_request_duration_seconds_count 1.0 4 http_requests_total{route="/checkout", status="200"} 1.0 http_requests_in_flight{route="/checkout"} 0.0 five samples, and not one new time series
A single request walks the instruments in order; step 2 emits nothing on its own, and every line it does produce lands in series that already existed.

Step 1 — Define a counter for event totals. Counters answer "how many" over time. Keep label sets small and bounded; the method and status fields below are bounded enumerations, not free text. Declare instruments once at module import so the same object is reused for the life of the process.

from prometheus_client import Counter

# method and status are bounded enumerations -> safe, low cardinality
REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests processed",
    labelnames=("method", "status"),
)

REQUESTS.labels(method="GET", status="200").inc()
REQUESTS.labels(method="POST", status="500").inc()

Step 2 — Define a gauge for current state. Gauges represent a value at a moment: queue depth, in-flight requests, connection pool size. Use inc, dec, or set. If the value is expensive to compute, prefer a callback-style collector so the work happens at scrape time rather than on the request path.

from prometheus_client import Gauge

IN_FLIGHT = Gauge(
    "http_requests_in_flight",
    "Requests currently being served",
)

IN_FLIGHT.inc()        # request started
# ... handle request ...
IN_FLIGHT.dec()        # request finished

Step 3 — Define a histogram with SLO-aligned buckets. The default buckets are general-purpose. For a latency SLO you must place bucket boundaries on the thresholds you actually report against, because histogram_quantile() interpolates linearly within a bucket and is only as precise as the boundary spacing.

from prometheus_client import Histogram

# Buckets chosen around a 250ms p99 SLO target, in seconds.
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency in seconds",
    labelnames=("route",),
    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)

with REQUEST_LATENCY.labels(route="/checkout").time():
    handle_checkout()

Step 4 — Use a summary only for single-instance, client-side quantiles. A summary is appropriate when you genuinely need a quantile from one process and will never aggregate it, and when you can tolerate the higher CPU cost of streaming quantile estimation.

from prometheus_client import Summary

# Quantiles are computed in-process; they describe THIS replica only.
GC_PAUSE = Summary(
    "gc_pause_seconds",
    "Garbage collection pause duration",
)

with GC_PAUSE.time():
    run_gc_cycle()

Step 5 — Bound every label at the call site. The cheapest cardinality control is a function that refuses to emit an unbounded value in the first place. Normalise the raw path into a route template and collapse anything unrecognised into a single other bucket, so a scanner probing random URLs cannot mint series.

from typing import Final

# The closed set of route templates this service is allowed to label with.
ROUTES: Final[frozenset[str]] = frozenset(
    {"/checkout", "/search", "/users/{id}", "/health"}
)

def bounded_route(matched_template: str | None) -> str:
    """Collapse anything outside the known set into one 'other' series."""
    if matched_template in ROUTES:
        return matched_template
    return "other"   # 404s, scanners and unmatched paths share ONE series

REQUEST_LATENCY.labels(route=bounded_route(request.url_rule)).observe(0.031)

Step 6 — Compute query-time quantiles from the histogram. With buckets in place, the server derives the p99 across all replicas. Summing by le first is what makes the result fleet-wide rather than per-instance; this is the query a latency SLO panel runs.

histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Configuration Reference

Setting Where Type Default Production value
labelnames instrument constructor tuple of str none Bounded enumerations only, 3-4 labels max
buckets Histogram(...) tuple of float (.005 … 10.0, +Inf) Override so a boundary sits on each SLO threshold
multiprocess_mode Gauge(...) str all livesum for additive gauges, liveall only if you need per-worker series
PROMETHEUS_MULTIPROC_DIR environment path unset Set for gunicorn/uwsgi; wipe the directory on boot
sample_limit scrape config int 0 (unlimited) 5000-20000 per target as a blast-radius guard
label_limit / label_value_length_limit scrape config int 0 (unlimited) 16 / 128 to reject accidental label sprawl
metric_relabel_configs scrape config list none Drop or aggregate offending labels before ingestion
Recording rule interval rule group duration global evaluation_interval 30s-60s for precomputed quantiles
Rule name form rule group str none level:metric:operations, e.g. route:http_request_duration_seconds:p99
OTEL_METRIC_EXPORT_INTERVAL environment int (ms) 60000 Match the scrape interval, typically 15000
Explicit bucket view OTel SDK View SDK default buckets ExplicitBucketHistogramAggregation with the same SLO boundaries

For the OpenTelemetry equivalents of these instruments and exporter wiring, see recording counters and histograms with OpenTelemetry and exporting OTLP metrics to the collector.

Async and Concurrency Considerations

Multiprocess exposition under gunicorn Each of four gunicorn workers keeps its own registry and writes memory-mapped files into the directory named by PROMETHEUS_MULTIPROC_DIR. A MultiProcessCollector reads every file at scrape time and generate_latest merges them into a single response on the metrics endpoint, so the file count grows with the union of label sets across all workers. gunicorn workers worker 1 own registry worker 2 own registry worker 3 own registry worker 4 own registry PROMETHEUS_MULTIPROC_DIR counter_101.db counter_102.db gauge_101.db gauge_102.db hist_101.db hist_102.db …_103.db …_104.db one file set per live worker MultiProcessCollector merges every mmap file generate_latest() GET /metrics one merged response file count grows with the union of label sets across all workers so cardinality costs twice: stored series and merge time on every scrape
Under gunicorn each worker owns its registry and writes memory-mapped files; the collector merges them per scrape, which makes label cardinality a file-count and latency problem as well as a storage one.

The Prometheus Python client's metric objects are process-global and thread-safe. Each labelled child holds its value in a MutexValue guarded by a threading.Lock, so inc, dec, set, and observe can be called from any thread or coroutine without application-level locking. The critical section is a few instructions long and the lock is per-series, so contention is invisible at ordinary request rates; the cost you can actually measure is the labels() lookup, which hashes the label tuple on every call. In a hot loop, hoist the child out: bind counter = REQUESTS.labels(method="GET", status="200") once and call counter.inc() inside the loop.

That safety does not extend across processes. Under a multi-process server such as gunicorn or uwsgi, each worker holds its own copy of every counter, and a default start_http_server would report only the worker that happened to serve the scrape. The fix is multiprocess mode: set PROMETHEUS_MULTIPROC_DIR to a writable directory and have each worker write its samples to memory-mapped files that a single collector aggregates at scrape time.

import os
from prometheus_client import CollectorRegistry, multiprocess, generate_latest

# Each gunicorn worker writes to PROMETHEUS_MULTIPROC_DIR; a registry
# backed by MultiProcessCollector merges them for the scrape response.
def metrics_app(environ, start_response):
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    data = generate_latest(registry)
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [data]

Multiprocess mode imposes a cardinality discipline of its own. Gauges must declare a multiprocess mode (livesum, liveall, min, max) because there is no single live value across workers, and each worker's distinct label children are stored separately, so the on-disk file set grows with the union of label combinations across all workers. High-cardinality labels therefore hurt twice under multiprocess: once in stored series and once in mmap file count and aggregation time on every scrape. Wipe the directory on process start, and register a multiprocess.mark_process_dead hook on worker exit, or dead workers' files accumulate until the scrape itself becomes the slowest endpoint in the service.

For asyncio services, observation is safe from any coroutine, but two details bite. First, timing must respect the event loop: use time.perf_counter() deltas around awaited work rather than the blocking with histogram.time(): form when the timed region contains awaits, because the context manager measures wall time correctly but wrapping the whole handler in it hides where the time actually went. Second, the scrape handler itself is synchronous work. Under multiprocess mode generate_latest() reads and merges every mmap file, which can take hundreds of milliseconds once cardinality grows, and doing that on the event loop stalls every in-flight request. Serve the exposition endpoint from a thread executor or from a dedicated port handled outside the loop. Finally, record one observation per logical operation; emitting an observation inside a tight inner loop multiplies sample volume and lock traffic without adding signal.

Label Cardinality and Why It Explodes

The number of time series a metric produces is the product of the distinct value counts of all its labels. A metric with a method label (about 5 values) and a status label (about 6 values) tops out near 30 series — trivial. Add a customer_id label drawn from a hundred thousand customers and the same metric now produces hundreds of thousands of series, each one a permanent in-memory index entry on the Prometheus server. Series count, not sample rate, is what exhausts Prometheus memory.

Series growth as labels are added On a logarithmic scale, one metric starts at a single series, grows to 5 with a method label, 30 with a status label and 600 with a route label — all far below the roughly one million series that costs gigabytes of head memory. Adding an unbounded user_id label of one hundred thousand values takes the same metric to sixty million series, far above that line. 1 100 10k 1M 100M series (log) ≈1M active series — gigabytes of head memory 1 5 30 600 60,000,000 base 1 metric + method × 5 + status × 6 + route × 20 + user_id × 100k series = product of label-value counts; one unbounded label ends the chart
Each bounded label multiplies the series count by a handful; a single unbounded label multiplies it by the size of your user base and clears the memory budget by orders of magnitude.

Three values are almost always cardinality bombs and must never become labels: user or customer IDs, request or trace IDs, and any free-form string such as a full URL path with embedded IDs, an email address, or an error message. Each of these is effectively unbounded, so every new request mints a new series. The same caution applies to span attributes, as covered in span lifecycle and attributes; high-cardinality identifiers belong in distributed traces and structured logs, where storage is per-event rather than per-series.

Churn is the failure mode teams miss, because it hides behind a cardinality number that looks acceptable. A label whose values rotate over time — a pod name, a container ID, a deployment hash, a build number — may only have a few dozen live values at any instant, yet every rollout replaces the whole set. The old series stop receiving samples but stay in the index for the retention period, so a service that deploys ten times a day multiplies its apparent cardinality by hundreds over a fortnight. Judge a label by the number of distinct values it will produce across your retention window, not by the number it holds right now.

Bound every label to a small, predictable set. Replace raw URL paths with the matched route template (/users/{id}, not /users/8123). Map open-ended error strings to a closed enum of error classes. If a value can grow with traffic or with your customer base, it does not belong in a label. The mechanics of detecting and trimming offending labels server-side — including metric_relabel_configs and dropping or aggregating labels with relabeling — are covered in controlling label cardinality in Prometheus.

Auditing what you already ship

Cardinality problems are found, not predicted. Prometheus exposes its own head-block statistics at /api/v1/status/tsdb, which ranks metric names and label names by series count and is the fastest way to identify the top offenders on a running server. Two queries cover the day-to-day checks: the first ranks metrics by series count, the second measures growth so you can catch a bad deploy before memory does.

# Which metric names own the most series right now?
topk(10, count by (__name__) ({__name__=~".+"}))

# Is total head series growing? Alert if this trends up after a deploy.
prometheus_tsdb_head_series

Add a scrape-time guard so a single bad deploy cannot take the server down: sample_limit fails the scrape if a target suddenly exposes more samples than expected, turning an unbounded label into one broken target rather than an out-of-memory event across the whole monitoring stack.

Keeping the high-cardinality dimension with exemplars

Dropping user_id from a label does not mean losing the ability to answer "which request was slow". Exemplars attach a trace_id to individual histogram observations, so a spike on a latency panel links directly to the trace that produced it, while the metric itself stays low-cardinality. The identifier lives in the trace, where storage is per-event and sampling already controls the volume, and the same trace_id in your logs closes the loop — see adding trace IDs to log records.

from opentelemetry import trace

span = trace.get_current_span().get_span_context()
# Exemplar keeps the identifier OUT of the label set; it rides with one sample.
REQUEST_LATENCY.labels(route="/checkout").observe(
    0.031, exemplar={"trace_id": format(span.trace_id, "032x")}
)

Histogram Bucket Design for Latency SLOs

Histogram accuracy is entirely a function of bucket placement. histogram_quantile() assumes values are uniformly distributed within each bucket and interpolates linearly between boundaries. If your SLO is "99% of checkout requests under 250ms" but your nearest bucket boundaries are 100ms and 500ms, the computed p99 can be off by hundreds of milliseconds because the estimator has no resolution between those edges.

Coarse versus SLO-aligned bucket boundaries An observed latency distribution peaks near 130 milliseconds and trails to the right. Below it, a coarse bucket ladder has boundaries only at 100 milliseconds, 500 milliseconds and 1 second, leaving the 250 millisecond objective inside a 400 millisecond gap that histogram_quantile must interpolate across. A tighter ladder places boundaries at 100, 200, 250, 300 and 500 milliseconds, putting an exact edge on the objective. SLO p99 = 250 ms latency distribution request latency → coarse 100 ms 500 ms 1 s no boundary near the SLO → p99 interpolated across a 400 ms gap tight a boundary ON 250 ms → exact good-event ratio, no interpolation 100 200 250 300 500 ms
Bucket accuracy is boundary placement: the coarse ladder leaves the objective stranded inside a 400 ms bucket, while an edge on 250 ms makes the SLO ratio exact.

Place a boundary exactly on each SLO threshold, then add a few boundaries on either side to capture the shape of the distribution. Spacing should be roughly geometric across the operating range and tight near the threshold you report against. Remember the cost: every bucket is an extra time series, and that count is multiplied by every other label on the histogram. A histogram with 12 buckets and a route label of 20 values produces 240 _bucket series (the +Inf bucket is always added on top of the boundaries you declare) plus 20 _sum and 20 _count series. Keep bucket counts modest and route cardinality bounded together.

A useful starting layout for HTTP latency is a roughly geometric ladder from ten milliseconds to a few seconds with extra resolution concentrated around the SLO threshold. If the SLO is a 250ms p99, boundaries at 100ms, 200ms, 250ms, 300ms, and 500ms give the estimator three nearby edges to interpolate against, which keeps the computed p99 within a small fraction of the true value. Boundaries far above the threshold still matter for catching tail blowups, but they need not be dense. Resist the temptation to add buckets everywhere "to be safe": each one is a permanent series multiplied by every label, and twenty buckets on a metric with a moderate route label can quietly become the largest metric in the system.

A boundary on the threshold buys something beyond quantile accuracy. Once le="0.25" exists, the SLO itself becomes a simple ratio — the fraction of observations at or below the threshold — with no interpolation at all, which is the numerically stable way to drive an error-budget burn-rate alert.

# Exact good-event ratio: no interpolation, because 0.25 IS a boundary.
sum(rate(http_request_duration_seconds_bucket{le="0.25"}[5m]))
  /
sum(rate(http_request_duration_seconds_count[5m]))

Validate bucket placement empirically rather than by intuition. After deploying a new layout, compare the histogram-derived p99 against a short-lived summary or against raw request logs for the same window. If they disagree by more than your error tolerance at the SLO threshold, the buckets are too coarse there and need a tighter boundary. This is the same discipline applied to span attribute limits described in span lifecycle and attributes: measure the cost and accuracy of your telemetry shape, do not assume it.

Native histograms (the newer Prometheus exponential-bucket format) sidestep manual boundary selection by storing exponentially spaced buckets compactly, but classic explicit buckets remain the portable default for the Python client and are what most existing dashboards expect.

Summary vs Histogram Trade-offs

Property Histogram Summary
Where quantiles compute Prometheus server, query time Client process, scrape time
Aggregatable across replicas Yes, via bucket sums No
Configurable quantiles after the fact Yes No, fixed at definition
Client CPU cost Low (bucket increment) Higher (streaming estimation)
Series per label combination buckets + 1, plus _sum and _count one per quantile, plus _sum and _count
Error bound Bucket-width interpolation Per-quantile target error
Best for Fleet-wide SLOs, burn rate Single-instance diagnostics
Choosing the metric type, then passing the cardinality gate Starting from the shape of the value: a number that only ever increases becomes a Counter read through rate; a value that rises and falls becomes a Gauge; a distribution splits on whether it must be aggregated across replicas, giving a Histogram when it must and a Summary only when it never will. Every leaf then passes a cardinality gate asking whether each label is a bounded enumeration. What shape is the value? only ever increases resets to 0 on restart rises and falls a current reading a distribution latency, payload size Counter rate(x[5m]) Gauge read the current value aggregate across replicas (fleet SLO)? yes no Histogram histogram_quantile() Summary one process only Cardinality gate — is every label on the chosen type a bounded enum? if not, the type no longer matters: move the identifier to traces or logs first
The type follows from the shape of the value, but every leaf passes through the same gate — an unbounded label makes the semantically perfect type the wrong one.

The series arithmetic is worth spelling out, because it is the one place a summary can look cheaper than it is. A histogram with nine declared boundaries emits eleven series per label combination: ten _bucket series once +Inf is added, plus _sum and _count. A summary configured with three quantiles emits five: three quantile series plus _sum and _count. The summary is smaller on the wire, but you have traded away every quantile you did not think to configure, and you cannot recover a p99.9 later from data that only ever recorded p50, p90, and p99. The Python client also makes this moot in most cases, since prometheus_client ships summaries without quantiles by default — you get _sum and _count only, which is a mean, not a distribution.

The practical rule: reach for a histogram by default, and only choose a summary when you need a precise quantile from exactly one process and will never aggregate it.

Recording Rules for Query-Time Aggregation

histogram_quantile() over a high-cardinality rate() of bucket series is one of the most expensive queries a dashboard can run, and re-running it on every panel refresh is wasteful. Recording rules precompute the expression on the server at a fixed interval and write the result to a new, lower-cardinality series that dashboards and alerts read cheaply.

Recording rules between raw buckets and dashboards Twenty routes times twelve buckets give roughly 240 raw bucket series. A rule group evaluates histogram_quantile over their summed rates every thirty seconds and writes twenty recorded series named route colon http_request_duration_seconds colon p99, which the dashboard panel and the alert rule read directly instead of re-running the expensive query. raw scraped series 20 routes × 12 buckets ≈ 240 _bucket series rate() rule evaluation every 30 s, server-side histogram_quantile(0.99,…) writes recorded series (20 total) route:…:p99{"/checkout"} route:…:p99{"/search"} route:…:p99{"/health"} dashboard alert rule recording rules cut query cost, not the cardinality of the raw scrape
The expensive quantile runs once per interval on the server; panels and alerts then read a handful of cheap precomputed series instead of re-deriving them on every refresh.
groups:
  - name: http_slo
    interval: 30s
    rules:
      # Precompute per-route p99 once; panels read this series directly.
      - record: route:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
          )

Expected Output: querying the recorded series returns the precomputed quantile per route.

route:http_request_duration_seconds:p99{route="/checkout"}  0.214
route:http_request_duration_seconds:p99{route="/search"}    0.087

Name recorded series with the conventional level:metric:operations form — the aggregation level first, then the source metric, then what was done to it — so route:http_request_duration_seconds:p99 tells a reader exactly what it contains without opening the rule file. Keep rules that feed each other in the same group, because groups evaluate sequentially and a rule can safely read a series recorded earlier in its own group but not one recorded in a group that may run later.

Recording rules reduce query cost and the cardinality of the derived series your dashboards touch, but they do not change the cardinality of the raw scraped series. To cut the raw series count you must fix the instrumentation or relabel at scrape time.

Production Code Examples

The example service end to end A request with a route bounded to a closed set enters handle, which updates an in-flight gauge, a latency histogram and a request counter. The client exposes them on port 8000, Prometheus scrapes that endpoint every fifteen seconds into its TSDB, a recording rule precomputes the per-route p99 every thirty seconds, and a dashboard panel and burn-rate alert read the recorded series. request route → bounded handle() IN_FLIGHT · Gauge LATENCY · Histogram REQUESTS · Counter :8000/metrics exposition text scrape Prometheus TSDB, 15 s recording rule · 30 s p99 panel burn-rate alert labels bounded at the call site: route ∈ {/checkout, /search, /health, other} so the series count is fixed no matter what traffic arrives
The whole path for the example service: three instruments in one handler, a scraped exposition endpoint, one recording rule, and the two consumers that read it.

This end-to-end Flask-style handler uses the right type for each measurement, keeps every label bounded, and exposes the metrics on a dedicated port. For the framework integration details, see instrumenting Flask with Prometheus metrics and the wider walkthrough in exposing custom metrics with the Prometheus client.

import time
from typing import Final
from prometheus_client import Counter, Gauge, Histogram, start_http_server

# 1. Instruments declared once at import; labels are bounded enumerations.
REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests",
    labelnames=("method", "route", "status"),
)
IN_FLIGHT = Gauge(
    "http_requests_in_flight", "In-flight requests",
    labelnames=("route",),
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency",
    labelnames=("route",),
    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)

# 2. The closed label domain: anything else collapses to one series.
ROUTES: Final[frozenset[str]] = frozenset({"/checkout", "/search", "/health"})

def bounded_route(template: str | None) -> str:
    return template if template in ROUTES else "other"

def handle(method: str, template: str | None) -> int:
    # 3. route is the matched template, never the raw path -> bounded label
    route = bounded_route(template)
    IN_FLIGHT.labels(route=route).inc()
    start = time.perf_counter()
    status = 500
    try:
        time.sleep(0.03)          # stand-in for real work
        status = 200
        return status
    finally:
        # 4. Record in finally so failures are counted, not silently dropped.
        LATENCY.labels(route=route).observe(time.perf_counter() - start)
        REQUESTS.labels(method=method, route=route, status=str(status)).inc()
        IN_FLIGHT.labels(route=route).dec()

if __name__ == "__main__":
    start_http_server(8000)  # exposition endpoint on :8000/metrics
    handle("GET", "/checkout")
    handle("GET", "/wp-admin.php")   # unmatched -> route="other"

Expected Output: scraping http://localhost:8000/metrics returns the exposition text. Note that the scanner request produced no new label value — it landed in the shared other series.

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/checkout",status="200"} 1.0
http_requests_total{method="GET",route="other",status="200"} 1.0
# HELP http_request_duration_seconds Request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/checkout",le="0.025"} 0.0
http_request_duration_seconds_bucket{route="/checkout",le="0.05"} 1.0
http_request_duration_seconds_bucket{route="/checkout",le="+Inf"} 1.0
http_request_duration_seconds_sum{route="/checkout"} 0.0306
http_request_duration_seconds_count{route="/checkout"} 1.0
http_requests_in_flight{route="/checkout"} 0.0

The second example is the guard rail: a unit test that asserts the label domain stays closed. Cardinality regressions arrive as ordinary pull requests, so the cheapest place to catch one is CI, before the series ever reach the server.

from prometheus_client import REGISTRY

def test_route_label_stays_bounded() -> None:
    """Fail the build if a new route label value escapes the closed set."""
    handle("GET", "/users/8123")     # a raw path with an embedded ID
    observed = {
        sample.labels["route"]
        for metric in REGISTRY.collect()
        if metric.name == "http_requests"
        for sample in metric.samples
    }
    assert observed <= ROUTES | {"other"}, f"unbounded label leaked: {observed}"

Expected Output: the test passes because bounded_route collapsed the raw path, and it would fail loudly the moment someone labels with request.path directly.

test_metrics.py::test_route_label_stays_bounded PASSED

Common Mistakes

  • Error signature: Prometheus memory climbs continuously and prometheus_tsdb_head_series grows without bound. Root cause: a label such as user_id or request_id mints a new series per request. Remediation: remove the label from the instrument and move the identifier into traces or logs; if it is already deployed, drop it at scrape time with metric_relabel_configs while the fix ships.
  • Error signature: the dashboard p99 disagrees with measured latency at the SLO threshold. Root cause: the default buckets are in use and no boundary sits on the threshold, so the interpolated quantile is coarse. Remediation: override buckets to place a boundary on the SLO value and tighten spacing on either side of it.
  • Error signature: a fleet-wide p99 panel returns meaningless or per-instance values. Root cause: a Summary was chosen and its quantiles are computed inside each process, so they cannot be merged. Remediation: switch to a Histogram and compute the quantile with histogram_quantile() over summed bucket rates.
  • Error signature: graphs show jagged drops to zero on every deploy. Root cause: a counter's raw value is being read instead of its rate, so restarts look like data loss. Remediation: always query counters through rate() or increase(), which detect and correct for resets.
  • Error signature: counters appear to halve or jump randomly between scrapes, and totals are far below reality. Root cause: gunicorn workers each hold an independent registry and the scrape hits one worker at a time. Remediation: enable multiprocess mode with PROMETHEUS_MULTIPROC_DIR and a MultiProcessCollector, declare a multiprocess_mode on every gauge, and clear the directory on boot.
  • Error signature: cardinality looks fine in a spot check but the server runs out of memory a week later. Root cause: a rotating label such as a pod name or build hash churns its whole value set on every deploy, so series accumulate across the retention window. Remediation: drop the rotating label with relabeling, or replace it with a stable one such as service name and version.

When to Reach for Each Type

Field guide: question, example, query Four cells. Counter answers how many events so far, for requests, errors and retries, read with rate over five minutes. Gauge answers what the value is right now, for queue depth, in-flight requests and pool size, read directly. Histogram answers how the value is distributed fleet-wide, for request latency and payload size, read with histogram_quantile over summed bucket rates. Summary answers what quantile this one process saw, for a garbage-collection pause in a single long-lived worker, read from its quantile label and never aggregated. one question, one example, one query per type Counter “how many events so far?” requests, errors, retries, bytes sent rate(http_requests_total[5m]) always a rate — never the raw value Gauge “what is it right now?” queue depth, in-flight, pool size max_over_time(queue_depth[5m]) read directly — no rate() Histogram “how is it distributed, fleet-wide?” request latency, payload size histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) Summary “what quantile did this process see?” GC pause in one long-lived worker gc_pause_seconds{quantile="0.99"} never aggregated across replicas
The everyday version of the decision: the question you are asking, something in a real service that fits it, and the query you will end up writing.

A short field guide ties the semantics back to everyday decisions. Reach for a Counter whenever you would naturally say "number of," because the rate of a counter is the throughput or error rate you actually want on a dashboard. Reach for a Gauge whenever a single instantaneous reading is the answer and the value can fall as well as rise; if you find yourself resetting a counter to model a falling value, you wanted a gauge. Reach for a Histogram whenever the question is "how is this value distributed across requests," especially for latency and payload size, and whenever the answer must hold across more than one replica. Reach for a Summary only in the narrow case of a precise quantile from a single, long-lived process where aggregation will never apply, accepting its higher CPU cost in exchange for a tight per-quantile error bound.

The cardinality lens overrides all of the above when they conflict. A type that is semantically perfect but attached to an unbounded label is the wrong choice, because the series explosion will cost more than the missing signal. When that tension appears, keep the high-cardinality dimension out of metrics entirely and recover it from traces or structured logs, then choose the metric type for the bounded view that remains. This is why the type decision and the cardinality decision in this guide are two halves of one design step, not separate concerns.

Frequently Asked Questions

How many time series does one labeled metric actually create?

One time series exists for every unique combination of metric name and label values. A metric with two labels of 50 and 20 distinct values produces up to 1000 series, and adding a third label multiplies that figure again. For a histogram, multiply once more by the number of buckets plus two.

Should I use a Histogram or a Summary for latency SLOs?

Use a Histogram. Histogram buckets are aggregatable across instances on the Prometheus server, so you can compute global quantiles and error-budget burn rates, while Summary quantiles are pre-computed per process and cannot be merged.

Why are user IDs and request IDs bad as metric labels?

They are unbounded high-cardinality values. Each new ID creates a fresh time series that occupies index memory long after it stops receiving samples, which inflates server memory, slows queries, and can crash Prometheus. Keep that detail in traces and structured logs instead.

What is the difference between a Counter and a Gauge?

A Counter only ever increases and resets to zero on restart, so it answers how many events have happened over time and must be read through rate() or increase(). A Gauge can go up and down and represents a current value such as queue depth or memory usage.

Do recording rules reduce cardinality?

Recording rules precompute expensive expressions and can aggregate away labels, lowering query cost and the cardinality of the derived series your dashboards touch. They do not reduce the cardinality of the raw series being scraped — only fixing the instrumentation or relabeling at scrape time does that.