Metric Types and Cardinality Control in Python
Choosing the right metric type and bounding label cardinality are the two decisions that determine whether a Python service produces cheap, queryable telemetry or an unmaintainable time-series explosion. This guide covers counter, gauge, histogram, and summary semantics, when each applies, how labels multiply series count, the high-cardinality anti-patterns that destroy Prometheus, histogram bucket design for latency SLOs, the quantile trade-off between summaries and histograms, and recording rules for query-time aggregation. It is part of the Python Metrics and Instrumentation guide. For the library mechanics referenced throughout, see Prometheus client instrumentation and the OpenTelemetry metrics SDK guides, and for two focused deep dives see controlling label cardinality in Prometheus and choosing between counter, gauge, histogram, and summary.
Prerequisites
Install pinned client libraries. The examples use the official Prometheus client and the OpenTelemetry metrics SDK.
pip install "prometheus-client>=0.20.0,<1.0.0"
pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
"opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"
Two environment variables matter for the material below. Set PROMETHEUS_MULTIPROC_DIR to a writable, empty-on-boot directory whenever the service runs under gunicorn or uwsgi, and set OTEL_METRIC_EXPORT_INTERVAL if you export OTLP and want the push cadence to match your scrape interval.
export PROMETHEUS_MULTIPROC_DIR=/dev/shm/prom-metrics # cleared on each boot
export OTEL_METRIC_EXPORT_INTERVAL=15000 # milliseconds
A Prometheus server (>=2.50,<3.0) scraping the exposition endpoint is assumed for the recording-rule and relabeling material. Python 3.10+ is assumed for the type annotations used in the snippets.
Concept and Architecture
A metric in the Prometheus data model is a named numeric measurement plus an optional set of key-value labels. The unit of storage is the time series: one append-only stream of timestamped samples, identified by the metric name and the exact set of label values. The cost of a metric on the backend is therefore not the number of metric names you define but the number of distinct label combinations those metrics emit. This is the single fact that drives every design decision below.
Prometheus distinguishes four metric types. A Counter is monotonic: it only increases and resets to zero when the process restarts, which is why queries always wrap it in rate() or increase(). A Gauge is a free-floating snapshot that rises and falls. A Histogram observes values into a fixed set of cumulative buckets and exposes _bucket, _sum, and _count series so that quantiles can be computed at query time. A Summary computes configurable quantiles inside the client process and exposes them directly alongside _sum and _count. The semantic differences and a full decision table live in choosing between counter, gauge, histogram, and summary.
The reason histograms dominate latency monitoring is aggregatability. Because every histogram exports raw bucket counts, the Prometheus server can sum buckets across all replicas of a service and then apply histogram_quantile() to the merged result. Summary quantiles are computed per process and cannot be averaged or summed without statistical nonsense, so they describe one instance only. For service-level objectives spanning a fleet, that distinction is decisive.
The four types also differ in what they cost the process that emits them. A Counter or Gauge is a single number guarded by a lock, so observing it is a dictionary lookup plus a short critical section, and the exposition footprint is one line per series. A Histogram allocates one counter per bucket plus a sum and a count, so its observe path is a bucket lookup and an increment, still cheap, but its exposition footprint is the bucket count plus two extra series per label combination. A Summary is the heaviest: it maintains a streaming quantile estimator per series, which costs more CPU per observation and more memory to hold the sketch. These costs scale with cardinality, so a histogram with many buckets attached to a high-cardinality label is doubly expensive — once for the buckets and once for the label fan-out. The right type is therefore a joint decision about semantics and cost, never semantics alone.
The server-side cost is easier to reason about than most teams assume. Each active series carries an entry in the inverted index plus an in-memory chunk that accumulates samples until it is flushed, and the practical figure most operators converge on is a few kilobytes of head memory per active series. A million active series is therefore measured in gigabytes of resident memory before you have run a single query, and queries themselves allocate proportionally to the number of series they touch. Sample rate barely moves this number; series count dominates it. That is why a service scraped every 15 seconds with 200 well-chosen series is free, and the same service with a user_id label is a capacity incident.
Two more model details matter in production. First, the OpenTelemetry data model maps onto these same shapes — a Counter maps to a monotonic Sum, a Gauge to a Gauge, and a Histogram to an explicit-bucket Histogram — but OpenTelemetry adds the notion of temporality (cumulative versus delta) that the Prometheus exposition format does not expose directly; the OpenTelemetry metrics SDK guide covers that mapping, and the trade-offs between the two pipelines are laid out in OpenTelemetry versus Prometheus for Python metrics. Second, none of these types tolerates an unbounded label, because every type multiplies its base series count by the cardinality of its labels, and for histograms that multiplier is applied to every bucket.
Step-by-Step Implementation
Step 1 — Define a counter for event totals. Counters answer "how many" over time. Keep label sets small and bounded; the method and status fields below are bounded enumerations, not free text. Declare instruments once at module import so the same object is reused for the life of the process.
from prometheus_client import Counter
# method and status are bounded enumerations -> safe, low cardinality
REQUESTS = Counter(
"http_requests_total",
"Total HTTP requests processed",
labelnames=("method", "status"),
)
REQUESTS.labels(method="GET", status="200").inc()
REQUESTS.labels(method="POST", status="500").inc()
Step 2 — Define a gauge for current state. Gauges represent a value at a moment: queue depth, in-flight requests, connection pool size. Use inc, dec, or set. If the value is expensive to compute, prefer a callback-style collector so the work happens at scrape time rather than on the request path.
from prometheus_client import Gauge
IN_FLIGHT = Gauge(
"http_requests_in_flight",
"Requests currently being served",
)
IN_FLIGHT.inc() # request started
# ... handle request ...
IN_FLIGHT.dec() # request finished
Step 3 — Define a histogram with SLO-aligned buckets. The default buckets are general-purpose. For a latency SLO you must place bucket boundaries on the thresholds you actually report against, because histogram_quantile() interpolates linearly within a bucket and is only as precise as the boundary spacing.
from prometheus_client import Histogram
# Buckets chosen around a 250ms p99 SLO target, in seconds.
REQUEST_LATENCY = Histogram(
"http_request_duration_seconds",
"Request latency in seconds",
labelnames=("route",),
buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
with REQUEST_LATENCY.labels(route="/checkout").time():
handle_checkout()
Step 4 — Use a summary only for single-instance, client-side quantiles. A summary is appropriate when you genuinely need a quantile from one process and will never aggregate it, and when you can tolerate the higher CPU cost of streaming quantile estimation.
from prometheus_client import Summary
# Quantiles are computed in-process; they describe THIS replica only.
GC_PAUSE = Summary(
"gc_pause_seconds",
"Garbage collection pause duration",
)
with GC_PAUSE.time():
run_gc_cycle()
Step 5 — Bound every label at the call site. The cheapest cardinality control is a function that refuses to emit an unbounded value in the first place. Normalise the raw path into a route template and collapse anything unrecognised into a single other bucket, so a scanner probing random URLs cannot mint series.
from typing import Final
# The closed set of route templates this service is allowed to label with.
ROUTES: Final[frozenset[str]] = frozenset(
{"/checkout", "/search", "/users/{id}", "/health"}
)
def bounded_route(matched_template: str | None) -> str:
"""Collapse anything outside the known set into one 'other' series."""
if matched_template in ROUTES:
return matched_template
return "other" # 404s, scanners and unmatched paths share ONE series
REQUEST_LATENCY.labels(route=bounded_route(request.url_rule)).observe(0.031)
Step 6 — Compute query-time quantiles from the histogram. With buckets in place, the server derives the p99 across all replicas. Summing by le first is what makes the result fleet-wide rather than per-instance; this is the query a latency SLO panel runs.
histogram_quantile(
0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
Configuration Reference
| Setting | Where | Type | Default | Production value |
|---|---|---|---|---|
labelnames |
instrument constructor | tuple of str | none | Bounded enumerations only, 3-4 labels max |
buckets |
Histogram(...) |
tuple of float | (.005 … 10.0, +Inf) |
Override so a boundary sits on each SLO threshold |
multiprocess_mode |
Gauge(...) |
str | all |
livesum for additive gauges, liveall only if you need per-worker series |
PROMETHEUS_MULTIPROC_DIR |
environment | path | unset | Set for gunicorn/uwsgi; wipe the directory on boot |
sample_limit |
scrape config | int | 0 (unlimited) |
5000-20000 per target as a blast-radius guard |
label_limit / label_value_length_limit |
scrape config | int | 0 (unlimited) |
16 / 128 to reject accidental label sprawl |
metric_relabel_configs |
scrape config | list | none | Drop or aggregate offending labels before ingestion |
Recording rule interval |
rule group | duration | global evaluation_interval |
30s-60s for precomputed quantiles |
| Rule name form | rule group | str | none | level:metric:operations, e.g. route:http_request_duration_seconds:p99 |
OTEL_METRIC_EXPORT_INTERVAL |
environment | int (ms) | 60000 |
Match the scrape interval, typically 15000 |
| Explicit bucket view | OTel SDK | View |
SDK default buckets | ExplicitBucketHistogramAggregation with the same SLO boundaries |
For the OpenTelemetry equivalents of these instruments and exporter wiring, see recording counters and histograms with OpenTelemetry and exporting OTLP metrics to the collector.
Async and Concurrency Considerations
The Prometheus Python client's metric objects are process-global and thread-safe. Each labelled child holds its value in a MutexValue guarded by a threading.Lock, so inc, dec, set, and observe can be called from any thread or coroutine without application-level locking. The critical section is a few instructions long and the lock is per-series, so contention is invisible at ordinary request rates; the cost you can actually measure is the labels() lookup, which hashes the label tuple on every call. In a hot loop, hoist the child out: bind counter = REQUESTS.labels(method="GET", status="200") once and call counter.inc() inside the loop.
That safety does not extend across processes. Under a multi-process server such as gunicorn or uwsgi, each worker holds its own copy of every counter, and a default start_http_server would report only the worker that happened to serve the scrape. The fix is multiprocess mode: set PROMETHEUS_MULTIPROC_DIR to a writable directory and have each worker write its samples to memory-mapped files that a single collector aggregates at scrape time.
import os
from prometheus_client import CollectorRegistry, multiprocess, generate_latest
# Each gunicorn worker writes to PROMETHEUS_MULTIPROC_DIR; a registry
# backed by MultiProcessCollector merges them for the scrape response.
def metrics_app(environ, start_response):
registry = CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
data = generate_latest(registry)
start_response("200 OK", [("Content-Type", "text/plain")])
return [data]
Multiprocess mode imposes a cardinality discipline of its own. Gauges must declare a multiprocess mode (livesum, liveall, min, max) because there is no single live value across workers, and each worker's distinct label children are stored separately, so the on-disk file set grows with the union of label combinations across all workers. High-cardinality labels therefore hurt twice under multiprocess: once in stored series and once in mmap file count and aggregation time on every scrape. Wipe the directory on process start, and register a multiprocess.mark_process_dead hook on worker exit, or dead workers' files accumulate until the scrape itself becomes the slowest endpoint in the service.
For asyncio services, observation is safe from any coroutine, but two details bite. First, timing must respect the event loop: use time.perf_counter() deltas around awaited work rather than the blocking with histogram.time(): form when the timed region contains awaits, because the context manager measures wall time correctly but wrapping the whole handler in it hides where the time actually went. Second, the scrape handler itself is synchronous work. Under multiprocess mode generate_latest() reads and merges every mmap file, which can take hundreds of milliseconds once cardinality grows, and doing that on the event loop stalls every in-flight request. Serve the exposition endpoint from a thread executor or from a dedicated port handled outside the loop. Finally, record one observation per logical operation; emitting an observation inside a tight inner loop multiplies sample volume and lock traffic without adding signal.
Label Cardinality and Why It Explodes
The number of time series a metric produces is the product of the distinct value counts of all its labels. A metric with a method label (about 5 values) and a status label (about 6 values) tops out near 30 series — trivial. Add a customer_id label drawn from a hundred thousand customers and the same metric now produces hundreds of thousands of series, each one a permanent in-memory index entry on the Prometheus server. Series count, not sample rate, is what exhausts Prometheus memory.
Three values are almost always cardinality bombs and must never become labels: user or customer IDs, request or trace IDs, and any free-form string such as a full URL path with embedded IDs, an email address, or an error message. Each of these is effectively unbounded, so every new request mints a new series. The same caution applies to span attributes, as covered in span lifecycle and attributes; high-cardinality identifiers belong in distributed traces and structured logs, where storage is per-event rather than per-series.
Churn is the failure mode teams miss, because it hides behind a cardinality number that looks acceptable. A label whose values rotate over time — a pod name, a container ID, a deployment hash, a build number — may only have a few dozen live values at any instant, yet every rollout replaces the whole set. The old series stop receiving samples but stay in the index for the retention period, so a service that deploys ten times a day multiplies its apparent cardinality by hundreds over a fortnight. Judge a label by the number of distinct values it will produce across your retention window, not by the number it holds right now.
Bound every label to a small, predictable set. Replace raw URL paths with the matched route template (/users/{id}, not /users/8123). Map open-ended error strings to a closed enum of error classes. If a value can grow with traffic or with your customer base, it does not belong in a label. The mechanics of detecting and trimming offending labels server-side — including metric_relabel_configs and dropping or aggregating labels with relabeling — are covered in controlling label cardinality in Prometheus.
Auditing what you already ship
Cardinality problems are found, not predicted. Prometheus exposes its own head-block statistics at /api/v1/status/tsdb, which ranks metric names and label names by series count and is the fastest way to identify the top offenders on a running server. Two queries cover the day-to-day checks: the first ranks metrics by series count, the second measures growth so you can catch a bad deploy before memory does.
# Which metric names own the most series right now?
topk(10, count by (__name__) ({__name__=~".+"}))
# Is total head series growing? Alert if this trends up after a deploy.
prometheus_tsdb_head_series
Add a scrape-time guard so a single bad deploy cannot take the server down: sample_limit fails the scrape if a target suddenly exposes more samples than expected, turning an unbounded label into one broken target rather than an out-of-memory event across the whole monitoring stack.
Keeping the high-cardinality dimension with exemplars
Dropping user_id from a label does not mean losing the ability to answer "which request was slow". Exemplars attach a trace_id to individual histogram observations, so a spike on a latency panel links directly to the trace that produced it, while the metric itself stays low-cardinality. The identifier lives in the trace, where storage is per-event and sampling already controls the volume, and the same trace_id in your logs closes the loop — see adding trace IDs to log records.
from opentelemetry import trace
span = trace.get_current_span().get_span_context()
# Exemplar keeps the identifier OUT of the label set; it rides with one sample.
REQUEST_LATENCY.labels(route="/checkout").observe(
0.031, exemplar={"trace_id": format(span.trace_id, "032x")}
)
Histogram Bucket Design for Latency SLOs
Histogram accuracy is entirely a function of bucket placement. histogram_quantile() assumes values are uniformly distributed within each bucket and interpolates linearly between boundaries. If your SLO is "99% of checkout requests under 250ms" but your nearest bucket boundaries are 100ms and 500ms, the computed p99 can be off by hundreds of milliseconds because the estimator has no resolution between those edges.
Place a boundary exactly on each SLO threshold, then add a few boundaries on either side to capture the shape of the distribution. Spacing should be roughly geometric across the operating range and tight near the threshold you report against. Remember the cost: every bucket is an extra time series, and that count is multiplied by every other label on the histogram. A histogram with 12 buckets and a route label of 20 values produces 240 _bucket series (the +Inf bucket is always added on top of the boundaries you declare) plus 20 _sum and 20 _count series. Keep bucket counts modest and route cardinality bounded together.
A useful starting layout for HTTP latency is a roughly geometric ladder from ten milliseconds to a few seconds with extra resolution concentrated around the SLO threshold. If the SLO is a 250ms p99, boundaries at 100ms, 200ms, 250ms, 300ms, and 500ms give the estimator three nearby edges to interpolate against, which keeps the computed p99 within a small fraction of the true value. Boundaries far above the threshold still matter for catching tail blowups, but they need not be dense. Resist the temptation to add buckets everywhere "to be safe": each one is a permanent series multiplied by every label, and twenty buckets on a metric with a moderate route label can quietly become the largest metric in the system.
A boundary on the threshold buys something beyond quantile accuracy. Once le="0.25" exists, the SLO itself becomes a simple ratio — the fraction of observations at or below the threshold — with no interpolation at all, which is the numerically stable way to drive an error-budget burn-rate alert.
# Exact good-event ratio: no interpolation, because 0.25 IS a boundary.
sum(rate(http_request_duration_seconds_bucket{le="0.25"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
Validate bucket placement empirically rather than by intuition. After deploying a new layout, compare the histogram-derived p99 against a short-lived summary or against raw request logs for the same window. If they disagree by more than your error tolerance at the SLO threshold, the buckets are too coarse there and need a tighter boundary. This is the same discipline applied to span attribute limits described in span lifecycle and attributes: measure the cost and accuracy of your telemetry shape, do not assume it.
Native histograms (the newer Prometheus exponential-bucket format) sidestep manual boundary selection by storing exponentially spaced buckets compactly, but classic explicit buckets remain the portable default for the Python client and are what most existing dashboards expect.
Summary vs Histogram Trade-offs
| Property | Histogram | Summary |
|---|---|---|
| Where quantiles compute | Prometheus server, query time | Client process, scrape time |
| Aggregatable across replicas | Yes, via bucket sums | No |
| Configurable quantiles after the fact | Yes | No, fixed at definition |
| Client CPU cost | Low (bucket increment) | Higher (streaming estimation) |
| Series per label combination | buckets + 1, plus _sum and _count |
one per quantile, plus _sum and _count |
| Error bound | Bucket-width interpolation | Per-quantile target error |
| Best for | Fleet-wide SLOs, burn rate | Single-instance diagnostics |
The series arithmetic is worth spelling out, because it is the one place a summary can look cheaper than it is. A histogram with nine declared boundaries emits eleven series per label combination: ten _bucket series once +Inf is added, plus _sum and _count. A summary configured with three quantiles emits five: three quantile series plus _sum and _count. The summary is smaller on the wire, but you have traded away every quantile you did not think to configure, and you cannot recover a p99.9 later from data that only ever recorded p50, p90, and p99. The Python client also makes this moot in most cases, since prometheus_client ships summaries without quantiles by default — you get _sum and _count only, which is a mean, not a distribution.
The practical rule: reach for a histogram by default, and only choose a summary when you need a precise quantile from exactly one process and will never aggregate it.
Recording Rules for Query-Time Aggregation
histogram_quantile() over a high-cardinality rate() of bucket series is one of the most expensive queries a dashboard can run, and re-running it on every panel refresh is wasteful. Recording rules precompute the expression on the server at a fixed interval and write the result to a new, lower-cardinality series that dashboards and alerts read cheaply.
groups:
- name: http_slo
interval: 30s
rules:
# Precompute per-route p99 once; panels read this series directly.
- record: route:http_request_duration_seconds:p99
expr: |
histogram_quantile(
0.99,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
Expected Output: querying the recorded series returns the precomputed quantile per route.
route:http_request_duration_seconds:p99{route="/checkout"} 0.214
route:http_request_duration_seconds:p99{route="/search"} 0.087
Name recorded series with the conventional level:metric:operations form — the aggregation level first, then the source metric, then what was done to it — so route:http_request_duration_seconds:p99 tells a reader exactly what it contains without opening the rule file. Keep rules that feed each other in the same group, because groups evaluate sequentially and a rule can safely read a series recorded earlier in its own group but not one recorded in a group that may run later.
Recording rules reduce query cost and the cardinality of the derived series your dashboards touch, but they do not change the cardinality of the raw scraped series. To cut the raw series count you must fix the instrumentation or relabel at scrape time.
Production Code Examples
This end-to-end Flask-style handler uses the right type for each measurement, keeps every label bounded, and exposes the metrics on a dedicated port. For the framework integration details, see instrumenting Flask with Prometheus metrics and the wider walkthrough in exposing custom metrics with the Prometheus client.
import time
from typing import Final
from prometheus_client import Counter, Gauge, Histogram, start_http_server
# 1. Instruments declared once at import; labels are bounded enumerations.
REQUESTS = Counter(
"http_requests_total", "Total HTTP requests",
labelnames=("method", "route", "status"),
)
IN_FLIGHT = Gauge(
"http_requests_in_flight", "In-flight requests",
labelnames=("route",),
)
LATENCY = Histogram(
"http_request_duration_seconds", "Request latency",
labelnames=("route",),
buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
# 2. The closed label domain: anything else collapses to one series.
ROUTES: Final[frozenset[str]] = frozenset({"/checkout", "/search", "/health"})
def bounded_route(template: str | None) -> str:
return template if template in ROUTES else "other"
def handle(method: str, template: str | None) -> int:
# 3. route is the matched template, never the raw path -> bounded label
route = bounded_route(template)
IN_FLIGHT.labels(route=route).inc()
start = time.perf_counter()
status = 500
try:
time.sleep(0.03) # stand-in for real work
status = 200
return status
finally:
# 4. Record in finally so failures are counted, not silently dropped.
LATENCY.labels(route=route).observe(time.perf_counter() - start)
REQUESTS.labels(method=method, route=route, status=str(status)).inc()
IN_FLIGHT.labels(route=route).dec()
if __name__ == "__main__":
start_http_server(8000) # exposition endpoint on :8000/metrics
handle("GET", "/checkout")
handle("GET", "/wp-admin.php") # unmatched -> route="other"
Expected Output: scraping http://localhost:8000/metrics returns the exposition text. Note that the scanner request produced no new label value — it landed in the shared other series.
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/checkout",status="200"} 1.0
http_requests_total{method="GET",route="other",status="200"} 1.0
# HELP http_request_duration_seconds Request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/checkout",le="0.025"} 0.0
http_request_duration_seconds_bucket{route="/checkout",le="0.05"} 1.0
http_request_duration_seconds_bucket{route="/checkout",le="+Inf"} 1.0
http_request_duration_seconds_sum{route="/checkout"} 0.0306
http_request_duration_seconds_count{route="/checkout"} 1.0
http_requests_in_flight{route="/checkout"} 0.0
The second example is the guard rail: a unit test that asserts the label domain stays closed. Cardinality regressions arrive as ordinary pull requests, so the cheapest place to catch one is CI, before the series ever reach the server.
from prometheus_client import REGISTRY
def test_route_label_stays_bounded() -> None:
"""Fail the build if a new route label value escapes the closed set."""
handle("GET", "/users/8123") # a raw path with an embedded ID
observed = {
sample.labels["route"]
for metric in REGISTRY.collect()
if metric.name == "http_requests"
for sample in metric.samples
}
assert observed <= ROUTES | {"other"}, f"unbounded label leaked: {observed}"
Expected Output: the test passes because bounded_route collapsed the raw path, and it would fail loudly the moment someone labels with request.path directly.
test_metrics.py::test_route_label_stays_bounded PASSED
Common Mistakes
- Error signature: Prometheus memory climbs continuously and
prometheus_tsdb_head_seriesgrows without bound. Root cause: a label such asuser_idorrequest_idmints a new series per request. Remediation: remove the label from the instrument and move the identifier into traces or logs; if it is already deployed, drop it at scrape time withmetric_relabel_configswhile the fix ships. - Error signature: the dashboard p99 disagrees with measured latency at the SLO threshold. Root cause: the default buckets are in use and no boundary sits on the threshold, so the interpolated quantile is coarse. Remediation: override
bucketsto place a boundary on the SLO value and tighten spacing on either side of it. - Error signature: a fleet-wide p99 panel returns meaningless or per-instance values. Root cause: a
Summarywas chosen and its quantiles are computed inside each process, so they cannot be merged. Remediation: switch to aHistogramand compute the quantile withhistogram_quantile()over summed bucket rates. - Error signature: graphs show jagged drops to zero on every deploy. Root cause: a counter's raw value is being read instead of its rate, so restarts look like data loss. Remediation: always query counters through
rate()orincrease(), which detect and correct for resets. - Error signature: counters appear to halve or jump randomly between scrapes, and totals are far below reality. Root cause: gunicorn workers each hold an independent registry and the scrape hits one worker at a time. Remediation: enable multiprocess mode with
PROMETHEUS_MULTIPROC_DIRand aMultiProcessCollector, declare amultiprocess_modeon every gauge, and clear the directory on boot. - Error signature: cardinality looks fine in a spot check but the server runs out of memory a week later. Root cause: a rotating label such as a pod name or build hash churns its whole value set on every deploy, so series accumulate across the retention window. Remediation: drop the rotating label with relabeling, or replace it with a stable one such as service name and version.
When to Reach for Each Type
A short field guide ties the semantics back to everyday decisions. Reach for a Counter whenever you would naturally say "number of," because the rate of a counter is the throughput or error rate you actually want on a dashboard. Reach for a Gauge whenever a single instantaneous reading is the answer and the value can fall as well as rise; if you find yourself resetting a counter to model a falling value, you wanted a gauge. Reach for a Histogram whenever the question is "how is this value distributed across requests," especially for latency and payload size, and whenever the answer must hold across more than one replica. Reach for a Summary only in the narrow case of a precise quantile from a single, long-lived process where aggregation will never apply, accepting its higher CPU cost in exchange for a tight per-quantile error bound.
The cardinality lens overrides all of the above when they conflict. A type that is semantically perfect but attached to an unbounded label is the wrong choice, because the series explosion will cost more than the missing signal. When that tension appears, keep the high-cardinality dimension out of metrics entirely and recover it from traces or structured logs, then choose the metric type for the bounded view that remains. This is why the type decision and the cardinality decision in this guide are two halves of one design step, not separate concerns.
Related Reading
- Python Metrics and Instrumentation — the parent guide covering the whole metrics pipeline for Python services.
- Choosing Between Counter, Gauge, Histogram, and Summary — the decision table in full, with one worked example per type.
- Controlling Label Cardinality in Prometheus — detecting offenders and trimming them with
metric_relabel_configs. - Prometheus Client Instrumentation in Python — registries, exposition, and multiprocess wiring in depth.
- The OpenTelemetry Metrics SDK in Python — the same instrument shapes under views, temporality, and OTLP export.
- OpenTelemetry vs Prometheus for Python Metrics — choosing the pipeline these instruments feed.
- Choosing histogram buckets for latency SLOs — a boundary on the objective, a ladder that matches the distribution, and the series budget it costs.
Frequently Asked Questions
How many time series does one labeled metric actually create?
One time series exists for every unique combination of metric name and label values. A metric with two labels of 50 and 20 distinct values produces up to 1000 series, and adding a third label multiplies that figure again. For a histogram, multiply once more by the number of buckets plus two.
Should I use a Histogram or a Summary for latency SLOs?
Use a Histogram. Histogram buckets are aggregatable across instances on the Prometheus server, so you can compute global quantiles and error-budget burn rates, while Summary quantiles are pre-computed per process and cannot be merged.
Why are user IDs and request IDs bad as metric labels?
They are unbounded high-cardinality values. Each new ID creates a fresh time series that occupies index memory long after it stops receiving samples, which inflates server memory, slows queries, and can crash Prometheus. Keep that detail in traces and structured logs instead.
What is the difference between a Counter and a Gauge?
A Counter only ever increases and resets to zero on restart, so it answers how many events have happened over time and must be read through rate() or increase(). A Gauge can go up and down and represents a current value such as queue depth or memory usage.
Do recording rules reduce cardinality?
Recording rules precompute expensive expressions and can aggregate away labels, lowering query cost and the cardinality of the derived series your dashboards touch. They do not reduce the cardinality of the raw series being scraped — only fixing the instrumentation or relabeling at scrape time does that.