Controlling Label Cardinality in Prometheus

A single label carrying an unbounded value — a user ID, a request ID, or a raw URL path with an identifier baked into it — can turn one metric into hundreds of thousands of permanent time series and exhaust a Prometheus server's memory. This walkthrough is for backend engineers and SREs who already have a Python service scraped by Prometheus and need to identify the blowup, bound the label values at the source, and collapse what is already stored. It sits under metric types and cardinality control and is part of the broader Python metrics and instrumentation reference; the first half of the same design decision — picking the instrument itself — is covered in choosing between counter, gauge, histogram, and summary.

Prerequisites

# Pinned so the labels() API and exposition format stay stable across deploys.
pip install "prometheus-client>=0.20.0,<1.0.0"

A Prometheus server (>=2.50,<3.0) with edit access to its scrape configuration is required for the relabeling steps, and the reload endpoint must be enabled with --web.enable-lifecycle so the configuration can be applied without a restart. The diagnostic queries run in the Prometheus expression browser at http://localhost:9090. No application environment variables are needed beyond the port your exposition endpoint already listens on — the pattern set up in exposing custom metrics with prometheus_client.

Why cardinality is the cost driver

Prometheus stores one independent append-only stream — a time series — for every unique combination of metric name and label values. Each active series occupies an entry in the in-memory head index plus per-sample storage, so the resource a series consumes is roughly constant regardless of how often it changes. The consequence is blunt: a metric scraped once every fifteen seconds with a million label combinations costs vastly more than a metric scraped every second with ten combinations. Sample frequency is cheap; series count is expensive.

The number of series a metric emits is the product of the distinct value counts of all its labels. Two bounded labels of five and six values produce at most thirty series. Introduce one unbounded label — a user ID, a request ID, a raw path with embedded identifiers, an email address, or a full exception message — and the product becomes the size of that unbounded set, which grows with traffic and never shrinks. Series are not reclaimed the moment a label value stops appearing; they age out of the head block only after the retention window, so a brief spike of unique IDs leaves a lingering trail of dead series. That is why an unbounded label is a one-way ratchet on memory, and why the durable fix always lives in the application rather than in the query layer.

Implementation

Step 1 — Confirm a blowup exists and rank by metric. In the Prometheus expression browser, rank metric names by series count. A handful of names usually account for the bulk of the head series.

topk(10, count by (__name__)({__name__=~".+"}))

Expected Output: one row per metric name, ordered by series count, with the offender an order of magnitude above everything else.

{__name__="http_requests_total"}          142031
{__name__="http_request_duration_seconds"} 8420
{__name__="python_gc_objects_collected_total"} 3

Step 2 — Find the offending label on that metric. For the worst metric, count distinct values per label to identify which dimension is unbounded.

count(count by (path) (http_requests_total))

A result in the tens of thousands for a single label means that label is unbounded. Compare it against prometheus_tsdb_head_series to gauge the share of total series the metric consumes. Two complementary signals confirm the diagnosis. The scrape_samples_scraped series reports how many samples a target returns per scrape, so a target returning hundreds of thousands of samples is emitting a high-cardinality metric at the source. The Prometheus TSDB status page (Status, then TSDB Status in the UI) lists the highest-cardinality label names and label-value pairs directly, which often pinpoints the offender faster than ad hoc queries.

When several labels share blame, count them together to see the multiplicative effect rather than inspecting each in isolation.

count(count by (method, route, status, user_id) (http_requests_total))

If removing user_id from that grouping drops the count by three orders of magnitude, you have isolated the unbounded dimension and can target it precisely in the steps below.

Step 3 — Bound the label in application code. The durable fix is to never emit the unbounded value. Map raw paths to the matched route template and free-form strings to a closed enum before they reach a label. The same rule applies to whichever instrument type you chose, whether that is a counter or a histogram; see the type walkthrough for the semantics behind the choice.

from prometheus_client import Counter

REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests",
    labelnames=("method", "route", "status"),
)

# Closed enum: anything the framework cannot classify collapses to "other".
KNOWN_METHODS = frozenset({"GET", "POST", "PUT", "DELETE", "PATCH"})


def record(method: str, route_template: str, status: int) -> None:
    REQUESTS.labels(
        # Step 1: constrain each label to a value set you control.
        method=method if method in KNOWN_METHODS else "other",
        route=route_template,   # "/users/{id}", never "/users/8123"
        # Step 2: bucket the status into a class if you do not need exact codes.
        status=str(status),
    ).inc()

Expected Output: the exposition endpoint shows one series per bounded combination rather than one per request.

# TYPE http_requests_total counter
http_requests_total{method="GET",route="/users/{id}",status="200"} 41837.0
http_requests_total{method="POST",route="/users",status="201"} 912.0

Bounding in code has a second benefit beyond series count: the exposition payload itself shrinks, so each scrape transfers and parses fewer bytes, which lowers both client memory in the Python process and scrape duration on the server. A process that builds a fresh labels() child for every unique identifier also leaks memory inside the client, because the client retains every child it has ever created for the lifetime of the process. Bounding the label set therefore caps client-side memory as well as server-side series count. Web frameworks make the route template easy to obtain — the pattern used in instrumenting Flask with Prometheus metrics reads the matched rule rather than the request path.

Step 4 — Drop a label at scrape time when you cannot redeploy. If the bad label is already in production, remove the dimension server-side. metric_relabel_configs runs after the scrape and before storage, so it shrinks series before they are indexed. This is a stopgap that buys time for a code fix, not a substitute for it: the application keeps generating the wide payload on every scrape, so client memory and network cost are unchanged even though stored series shrink.

scrape_configs:
  - job_name: "python-app"
    static_configs:
      - targets: ["app:8000"]
    metric_relabel_configs:
      # Collapse the unbounded "path" dimension entirely.
      - regex: "path"
        action: labeldrop

Step 5 — Rewrite a label to a bounded form instead of dropping it. When the dimension is useful but the raw value is too granular, replace it with a regex-extracted bounded value rather than discarding it.

    metric_relabel_configs:
      # Rewrite /users/123 -> /users/{id} into a new "route" label.
      - source_labels: [path]
        regex: "(/users/)[0-9]+"
        target_label: route
        replacement: "${1}{id}"
      # This must run AFTER the replace, or the source is already gone.
      - regex: "path"
        action: labeldrop

Step 6 — Allowlist known-good values when a pattern is hard to express. Some labels carry a small set of legitimate values mixed with occasional garbage from malformed requests. Rather than enumerate every bad value, keep only the good ones. A keep action on the label drops samples whose value is not in the allowed set.

    metric_relabel_configs:
      # Keep only the known HTTP methods; anything else is discarded.
      - source_labels: [method]
        regex: "GET|POST|PUT|DELETE|PATCH"
        action: keep

Step 7 — Drop an entire noisy metric. When a whole metric is not worth its cardinality, drop it with a __name__ match so it is never stored.

    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "expensive_debug_metric"
        action: drop

Configuration options

These actions run under metric_relabel_configs, which is applied post-scrape and affects stored data. Distinguish them from relabel_configs, which run during target discovery and shape which targets are scraped, not which samples are kept. Order matters: rules execute top to bottom, and each operates on the label set produced by the preceding rule. A replace that derives a bounded label must therefore appear before the labeldrop that removes the source it reads from, or the source will already be gone. When a regex in a replace does not match, the rule is a no-op and the target label is left untouched, so an allowlisting keep rule is often safer than a brittle replace for values that do not follow a single pattern.

For large fleets, enforce limits rather than relying solely on relabeling. The scrape-level sample_limit rejects an entire scrape that returns more than the configured number of samples, converting a silent cardinality leak into an alertable target failure before it floods the TSDB. Pair it with the ranking query from Step 1 so a tripped limit points directly at the responsible job.

Verification

After reloading the Prometheus configuration, re-run the per-label count. The unbounded dimension should be gone or collapsed to a small bounded set.

# Requires --web.enable-lifecycle on the Prometheus server.
curl -X POST http://localhost:9090/-/reload

count(count by (path) (http_requests_total))

Expected Output: the path label no longer exists after labeldrop, and the series collapse into the bounded route set.

# before relabeling
count by (route) (http_requests_total)  -> {route="/users/{id}"}  142031 series

# after labeldrop / replace + reload
count by (route) (http_requests_total)  -> {route="/users/{id}"}  1 series

Watch prometheus_tsdb_head_series flatten within a couple of scrape intervals once the offending series stop being created. The flattening is not instant: relabeling stops new series from being added immediately, but series already in the head block persist until they fall outside the retention window and the block is compacted. Expect total head series to plateau within one scrape interval and to decline gradually over the following hours as stale series expire. If memory does not plateau, a second metric or a second label is still unbounded, so repeat the ranking query from Step 1 to find it.

A good closing check is to confirm the relabeling did not silently break a dashboard. Run the queries your panels use against the rewritten label and verify they still return the expected shape. A replace that introduced a new route label, for example, means panels grouping by the old path label now return nothing, so those panels must be updated to group by route as part of the same change.

Common mistakes

• Error signature: head series do not drop after adding a replace rule. Root cause: the original high-cardinality label is still stored alongside the new bounded one, so every raw value continues to define its own series. Remediation: add a labeldrop for the source label in a rule that runs after the replace, and confirm with count(count by (path) (...)) that the source dimension is gone.

• Error signature: cardinality returns whenever the scrape configuration is reset, or when a second Prometheus server scrapes the same target. Root cause: the application still emits the unbounded value, so relabeling is a patch applied per server rather than a fix. Remediation: bound the label in code as the primary fix and keep the relabel rules as defence in depth for anything you do not control.

• Error signature: after a labeldrop, a counter shows impossible decreases or doubled rates. Root cause: collapsing a label merges several monotonic series into one, and if those series reset independently the merged stream is no longer monotonic. Remediation: drop labels on counters only when the merged series remains sensible, and prefer aggregating with sum by (...) at query time when correctness across resets matters.

Where the high-cardinality data belongs instead

Bounding a label does not mean losing the detail — it means routing it to a signal built for it. Per-request identifiers belong in traces and in structured logging with the Python standard library, where each record is an independent document rather than a permanent index entry, and where sampling strategies for distributed tracing keep the volume affordable. The bridge between the two is the exemplar: a histogram observation can carry a trace ID that a dashboard turns into a link, so a slow bucket still leads to one specific request without that request ever becoming a series.

from prometheus_client import Histogram

LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency",
    labelnames=("route",),                # bounded label set only
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)

# The identifier rides along as an exemplar, not as a label.
def record_latency(route_template: str, seconds: float, trace_id: str) -> None:
    LATENCY.labels(route=route_template).observe(
        seconds, exemplar={"trace_id": trace_id},
    )

Expected Output: the OpenMetrics exposition attaches the identifier to a bucket line instead of creating a new series.

http_request_duration_seconds_bucket{route="/users/{id}",le="0.5"} 8412.0 # {trace_id="4bf92f3577b34da6"} 0.31 1750000000

If you instrument with the OpenTelemetry metrics SDK rather than the Prometheus client, the equivalent control is a view that restricts an instrument to an explicit set of attribute keys, discarding the rest before aggregation — the SDK-side counterpart of labelkeep. The trade-offs between the two pipelines are compared in OpenTelemetry vs Prometheus for Python metrics; the cardinality discipline is identical in both, because both ultimately store one stream per unique attribute combination.

Related

• Metric types and cardinality control — the parent reference covering instrument choice and label budgets together.
• Choosing between counter, gauge, histogram, and summary — the first half of the design decision: settling the metric type before you bound its labels.
• Exposing custom metrics with prometheus_client — registry, exposition endpoint, and multiprocess mode for the bounded metrics defined here.
• Instrumenting Flask with Prometheus metrics — reading the matched route template so path labels are bounded from the first request.
• Recording counters and histograms with OpenTelemetry — the same attribute discipline expressed as SDK views.

Frequently Asked Questions