Dropping and Aggregating Metrics in the Collector

A metrics bill is series count multiplied by sample interval, and series count is decided by labels. That makes collector-side trimming unusually effective: removing one label divides the series count by that label's cardinality, which is a factor of fifty rather than a percentage. This page covers finding the series that cost the most, removing them without losing the metric, and verifying that nothing anybody queries breaks. It is a task article under telemetry cost and data volume control, part of the Python telemetry pipelines and delivery section, and it pairs with controlling label cardinality in Prometheus.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Implementation

Step 1 — Rank metrics by series count and check which are queried. Two lists, intersected. The first is the metrics store's own count of series per metric name, which every store exposes. The second is the set of metric names appearing in dashboards and alert rules, which can be extracted from their definitions. A metric high on the first list and absent from the second is pure cost, and in most fleets there are several.

# the series count per metric, most expensive first
topk(20, count by (__name__) ({__name__=~".+"}))

Step 2 — Drop whole metrics nothing references. This is the cheapest possible reduction: no capability is lost because no capability was being used. Instrumentation libraries are the usual source, since they emit a broad default set and most fleets use a handful of them.

processors:
  filter/drop_unused:
    error_mode: ignore
    metrics:
      metric:
        - 'name == "http_server_active_requests"'
        - 'name == "process_runtime_cpython_cpu_time"'
        - 'IsMatch(name, "^rpc_client_.*")'

Step 3 — Aggregate away a label rather than dropping the metric. Where the metric is used but one of its labels is not, summing over that label keeps every total exactly correct while dividing the series count. This is almost always better than dropping the metric, and it is the single highest-leverage action available.

processors:
  metricstransform/trim_pod:
    transforms:
      - include: http_server_duration
        action: update
        operations:
          - action: aggregate_labels
            label_set: [http_route, http_request_method, http_response_status_code]
            aggregation_type: sum

Step 4 — Rewrite unbounded values into bounded ones. Sometimes the label is genuinely wanted and its values are the problem: a raw URL path, a customer identifier, an error message. Rewriting the value — a path to its route template, an identifier to a tier, a message to an error class — keeps the dimension with a value set small enough to be affordable.

processors:
  transform/bound_labels:
    metric_statements:
      - context: datapoint
        statements:
          # /orders/8812 -> /orders/{id}
          - replace_pattern(attributes["http.route"], "/[0-9]+", "/{id}")
          # keep a tier, not a customer
          - set(attributes["tenant_tier"], "enterprise") where attributes["tenant"] != nil
          - delete_key(attributes, "tenant")

Step 5 — Add a safety limit for runaway series. Trimming known problems does not prevent new ones. A limit that starts discarding new series past a threshold turns a cardinality explosion from an outage of the metrics store into a bounded loss of one metric, which is a much better failure. It should alert loudly when it engages, because it is a symptom rather than a solution.

Step 6 — Run both pipelines until the queries are verified. A trimmed metric that breaks an alert is discovered at the worst moment. Exporting both the trimmed and untrimmed versions to different destinations for a week, and checking every dashboard and alert against the trimmed one, costs a week of double storage and avoids the alternative.

Expected Output: the series count falling sharply with no change in any rate or quantile query.

before   active series 284_902   ingest 41.2 MB/min
after    active series   9_841   ingest  2.1 MB/min
query    sum(rate(http_server_duration_count[5m]))  unchanged: 418.2

Why this belongs in the collector

There is a reasonable objection to all of this: the correct fix is to stop the application emitting the label in the first place, and everything here is a workaround. Both halves of that are true, and the workaround is still worth having for three reasons.

Speed. A cardinality explosion is an operational event with a live impact on the metrics store, and it needs a response in minutes. A collector configuration change is minutes; a release across every affected service is days. Having the mechanism ready before it is needed is the difference between a contained incident and an extended one.

Reversibility. Removing an instrument from application code is a decision that is expensive to revisit — the data for the intervening period simply does not exist. A collector-side aggregation can be removed and the full series resume immediately, because the application never stopped producing them. That makes it a much safer thing to try.

Fleet-wide consistency. Forty services that each emit a slightly different set of labels can be normalised in one place. Achieving the same thing in application code means forty changes, agreed and shipped in step, which in practice means it does not happen.

The honest caveat is that collector-side trimming hides the problem from the people creating it. A service whose cardinality is being cleaned up downstream has no feedback that it is doing anything wrong, and will do it again. Pairing the trimming with a report — these metrics are being aggregated for you, and here is what it is costing — keeps the fix visible, and is usually what eventually produces the proper repair in the instrumentation.

Finding the label before it finds you

The techniques above assume the problem has already been identified. Identifying it quickly is a skill worth having, because the first symptom is usually a metrics store under memory pressure rather than a helpfully named error.

The fastest route is to rank metric names by series count, take the top entry, and print the label set of a single one of its series. In the overwhelming majority of cases the offending label is visible at a glance: a pod name, a full URL, an identifier, a message string, a version tag that changes on every build. There is rarely any subtlety to it — the label that is wrong looks wrong.

The second check is whether the series count is growing or stable. A stable high count is an instrumentation decision that can be trimmed at leisure. A count that grows monotonically is an active problem, because the store never forgets a series until retention expires, and something is creating new ones continuously. Growth that tracks request rate means a per-request value became a label; growth that tracks deployments means a build identifier or a pod name did.

The third check is which service is responsible, which matters because the fix eventually belongs there. Grouping the series count by the service label gives it directly, and it is worth doing even when the collector-side trim is applied immediately, because the report back to that team is what stops the next occurrence. A cardinality problem fixed only in the collector will recur with the next metric that team adds.

Configuration options

Verification

Confirm the totals are unchanged, because aggregation preserves them and a mistake does not.

# the same query against the trimmed and untrimmed destinations
for host in metrics-full metrics-trimmed; do
  curl -sG "http://$host:9090/api/v1/query" \
    --data-urlencode 'query=sum(rate(http_server_duration_count[5m]))' \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["result"][0]["value"][1])'
done

Expected Output: identical values, to the precision the sampling interval allows.

418.21
418.21

A difference means the aggregation is summing something it should not, most often a gauge being summed where it should have been averaged — a mistake that is invisible in the series count and obvious in this comparison.

Common mistakes

Summing a gauge across a removed label. Error signature: a gauge whose value is now several times too large. Root cause: sum applied where the label distinguished instances of the same measurement. Remediation: use an appropriate aggregation, or keep the label for gauges where instance identity is the point.

Dropping a metric an alert depends on. Error signature: an alert that silently never fires again. Root cause: the queried-metric list was taken from dashboards only. Remediation: extract metric names from alert rules as well, and run both pipelines until verified.

Trimming without telling anyone. Error signature: an engineer spending an afternoon on a breakdown that no longer exists. Root cause: a collector change invisible from the application side. Remediation: document what is aggregated where, and report it back to the owning teams.

Relying on a series limit as the fix. Error signature: a metric that works for some label values and not others, unpredictably. Root cause: a limit discarding whichever series arrived last. Remediation: treat the limit as a circuit breaker with a loud alert, and fix the cardinality it caught.

Reaching for the scrape interval first. Error signature: a small saving and noticeably worse resolution. Root cause: interval affects samples, not series, and series dominate the cost. Remediation: work the list above in order.

Frequently Asked Questions