Telemetry Cost and Data Volume Control

Telemetry bills grow quietly, because every individual addition is small and nothing in the pipeline objects. This guide covers how to work out what a Python fleet is actually producing, which of the three cost multiplications dominates, and the controls that reduce each one — along with what each reduction costs in questions you can no longer answer. It is part of the Python telemetry pipelines and delivery section. The focused articles in this topic are Dropping and Aggregating Metrics in the Collector, Estimating Telemetry Volume from a Python Service, Log Retention and Tiering Strategy and Tail Sampling in the OpenTelemetry Collector.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "prometheus-client>=0.20.0,<1.0.0"

Concept and architecture

Telemetry cost has one property that makes it different from most infrastructure cost: it is produced by the same engineers who benefit from it, at no visible price at the point of use. A developer adding a span, a log line or a label is making a spending decision without any feedback, and the aggregate of thousands of such decisions is a bill nobody intended.

The consequence is that volume control is mostly not a technical problem. The technical controls are straightforward — sampling, filtering, attribute removal, retention tiering — and all of them are configuration. The difficult part is knowing what to cut, which requires attributing volume to services and, ideally, to the code paths inside them.

There are three places to intervene, and they are complements rather than alternatives.

In the code. Not producing telemetry is the only reduction that saves CPU inside the request path as well as bytes downstream. It is also the slowest to change, because it takes a release per service, and the most durable, because it removes the cost permanently.

In the SDK. Sampling and level configuration, driven by environment variables rather than code. This changes with a restart rather than a release, applies per service, and is the only place a head sampling decision can be made — which matters because a span not sampled at the head is never created at all.

In the collector. Filtering, tail sampling, attribute removal, aggregation. This changes in minutes, fleet-wide, with no application involvement. It cannot reduce what the application spends producing the data, but it decides everything about what is stored and therefore about what is billed.

Step-by-step implementation

Step 1 — Measure bytes per request before changing anything. Capture a real export payload, divide by the requests it represents, and you have a number that stays meaningful as traffic changes. Totals do not: a fleet whose bill doubled because traffic doubled has no volume problem, and one whose per-request cost doubled does.

# measure.py — bytes on the wire per request, from a real payload
import gzip, json, pathlib

payload = pathlib.Path("captured-export.json").read_bytes()
doc = json.loads(payload)

spans = sum(len(ss["spans"])
            for rs in doc["resourceSpans"]
            for ss in rs["scopeSpans"])
requests = len({s["traceId"] for rs in doc["resourceSpans"]
                for ss in rs["scopeSpans"] for s in ss["spans"]})

print(f"raw        {len(payload)/1024:8.1f} KiB")
print(f"compressed {len(gzip.compress(payload))/1024:8.1f} KiB")
print(f"spans      {spans}  ·  traces {requests}")
print(f"per request {len(payload)/requests:8.0f} bytes raw, "
      f"{spans/requests:.1f} spans")

Expected Output:

raw           318.7 KiB
compressed     41.2 KiB
spans      512  ·  traces 47
per request     6944 bytes raw, 10.9 spans

Nearly seven kilobytes and eleven spans per request is a service that has been instrumented enthusiastically. The compression ratio — about eight to one — is worth noting too, since it means the billed figure may be far below the raw one depending on how the backend charges.

Step 2 — Attribute the volume to services and to span names. A fleet-wide total is not actionable. A ranking of services by bytes per second, and within the top service a ranking of span names by count, usually reveals that a large fraction of the volume comes from a handful of operations nobody queries — a health check, a cache lookup, an ORM's internal statements.

# bytes per second, by service
sum by (service_name) (rate(otelcol_exporter_sent_spans[5m])) * 1100

# the span names doing the most volume in one service
topk(10, sum by (span_name) (rate(spans_received_total{service="checkout"}[5m])))

Step 3 — Cut the largest term, not every term. A ten percent reduction applied everywhere costs ten percent of every future investigation. A ninety percent reduction applied to the one operation producing forty percent of the volume costs almost nothing, because nobody was querying it. Filtering by span name in the collector is the sharpest tool available and needs no application change.

processors:
  filter/drop_noise:
    error_mode: ignore
    traces:
      span:
        # Health checks: high volume, zero diagnostic value.
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
        # Cache hits under a millisecond: the aggregate is in a metric already.
        - 'name == "redis.get" and (end_time_unix_nano - start_time_unix_nano) < 1000000'

Step 4 — Sample what remains, with exemptions. Once the noise is gone, a probabilistic sample reduces the rest linearly. The exemptions are what keep it useful: errors and slow requests are the ones anybody will look for, and they are a tiny fraction of the volume, so keeping all of them costs almost nothing.

Step 5 — Reduce the metrics bill differently. Metrics do not respond to sampling, because their cost is series count. The equivalent reduction is dropping a label, which divides the series count by that label's cardinality — a far more dramatic effect than any sampling rate. Controlling label cardinality in Prometheus covers the identification; the collector-side removal is one processor.

processors:
  metricstransform/trim:
    transforms:
      - include: http_server_duration
        action: update
        operations:
          - action: aggregate_labels
            label_set: [http_route, http_method, http_status_code]
            aggregation_type: sum        # everything else summed away

Step 6 — Set retention by age and by signal. The last multiplication is time. Most queries are against the last few hours; almost none are against last quarter. Tiering — hot for a week, cheaper storage for a month, archive beyond — costs a policy and typically removes more from a bill than any sampling decision.

Configuration reference

Async and concurrency considerations

Reducing volume in the collector does nothing for the cost inside the Python process, and that cost is real at high span rates. Creating a span allocates, records timestamps, and enqueues; serialising it to protobuf holds the interpreter lock. At a few hundred spans per second none of this is measurable; at tens of thousands it is a visible fraction of a core.

This is the argument for head sampling existing at all. A span not sampled at the root is never created, so none of that work happens, and the reduction shows up as CPU rather than only as bytes. Where a service is CPU-bound and heavily instrumented, moving the sampling decision from the collector to the SDK is a performance change rather than a cost change.

Log records have the same property in a sharper form, because formatting happens on the calling thread. A debug statement that formats a dictionary is doing that work whether or not the record is ultimately stored, unless the level check short-circuits it — which is the entire subject of lazy formatting and expensive log arguments. Raising the level in production is therefore both the cheapest volume control and a genuine latency improvement on hot paths.

Metrics are the exception. Recording an observation is a lock acquisition and an addition, independent of how many series exist, so cardinality costs storage and query time rather than application CPU. A service with a cardinality problem will not feel it; its monitoring system will.

Production code examples

A sampler that keeps everything from a small set of interesting conditions and a fraction of the rest, implemented at the SDK so the savings include CPU:

# sampling.py
from opentelemetry.sdk.trace.sampling import (
    ParentBased, Sampler, SamplingResult, Decision, TraceIdRatioBased,
)
from opentelemetry.trace import SpanKind
from opentelemetry.util.types import Attributes


class RouteAwareSampler(Sampler):
    """Drop health checks entirely; sample the rest by ratio."""

    def __init__(self, ratio: float = 0.05):
        self._ratio = TraceIdRatioBased(ratio)
        self._skip = {"/healthz", "/readyz", "/metrics"}

    def should_sample(self, parent_context, trace_id, name, kind=None,
                      attributes: Attributes = None, links=None, trace_state=None):
        route = (attributes or {}).get("http.route")
        if kind is SpanKind.SERVER and route in self._skip:
            # 1. Never created, so no CPU and no bytes.
            return SamplingResult(Decision.DROP, attributes, trace_state)
        return self._ratio.should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state)

    def get_description(self) -> str:
        return f"RouteAwareSampler({self._ratio.get_description()})"


SAMPLER = ParentBased(root=RouteAwareSampler(ratio=0.05))

Expected Output: the span rate falls immediately, and the CPU profile of the process changes with it.

before   spans/s 4180   cpu 1.42 cores
after    spans/s  318   cpu 1.19 cores

A budget report that turns the measurement into something a team can act on:

# budget.py — per-service telemetry cost, expressed per request
SERVICES = {
    "checkout":  {"rps": 420, "span_bytes": 6944, "log_bytes": 1180, "series": 2400},
    "inventory": {"rps": 180, "span_bytes": 2100, "log_bytes":  310, "series":  900},
}
PRICE_PER_GB = 0.55
PRICE_PER_SERIES_MONTH = 0.0009

for name, s in SERVICES.items():
    gb_month = (s["rps"] * (s["span_bytes"] + s["log_bytes"]) * 2_592_000) / 1e9
    cost = gb_month * PRICE_PER_GB + s["series"] * PRICE_PER_SERIES_MONTH
    print(f"{name:12s} {gb_month:9.1f} GB/month  ${cost:9.2f}  "
          f"${cost / (s['rps'] * 2_592_000) * 1000:.4f} per 1k requests")

Expected Output:

checkout       8862.1 GB/month  $ 4876.32  $0.0045 per 1k requests
inventory      1120.9 GB/month  $  617.30  $0.0013 per 1k requests

Cost per thousand requests is the figure worth putting in front of a team, because it is comparable across services of very different sizes and it moves when they change something.

What each cut costs in answers

Every reduction removes the ability to answer something, and the discipline worth adopting is to name that something before applying the cut.

Dropping health check spans costs the ability to see, in a trace, that a probe was slow. Nobody has ever needed this; the probe's own metrics cover it. This cut is free.

Dropping fast cache spans costs the ability to count cache operations per request from traces. If that count exists as a metric — and it should — the cut is nearly free. If it does not, add the metric first, then make the cut.

Probabilistic sampling at five percent costs the ability to find a specific customer's request from an hour ago. This is a real loss and it is felt during support escalations. Exempting errors and slow requests recovers most of the value; adding a way to force sampling for a specific tenant, via baggage, recovers most of the rest.

Raising the log level to INFO costs the ability to reconstruct a code path from debug output after the fact. The mitigation is a mechanism to lower a single logger's level at runtime, covered in changing log levels at runtime, which converts a permanent loss into a temporary one.

Removing a metric label costs the ability to break that metric down by the removed dimension. This is the cut most likely to be regretted, because breakdowns are discovered to be necessary at the worst moment. Removing a label from a copy while keeping the full series for a shorter retention is a middle path worth considering.

Shortening retention costs the ability to compare against last quarter. Most teams overestimate how often they do this and underestimate how annoying it is when they cannot, which argues for keeping a heavily aggregated long-term copy rather than nothing.

Making cost visible to the people who create it

Technical controls reduce volume once. Keeping it reduced requires that the engineers adding telemetry can see what they are spending, and that is an organisational mechanism rather than a processor configuration.

Three arrangements work, in increasing order of effort and effectiveness.

A per-service cost figure on a dashboard. The budget script above, run daily, with cost per thousand requests per service. This costs an afternoon to build and changes behaviour more than any policy document, because it turns an invisible shared resource into a number with somebody's name on it. The figure must be normalised per request; a raw total simply tells the largest service that it is large.

A budget, with a conversation when it is exceeded. A service that agrees a per-request telemetry budget and exceeds it gets a review rather than a block. The value is not enforcement — nobody wants telemetry disabled by a quota during an incident — but the conversation, which reliably surfaces a debug statement left on a hot path or a span added inside a loop.

Cost attribution in the bill. Where the platform charges teams for infrastructure, including telemetry in that charge closes the loop completely. This is the arrangement that makes the sidecar topology worth considering for accounting reasons, since a sidecar's resource usage is charged to the pod rather than absorbed as platform overhead.

What does not work is a review gate on adding instrumentation. It slows down the thing you want people to do, catches the small additions rather than the large ones, and produces services that are under-instrumented in exactly the places where an incident will later need detail. The goal is not less telemetry; it is telemetry whose cost is proportional to its value, and that requires feedback rather than friction.

When to spend more rather than less

It is worth stating the other direction, because a page about cost control can read as though less is always better.

Three situations justify increasing telemetry spend deliberately. A new service, where nobody yet knows which operations matter, benefits from instrumenting generously and cutting after a month of real traffic has shown what is queried — the reverse order is guesswork. A service under active incident investigation should have its sampling raised, temporarily and deliberately, because during that window the data is worth far more than it costs; having a documented way to do this in the collector, applied per service, converts an argument into a configuration change. And a system undergoing migration benefits from the overlap period described in routing telemetry to multiple backends, where paying twice for a few weeks is cheap relative to discovering a discrepancy after cutover.

The common thread is that telemetry spend should track uncertainty. A well-understood service running the same code for a year needs less; a service being changed, scaled or debugged needs more. A fixed sample rate applied uniformly and forever is the arrangement that serves neither case, and the ability to change it quickly — which is the argument for collector-side controls throughout this guide — is what lets the spend follow the need.

Common mistakes

Cutting uniformly. Error signature: a ten percent saving and universal complaints. Root cause: a flat reduction applied without measuring where the volume is. Remediation: rank by operation and cut the top of the list.

Sampling to fix a metrics bill. Error signature: sampling applied, metrics cost unchanged. Root cause: metric cost is series count, which sampling does not touch. Remediation: find and remove the high-cardinality label.

Cutting in the SDK when the goal is flexibility. Error signature: a volume emergency that takes a week of deploys to address. Root cause: the only controls are in application code. Remediation: keep a collector-side filter available so the fleet can be turned down in minutes.

Dropping data that a metric does not cover. Error signature: a dashboard that breaks after a volume reduction. Root cause: an aggregate that was being computed from stored spans. Remediation: create the metric first, verify it matches, then drop the spans.

Measuring totals rather than per-request cost. Error signature: a cost review that concludes the problem is traffic growth. Root cause: no normalised figure. Remediation: track bytes per request per service, which separates growth from waste.

Forgetting compression. Error signature: a volume estimate several times the invoice, or vice versa. Root cause: comparing raw bytes against a backend that bills compressed, or the reverse. Remediation: measure both, and check which one the contract uses.

Frequently Asked Questions