Tail Sampling in the OpenTelemetry Collector

Head sampling decides whether to keep a trace before anything has happened in it. Tail sampling decides afterwards, which means it can keep exactly the traces an engineer will want — the failures, the slow ones, the odd ones — and discard the overwhelming majority that are identical to each other. This page covers the policies, the decision window, the memory it costs and the routing it depends on. It is a task article under telemetry cost and data volume control, part of the Python telemetry pipelines and delivery section, and it builds on sampling strategies for distributed tracing.

Prerequisites

The application needs no changes; tail sampling is entirely a collector concern. What it does need is that spans reach the sampling tier with the errors and durations correctly recorded, which depends on recording exceptions and span events being done properly in the application.

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

# the application samples everything at the head, so the tail can decide
export OTEL_TRACES_SAMPLER=always_on

Implementation

Step 1 — Route whole traces to one replica. This comes first because nothing else works without it. A gateway pool behind an ordinary load balancer splits a trace's spans across replicas, each of which then makes an independent decision on a fragment. The result is traces stored with most of their spans missing, which is worse than storing nothing because it looks complete. A load-balancing tier that hashes the trace identifier solves it.

# tier 1: routing only — no sampling here
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { timeout: 5s, tls: { insecure: true } }
    resolver:
      k8s: { service: otel-sampling.observability }

Step 2 — Set the decision window from the trace duration distribution. The collector holds a trace until the window expires, then applies the policies to whatever it has. A window shorter than a slow trace means that trace is judged on its first few spans, which is exactly backwards: the traces the sampler most wants to keep are the ones most likely to be decided prematurely. The window must cover the p99 trace duration plus the time for the last span to arrive.

Step 3 — Write the keep-everything policies first. Errors and slow traces are the reason tail sampling exists. Both are cheap to keep because both are rare, and both are what an investigation begins from.

processors:
  tail_sampling:
    decision_wait: 15s            # longer than the p99 trace duration
    num_traces: 200000            # in-flight traces held
    expected_new_traces_per_sec: 4000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: keep-flagged
        type: boolean_attribute
        boolean_attribute: { key: sampling.force, value: true }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Step 4 — Add an escape hatch for deliberate capture. The keep-flagged policy above reads a boolean attribute, which an application can set when it knows a request is interesting: a support escalation, a canary, a specific tenant under investigation. Combined with baggage for tenant and feature context, it gives on-call a way to guarantee capture without changing the sample rate for everyone.

from opentelemetry import trace

span = trace.get_current_span()
if request.headers.get("x-debug-capture") == INTERNAL_TOKEN:
    # 1. The tail sampler will keep this trace regardless of the ratio.
    span.set_attribute("sampling.force", True)

Step 5 — Budget the memory explicitly. The collector holds every in-flight trace for the duration of the window. The memory is the span rate multiplied by the window multiplied by the average span size, and it is a real number that must fit inside the container limit with room for the memory limiter to act before the platform does.

SPANS_PER_SECOND = 20_000
DECISION_WAIT_S = 15
AVG_SPAN_BYTES = 1_000
held = SPANS_PER_SECOND * DECISION_WAIT_S
print(f"{held:,} spans held · {held * AVG_SPAN_BYTES / 1e9:.2f} GB before overhead")

Expected Output:

300,000 spans held · 0.30 GB before overhead

Step 6 — Verify the policies keep what you think. A policy list is evaluated as an OR: a trace matching any policy is kept. It is easy to write a set that keeps far more than intended — a latency threshold set below the service's median, for instance, keeps half of everything and produces a sampling ratio that saves nothing.

Designing the policy set

A policy list is a specification of what is worth keeping, and writing it well is mostly about resisting the urge to keep too much.

Start with three policies and add reluctantly. Errors, slow traces, and a probabilistic remainder cover the overwhelming majority of real needs. Each additional policy makes the effective sample rate harder to predict, because the policies compose as a union and their overlaps are not obvious. A fleet with eleven policies usually cannot say what fraction of traffic it is storing, which defeats the cost purpose entirely.

Set the latency threshold from the distribution, not from a round number. A threshold of five hundred milliseconds keeps four percent of traffic in one service and sixty percent in another. Setting it per service, at roughly the p95, gives a predictable volume and a consistent meaning: "slower than most". A single global threshold produces neither.

Beware policies keyed on attributes that only some spans carry. A policy matching on an attribute present on the server span but not on child spans still works, because the policy applies to the trace as a whole — but a policy whose attribute is set by only one service in a multi-service trace will behave differently depending on whether that service participated. Keying on resource attributes avoids this, for the same reason routing does.

Remember that policies cannot see what was never sent. Tail sampling operates on spans that reached it, so a head sampler dropping ninety-five percent upstream means the tail sampler is choosing from the remaining five percent and its error policy keeps five percent of errors, not all of them. Head and tail sampling compose multiplicatively, and running both with aggressive settings is the most common way to end up with a trace store that contains nothing useful.

What tail sampling does to aggregates

One consequence is easy to miss and causes real confusion later: a sampled trace store is not a sample of traffic, so any number computed from it is wrong in a specific and predictable direction.

Because errors and slow traces are kept at one hundred percent and everything else at five, the stored population is dramatically over-representative of failure. Counting traces in the store and computing an error rate gives a figure perhaps ten times the real one. Computing a p99 latency from stored traces gives a number well above the true p99, because the slow tail is preserved while the fast body is thinned. Both mistakes are easy to make on a dashboard built from trace data, and both produce a service that looks far worse than it is.

The correct source for any aggregate is metrics, which are computed from every request before any sampling happens. Traces answer "what happened in this request"; metrics answer "how often and how slow", and the division of labour is not negotiable once tail sampling is in place. Where a backend can generate metrics from spans, that generation must happen before the sampling processor in the pipeline, so the counts reflect all traffic rather than the retained subset — a pipeline ordering detail that is trivial to get wrong and produces silently biased dashboards.

Configuration options

Verification

Check that errors really are being kept at one hundred percent, because that is the property the whole arrangement is for.

# traces stored that contain an error, against errors counted in metrics
sum(rate(traces_stored_total{outcome="error"}[10m]))
  /
sum(rate(http_server_requests_total{status=~"5.."}[10m]))

Expected Output: a ratio at or very near one, and a total volume well below the unsampled rate.

error trace capture ratio   0.998
overall sampling ratio      0.072

A capture ratio meaningfully below one means either the decision window is too short for the failing requests — plausible, since failures are often slow — or a head sampler upstream is dropping traces before they reach the tail.

Common mistakes

Tail sampling without trace-aware routing. Error signature: stored traces missing spans from several services. Root cause: spans of one trace decided by different replicas. Remediation: add a routing tier keyed on the trace identifier.

A decision window shorter than the slow traces. Error signature: a latency policy that keeps almost nothing. Root cause: the trace is judged before it finishes. Remediation: set the window above the p99 duration plus arrival delay.

Head and tail sampling both aggressive. Error signature: an error capture ratio far below one. Root cause: the two multiply. Remediation: set the application's sampler to always on and let the tail tier make the decision.

A latency threshold below the median. Error signature: no volume reduction despite a five percent probabilistic policy. Root cause: the slow policy matching most traffic. Remediation: set it per service from the actual distribution.

Memory sized without the window. Error signature: the sampling collector killed for memory under normal load. Root cause: in-flight traces held for the window were not budgeted. Remediation: compute span rate times window times span size, and set the memory limiter beneath the container limit.

Frequently Asked Questions