Sampling Strategies for Distributed Tracing in Python OpenTelemetry

Sampling decides which traces you keep, and the wrong configuration either floods your backend with cost or silently drops the one error trace you needed at 3 a.m. This page is for engineers who already emit spans and now need to control their volume without losing the traces that matter. It gives exact OpenTelemetry Python SDK configurations for head-based, parent-based, and tail-based sampling and shows how they interact. It belongs to the Span Lifecycle and Attributes guide within Distributed Tracing and OpenTelemetry in Python; for the provider bootstrap these examples assume, see OpenTelemetry SDK Setup.

Prerequisites

You need a working TracerProvider, pinned SDK packages, and one decision made up front: whether a Collector sits between your services and the backend.

pip install \
  "opentelemetry-api>=1.30.0,<2.0.0" \
  "opentelemetry-sdk>=1.30.0,<2.0.0"

Head sampling can also be driven by environment variables read at startup, which is the recommended way to change rates per environment without touching code:

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"   # 10% of root traces

How Sampling Fits the Trace Lifecycle

Sampling answers one question — keep this trace or drop it — but where and when you answer it changes everything. Head sampling answers at the trace origin, the instant the root span is created, using only what is known then: the trace ID, the span name, the span kind, and the attributes passed at creation. It is cheap and predictable, and because the decision is encoded in the sampled bit of the traceparent header, it rides for free to every downstream service through context propagation and baggage. The cost is blindness: at span start you do not yet know whether the request will error or run slow, so you cannot preferentially keep the traces you most want.

Tail sampling answers after the trace completes, in the Collector, where the full set of spans, their statuses, and their latencies are all visible. That visibility is the whole point — keep every error and every slow path, sample the rest — but it requires buffering complete traces in memory and routing every span of a trace to the same Collector instance. Between the two sits parent-based sampling, which is not a third policy so much as the rule that makes head sampling coherent across services: a child honors its parent's decision so a trace is never half-kept.

The arithmetic is what usually settles the argument. At a flat 10% head rate you keep a tenth of your storage bill and a tenth of your error traces, because an error is no more likely to be sampled than a success. A service that produces 200 failed requests a day surfaces 20 of them at 10%, and two at 1% — which is no longer an investigation, it is a rumour. Tail sampling inverts that trade: keep 100% of errors and slow requests, sample the successful remainder at 1%, and total volume typically lands near the flat-10% figure while error coverage stays complete. Most production deployments therefore combine all three mechanisms: ParentBased(ALWAYS_ON) at the edge to capture whole traces, and a Collector tail_sampling policy to do the actual keep-or-drop on outcome. Pure head probability is the right choice only when Collector cost or operational complexity rules out tail sampling.

One consequence of dropping at the head is worth naming: a dropped span is a non-recording span, so the SDK stores no attributes, records no events, and never exports it. The residual cost is the sampler call itself plus a lightweight span object. Your own code is not free, though — an f-string or a database lookup used to build an attribute value still runs, so guard expensive attribute computation with span.is_recording(), exactly as covered in the span lifecycle and attributes guide.

Implementation

Step 1 — Pick the head sampler. TraceIdRatioBased(rate) makes a deterministic decision from the trace ID, so the same trace ID always yields the same keep-or-drop outcome across every service. That determinism is what keeps a trace whole: if the edge keeps it, downstream services keep it too.

Step 2 — Wrap it in ParentBased. ParentBased inspects the incoming span context. If a remote parent already decided to sample, the child honors that decision; only when there is no parent does it fall back to the root sampler. This is the correct default for every service except the very edge. Re-deciding sampling on a child service is the fastest way to fragment a trace, since half the spans get dropped while the rest are kept.

ParentBased exposes four delegate slots beyond the root: remote_parent_sampled, remote_parent_not_sampled, local_parent_sampled, and local_parent_not_sampled. The defaults are sensible — honor whatever the parent decided — but the slots let you, for example, force-sample whenever a remote parent was sampled while applying a probability to unsampled remote parents. In practice the only reason to touch them is to recover a fraction of unsampled traces for a specific high-value service; for everything else the bare ParentBased(root=...) form is correct. The determinism of TraceIdRatioBased is the other half of why this works: because the decision is a pure function of the 128-bit trace ID, an edge service and a service three hops downstream computing the same ratio against the same trace ID reach the same answer even if the parent flag were somehow lost.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

# 10% of root traces; downstream services inherit this decision
sampler = ParentBased(root=TraceIdRatioBased(0.1))
provider = TracerProvider(sampler=sampler)
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

The sampler is fixed for the life of the provider: it is read once in the constructor and never consulted for changes, so a rate change means a restart, or a custom sampler that reads a cached value. Set it before any instrumentation runs, ahead of the framework instrumentors described in instrumenting Python web frameworks, or the first requests are decided by the default ParentBased(ALWAYS_ON).

Step 3 — Add a custom sampler when business logic must override probability. Implement the Sampler interface and return a SamplingResult. Keep should_sample free of I/O; it runs on the request thread, and a blocking call there shows up directly in P99 latency. A custom sampler also sees only the attributes passed at span creation through the attributes= argument — anything added later with set_attribute is invisible to the decision — so route, tenant, or any other sampling key must be supplied at start_as_current_span() time. The sampler below always keeps critical routes and otherwise defers to a probabilistic delegate.

from opentelemetry.sdk.trace.sampling import (
    Sampler, SamplingResult, Decision, ParentBased, TraceIdRatioBased,
)
from opentelemetry.trace import SpanKind


class PriorityRouteSampler(Sampler):
    """Force-sample critical routes; delegate everything else to a 10% sampler."""

    def __init__(self):
        self._fallback = ParentBased(root=TraceIdRatioBased(0.1))

    def should_sample(self, parent_context, trace_id, name,
                      kind=SpanKind.INTERNAL, attributes=None,
                      links=None, trace_state=None) -> SamplingResult:
        route = (attributes or {}).get("http.route", "")
        if route.startswith("/api/critical"):
            return SamplingResult(Decision.RECORD_AND_SAMPLE, attributes, trace_state)
        return self._fallback.should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state)

    def get_description(self) -> str:
        return "PriorityRouteSampler{fallback=parentbased_traceidratio:0.1}"

Two variations on this shape cover most real requirements. Returning Decision.DROP for noisy paths — health checks, readiness probes, metrics scrapes — is usually worth more than any rate change, because those requests are high-volume and carry no diagnostic value. And Decision.RECORD_ONLY keeps the span recording locally without setting the sampled flag, which is how you feed span-derived metrics or local debugging without paying to export. A token-bucket rate limiter fits here too: the SDK ships no rate-limiting sampler, so cap per-second span starts yourself with a monotonic-clock bucket refilled inside should_sample, and never with a lock held across I/O.

Step 4 — Add tail sampling in the Collector for outcome-based retention. The SDK cannot tail-sample because it decides at span start, before latency or status is known. Run the Collector with a tail_sampling processor, and crucially set the SDK head sampler to ParentBased(ALWAYS_ON) so the Collector receives complete traces to evaluate.

The decision_wait window is the parameter that most often bites teams. It must be longer than your slowest realistic trace, because the processor evaluates a trace only after decision_wait elapses from the first span it sees. Set it too short and a slow trace's late spans arrive after the decision is made, so they are evaluated as a separate, incomplete trace and the latency policy misfires. Set it too long and the in-memory num_traces buffer fills, evicting traces before they are decided. Size num_traces to roughly expected_new_traces_per_sec × decision_wait with headroom, and remember that tail sampling is stateful per Collector instance: a load-balanced fleet must route all spans of a trace to the same instance, usually via a loadbalancing exporter keyed on trace ID, or partial traces land on different instances and each sees an incomplete picture.

Policies are evaluated as an OR: a trace is kept if any policy votes to keep it. So the idiomatic production set is errors, plus slow, plus a small probabilistic floor — the floor is what keeps a representative baseline of healthy traffic for latency comparisons, and it is easy to forget until every trace in your backend is a failure.

processors:
  tail_sampling:
    decision_wait: 10s              # window to assemble a full trace
    num_traces: 50000               # in-memory trace buffer
    expected_new_traces_per_sec: 100
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]

Configuration Options

The accepted values for OTEL_TRACES_SAMPLER are always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, and parentbased_traceidratio. Environment configuration is the right lever for per-environment rates — staging at 1.0, production at 0.1 — because it needs no code branch and no rebuild.

Verification

Run the head-sampling provider with a critical route and inspect the console export. A sampled span prints; a dropped one produces no output and reports False from is_recording().

with trace.get_tracer(__name__).start_as_current_span(
    "checkout", attributes={"http.route": "/api/critical"}
) as span:
    print("recording:", span.is_recording())

Expected Output:

recording: True
{
  "name": "checkout",
  "context": {"trace_id": "0x7b8a...", "span_id": "0x1234...", "trace_state": "[]"},
  "kind": "SpanKind.INTERNAL",
  "parent_id": null,
  "status": {"status_code": "UNSET"},
  "attributes": {"http.route": "/api/critical"}
}

To check the ratio itself rather than a single span, exercise the sampler directly over synthetic trace IDs. Because the decision is a pure function of the trace ID, this converges on the configured rate without generating any traffic:

import random
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, Decision

sampler = TraceIdRatioBased(0.1)
kept = sum(
    sampler.should_sample(None, random.getrandbits(128), "probe").decision
    is not Decision.DROP
    for _ in range(100_000)
)
print(f"kept {kept / 1000:.2f}% of 100000 trace ids")

Expected Output:

kept 10.03% of 100000 trace ids

In a live service, cross-check the same ratio end to end: the trace count your backend reports for a window, divided by the request count your metrics report for that window, should track the configured rate. That comparison relies on request-level counters — see Python metrics and instrumentation — and a measured keep rate that drifts far from the configured ratio almost always means a child service is re-deciding sampling instead of inheriting the parent decision.

On the Collector side, the tail_sampling processor publishes its own metrics: otelcol_processor_tail_sampling_count_traces_sampled broken down by policy tells you which policies actually fire, and otelcol_processor_tail_sampling_sampling_trace_dropped_too_early is a direct signal that decision_wait is shorter than your real trace durations. Watch both after any policy change.

Common Mistakes

Mixing head and tail sampling without coordination. Error signature: fragmented trace graphs and trace-ID lookups returning partial or no spans. Root cause: an edge service head-samples at 10% while the Collector runs tail policies, so the policies only ever see the 10% that survived. Remediation: when tail sampling, set the edge SDK to ParentBased(ALWAYS_ON) and never head-drop upstream of a tail policy.

Re-deciding sampling in a child service. Error signature: broken traceparent continuity, orphaned spans, keep rates that differ per service. Root cause: a downstream TracerProvider was initialized with a bare TraceIdRatioBased root sampler instead of ParentBased, so it ignores the inbound decision. Remediation: use ParentBased everywhere except the edge, and confirm the inbound decision actually arrives — that depends on correct context propagation and baggage, including over brokers when propagating trace context across Celery tasks.

Blocking I/O inside a custom sampler. Error signature: P99 latency spikes and thread-pool or event-loop stalls under load. Root cause: should_sample calls a database, a feature-flag service, or a config endpoint on the request path. Remediation: keep should_sample to attribute lookups and arithmetic, refresh any dynamic rate on a background thread into a plain variable, and target sub-100-microsecond evaluation.

Tail sampling behind a round-robin load balancer. Error signature: traces arrive in the backend missing whole services, and the drop-too-early metric climbs. Root cause: spans of one trace are spread across Collector replicas, so no single replica ever sees the complete trace. Remediation: put a Collector layer with a loadbalancing exporter keyed on trace ID in front of the tail-sampling layer, so every span of a trace lands on the same instance.

Related

• Span lifecycle and attributes — the parent guide covering span creation, attributes, status, and export.
• OpenTelemetry SDK setup — provider bootstrap and processor tuning that every sampler configuration sits on.
• Context propagation and baggage — how the sampled flag travels between services and why parent-based sampling depends on it.
• Setting up OpenTelemetry in FastAPI — where to place the sampler in an ASGI startup path.
• Adding trace IDs to log records — how to keep debugging context for the traces sampling drops.

Frequently Asked Questions