Backpressure, Retries and Delivery Guarantees

A telemetry pipeline that never loses anything does not exist, and one that promises it is usually about to convert a backend problem into a latency problem. This guide covers what a Python telemetry path actually guarantees, the three places it drops, how to bound retries so they help rather than hurt, and how to buy a stronger guarantee for the small subset of records that deserve one. It is part of the Python telemetry pipelines and delivery section and assumes the topology decisions in collector topology and deployment are already made. The focused articles in this topic are Buffering Telemetry During Backend Outages, Detecting Dropped Spans and Metrics, Graceful Shutdown and Telemetry Flush and Handling OTLP Export Retries and Timeouts.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

Concept and architecture

Delivery guarantees are usually described with three phrases, and only one of them is honest about a default telemetry pipeline.

At-most-once means a record is delivered zero or one times: it is never duplicated, and it may be lost. This is what an OpenTelemetry SDK and a default collector provide, and for spans and debug logs it is the correct choice, because the cost of the alternatives is paid continuously while the benefit is realised only during failures.

At-least-once means a record is delivered one or more times: nothing is lost, and duplicates are possible. Buying it requires durable storage before acknowledgement — a file-backed queue in the collector, or a broker between tiers — and it requires the consumer to tolerate duplicates. For telemetry with business meaning, an audit trail or a billing event, it is worth the disk.

Exactly-once does not exist across a network boundary without a transactional protocol on both ends, and nothing in a telemetry pipeline offers one. Systems that claim it are providing at-least-once delivery plus deduplication at the consumer, which is a reasonable thing to build and a misleading thing to call exactly-once.

The decision that matters is per signal, not per pipeline. A fleet can sensibly run at-most-once for spans and debug logs, and a separate durable path for the handful of record types whose loss would be a business problem. Trying to give everything the strongest guarantee produces a pipeline that is expensive, slow and — because the disk becomes a bottleneck — frequently less reliable than the simple one it replaced.

Step-by-step implementation

Step 1 — Decide, per signal, what a loss costs. Write it down, because every parameter below follows from it. Spans during a backend outage: acceptable, because the traces from a period when the backend was down are rarely the ones anybody queries. Metrics: mostly acceptable, since a counter's next value carries the total anyway. Error-level logs: uncomfortable. Audit records: not acceptable, and therefore not this pipeline's problem.

Step 2 — Bound the retry, and understand what the bound trades. An exporter retrying a failed batch holds the exporter, so nothing else is being sent during the retry. A deadline of thirty seconds means a thirty-second backend blip costs nothing, and a five-minute outage costs the queue. A deadline of ten minutes means the same five-minute outage costs everything produced during it, because the first failed batch is still being retried while the queue fills and discards.

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

processor = BatchSpanProcessor(
    # 1. Per-attempt timeout: short, because the next hop is local.
    OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True, timeout=5),
    # 2. Queue: how much production the pipeline can absorb before dropping.
    max_queue_size=4096,
    max_export_batch_size=512,
    schedule_delay_millis=5000,
)

Step 3 — Size the queue from arithmetic rather than habit. The queue holds what production produces while export is not draining it. Spans per second multiplied by the seconds you intend to survive gives the size; multiply by the average span size to get the memory. A service producing 400 spans per second, intending to survive a 20-second collector restart, needs 8000 slots and roughly 8 megabytes — which is affordable, and would not have been at 200 seconds.

Step 4 — Keep the drop-rather-than-block default. The SDK discards when the queue is full. It is worth being explicit about why this is right: the alternative applies backpressure into the request path, so a telemetry backend outage becomes elevated latency on every endpoint, then timeouts, then a real incident caused entirely by the system that was supposed to observe it.

Step 5 — Add durability only on the path that needs it. A collector's sending queue can be backed by a file, so records survive the collector restarting. This costs a write per batch and a volume to manage, and it is the right tool for a gateway holding many agents' data.

exporters:
  otlp/vendor:
    endpoint: ingest.vendor.example:443
    sending_queue:
      enabled: true
      queue_size: 10000
      storage: file_storage/telemetry     # survives a restart
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 300s

extensions:
  file_storage/telemetry:
    directory: /var/lib/otelcol/queue
    timeout: 1s

Step 6 — Flush before exit, every time. The third loss point has no counter and no automatic remedy. A termination handler that shuts down the providers is three lines and recovers the records describing the shutdown, which are disproportionately the interesting ones.

Configuration reference

Async and concurrency considerations

None of this machinery runs on the event loop, which is the first thing worth knowing for an asyncio service. The batch processor exports from a dedicated thread, the metric reader from its own, and the log processor from a third. A slow or failing next hop therefore cannot cause event loop lag, and a service whose latency rises during a telemetry outage is almost certainly suffering from something else.

What the export threads do consume is the interpreter lock, during protobuf serialisation. That cost is proportional to the bytes exported and appears as a small steady CPU tax rather than a latency spike. It becomes visible only at very high span rates, and the remedy at that point is fewer spans rather than different export settings.

There is one genuine interaction worth watching. BatchSpanProcessor.on_end is called on whichever thread ended the span, and it performs a bounded enqueue. The enqueue is cheap, but it is not free, and when the queue is full the SDK's drop path runs on the caller's thread. At very high drop rates this is measurable inside request handling — which is a strange failure mode, because it means the cost of telemetry rises precisely when the telemetry is not being delivered. Keeping the queue from saturating therefore has a performance justification as well as a data one.

For multiprocess servers, each worker has its own provider, its own queue and its own exporter. The arithmetic in step 3 is per worker, and the memory cost is multiplied by the worker count — a detail that turns an affordable queue size into an expensive one on a thirty-two worker box.

Production code examples

A configuration module that makes each of the three loss points explicit and measurable:

# delivery.py
import logging
import os
import signal
import threading

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

log = logging.getLogger(__name__)

SPANS_PER_SECOND = int(os.environ.get("EXPECTED_SPAN_RATE", "400"))
SURVIVE_SECONDS = int(os.environ.get("SURVIVE_STALL_SECONDS", "20"))

provider = TracerProvider(resource=Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
}))
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint=os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"], timeout=5),
    # 1. Queue sized from the stall you intend to survive, not from a default.
    max_queue_size=SPANS_PER_SECOND * SURVIVE_SECONDS,
    max_export_batch_size=512,
    schedule_delay_millis=5000,
))
trace.set_tracer_provider(provider)

_flushed = threading.Event()


def flush(signum=None, frame=None):
    # 2. The third loss point, closed.
    if not _flushed.is_set():
        provider.shutdown()
        _flushed.set()
        log.info("telemetry flushed at shutdown", extra={"signal": signum})


signal.signal(signal.SIGTERM, flush)
signal.signal(signal.SIGINT, flush)

Expected Output: a clean shutdown, with the flush accounted for.

2026-09-18T12:41:02Z INFO  draining http server
2026-09-18T12:41:03Z INFO  telemetry flushed at shutdown signal=15

A probe that reports which of the three loss points is active, which is the question worth asking during an incident:

# 1 — the SDK's own queue (exposed when the SDK's internal metrics are enabled)
curl -s localhost:9464/metrics | grep -E 'span_processor_(queue_size|dropped)'

# 2 — the exporter's failures, at the collector
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_send_failed_spans'

# 3 — shutdown losses, from the application's own log
kubectl logs deploy/checkout --previous | grep -c "telemetry flushed at shutdown"

Expected Output: a healthy pipeline reports a small queue, no failures, and a flush on every previous termination.

otel_sdk_span_processor_queue_size 41
otel_sdk_span_processor_dropped_spans_total 0
otelcol_exporter_send_failed_spans 0
1

What a guarantee is worth per signal

It is tempting to treat delivery as a single dial for the whole pipeline, and the reason not to is that the three signals have very different loss profiles.

Metrics recover on their own. A counter is cumulative, so a missed export loses resolution rather than information: the next successful export carries the total, and the only thing lost is the shape of the curve during the gap. A gauge loses more, since its value is a point in time, and a histogram loses the distribution for the interval. In practice a metrics pipeline can tolerate significant loss without anybody being unable to answer a question, which is why metrics are usually the first signal to be given the smallest queue.

Traces lose disproportionately. A trace missing some of its spans is worse than a trace that is absent, because it is silently incomplete and an engineer reading it draws conclusions from an incomplete picture. This argues for making the trace pipeline's loss all-or-nothing where possible — which is what a tail sampling gateway provides, since it decides on whole traces — rather than for making it lossless.

Logs are where loss is noticed. An engineer looking for a specific record and not finding it cannot distinguish "it was never logged" from "it was dropped", and will usually conclude the former and go looking in the code. This is the strongest argument for the sequence numbering described in log shipping and collection: it makes the distinction observable, and an observable loss is a much smaller problem than an invisible one.

Where backpressure actually goes

Backpressure is the word for what a full buffer does to the thing filling it, and in a telemetry pipeline there are exactly three possible answers: drop, block, or push the problem to the previous tier. Each appears somewhere in a normal deployment, and confusing them is the source of most surprising behaviour.

The SDK drops. When a span processor's queue is full, the span is discarded on the thread that ended it, and a counter increases. Nothing waits. This is the right default because the alternative reaches into request handling, and it is also the reason the queue's size is a real decision rather than a tuning detail: it is the entire buffer between a stalled pipeline and data loss.

The collector refuses. The memory limiter watches the process's memory and, past its threshold, starts rejecting incoming data at the receiver with an error the sender can see. This is genuine backpressure: the previous tier learns that the next one is saturated, and can retry. An agent refusing data causes the application's exporter to retry, which fills the application's queue, which eventually drops — the chain terminates in the SDK, as it must, because the application cannot store data indefinitely either.

The persistent queue defers. A file-backed sending queue converts a network problem into a disk problem, which is a good trade until the volume is exhausted. At that point it also refuses, and the chain proceeds as above. What persistence buys is time, not immunity, and the amount of time is the volume size divided by the production rate — a number worth calculating before trusting it.

The practical consequence is that every tier's buffer only delays the decision, and the decision is always made at the source. A pipeline designed as though the queues will never fill is a pipeline whose behaviour under stress is undefined by accident rather than by choice.

Measuring the guarantee you actually have

The gap between the guarantee a pipeline is configured for and the one it delivers is measurable, and measuring it is cheaper than reasoning about it.

The most direct method is a synthetic producer. A small job emits a known number of records per minute, each carrying a sequence number and a marker attribute, through exactly the same pipeline as production telemetry. A query against the backend counts what arrived. The ratio is the pipeline's delivery rate, continuously, without any dependence on what the real services happen to be doing — and because the producer's rate is known, a shortfall is unambiguous rather than being confused with a quiet period.

The second method is reconciliation at the boundaries. The application knows how many spans it created; the agent knows how many it accepted and sent; the gateway knows the same; the backend knows how many it stored. Four numbers that should agree, and the pair that disagrees names the tier responsible. This is more work to set up than a synthetic producer and considerably more informative during an incident, because it localises the loss instead of only detecting it.

The third is the sequence-gap approach described elsewhere in this section, which needs one integer per record and detects loss in the real data rather than in a probe. Its advantage is that it measures the pipeline that actually matters; its limitation is that it only works per producing process, so it tells you a service lost records without telling you where.

Most fleets need only the first. The others earn their cost once a pipeline has misbehaved in a way nobody could explain, which is, in practice, how most of them get adopted.

Common mistakes

A long retry deadline treated as a reliability improvement. Error signature: during outages, complete data for the first minute and nothing afterwards. Root cause: one batch occupying the exporter while the queue discards everything behind it. Remediation: shorten the deadline; a thin sample across the whole outage is more useful.

A queue sized by copying an example. Error signature: drops during ordinary traffic peaks, or memory pressure at idle. Root cause: no relationship between the configured size and the service's span rate. Remediation: compute it, as in step 3, and record the assumption next to the number.

Blocking instead of dropping. Error signature: request latency rising in step with telemetry backend latency. Root cause: a queue configured to apply backpressure into the caller. Remediation: restore the drop behaviour and route genuinely critical records to a separate durable path.

Durability everywhere. Error signature: collector throughput collapsing under load, with disk wait dominating. Root cause: a file-backed queue applied to the whole telemetry volume. Remediation: reserve persistence for the pipeline that carries records with business meaning.

No flush on shutdown. Error signature: pods that terminate cleanly with no telemetry from their final seconds. Root cause: the third loss point, which has no counter. Remediation: a termination handler that calls shutdown on every provider, and a grace period long enough for it to complete.

Nothing scraping the drop counters. Error signature: a pipeline that has been losing a fifth of its data for months. Root cause: the counters exist and nobody reads them. Remediation: scrape the SDK's and the collector's own metrics from somewhere that is not downstream of the collector.

Frequently Asked Questions