Python Observability

Handling OTLP Export Retries and Timeouts

Retry configuration looks like a detail and behaves like a policy: the same four numbers decide whether a thirty-second backend hiccup is invisible or costs five minutes of telemetry. This page covers what each number controls, how they interact, and which failures are worth retrying at all. It is a task article under backpressure, retries and delivery guarantees, part of the Python telemetry pipelines and delivery section.

The two clocks running at once A single batch is attempted repeatedly along a timeline. Each attempt is bounded by the per-attempt timeout, drawn as a short block, and the gaps between attempts grow exponentially from one second to two, four, eight and sixteen, up to a ceiling of thirty seconds. Above the attempts runs the total deadline, a bar that begins with the first attempt and ends at the configured maximum elapsed time, at which point the batch is abandoned regardless of how many attempts remain possible. Below the attempts is the queue behind this batch, drawn filling steadily throughout because nothing is draining while the exporter is occupied. The point drawn out is that the deadline does not only decide how hard the pipeline tries; it decides how much newer data is discarded while it is trying. one batch, many attempts, two clocks total deadline — max_elapsed_time 1s 2s 4s 8s 16s 30s ceiling each block is one attempt, bounded by the per-attempt timeout meanwhile, behind this batch queue filling queue full newer spans discarded at the queue, unattempted the deadline decides how hard the pipeline tries — and how much fresher data it throws away while trying this is why "retry longer" is not the same as "lose less"
Two clocks run during an outage: the one bounding this batch, and the one filling the queue behind it. Only the first is usually configured.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

Implementation

Step 1 — Set the per-attempt timeout from the hop's real latency. A timeout is the longest a single attempt is allowed to occupy the exporter, so it should be comfortably above the normal round trip and well below the point where waiting stops being useful. Against a collector on the same node, a healthy export completes in single-digit milliseconds, and five seconds is already three orders of magnitude of headroom. Against a remote backend across the internet, ten to fifteen seconds is reasonable. Leaving it at a large default means a single stalled connection blocks the exporter for that entire period while the queue fills.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",
    insecure=True,
    timeout=5,        # seconds, per attempt
)

Step 2 — Bound the total retry time in the collector. The collector is where the serious retry policy belongs, because it has the richest controls and because a change there does not require redeploying anything. Three parameters interact: the initial wait, the ceiling on that wait, and the total elapsed time after which the batch is abandoned.

exporters:
  otlp/vendor:
    endpoint: ingest.vendor.example:443
    timeout: 10s                   # per attempt
    retry_on_failure:
      enabled: true
      initial_interval: 1s         # first wait
      max_interval: 30s            # ceiling on the growth
      max_elapsed_time: 120s       # give up here, and move on

Step 3 — Understand what the deadline trades. Extending max_elapsed_time does not reduce total loss; it changes which records are lost. A short deadline abandons batches quickly and keeps attempting newer ones, so the data that survives an outage is a thin sample spread across it. A long deadline keeps working on the oldest batch while everything newer piles up behind and is discarded at the queue, so the data that survives is a complete record of the outage's first moments and nothing after. For diagnosing an incident, the thin sample is almost always more useful, because it shows what happened throughout rather than what happened before anyone noticed.

Step 4 — Do not retry what cannot succeed. Retry is worthwhile only for conditions that can resolve without the payload changing: the endpoint being unavailable, a timeout, a rate limit, a transient network failure. A payload rejected as malformed, too large, or unauthorised will be rejected identically every time, and retrying it wastes the exporter for the full deadline while valid data queues behind it. Both the SDK and the collector distinguish these, but a custom exporter or a proxy in the path can blur the distinction, which is worth checking when retries are high and the backend reports nothing wrong.

Step 5 — Add jitter, or add it by accident. When a backend recovers from an outage, every client whose backoff expired at the same moment attempts simultaneously, which can push it straight back down. The collector's backoff includes randomisation; a hand-rolled retry loop usually does not. If you are writing one — and mostly you should not be — the randomisation is not optional.

Expected Output: an outage as it appears in the collector's own log, with the backoff visible and the abandonment explicit.

2026-09-18T13:02:11Z warn  exporterhelper  Exporting failed. Will retry. {"error": "rpc error: code = Unavailable", "interval": "1s"}
2026-09-18T13:02:14Z warn  exporterhelper  Exporting failed. Will retry. {"error": "rpc error: code = Unavailable", "interval": "2.4s"}
2026-09-18T13:02:21Z warn  exporterhelper  Exporting failed. Will retry. {"error": "rpc error: code = Unavailable", "interval": "4.8s"}
2026-09-18T13:04:11Z error exporterhelper  Dropping data because sending_queue is full or retry deadline exceeded {"dropped_items": 2048}
Which failures are worth retrying Two columns of response conditions. The left column holds conditions that can resolve without the payload changing: unavailable, deadline exceeded, resource exhausted from rate limiting, and connection reset. Each is marked as worth retrying with backoff, because the same bytes may well succeed later. The right column holds conditions that are properties of the payload or the credential: invalid argument for a malformed message, unauthenticated for a bad or expired token, permission denied, and a payload that exceeds the receiver's size limit. Each is marked as permanent, because retrying transmits identical bytes to a receiver that has already evaluated them. Below both columns is the cost of getting this wrong, which is that a permanently failing batch occupies the exporter for its whole deadline while valid data queues behind it and is dropped. retry only what the passage of time can fix worth retrying, with backoff UNAVAILABLE — the endpoint is down or restarting DEADLINE_EXCEEDED — it was slow this time RESOURCE_EXHAUSTED — rate limited, so wait connection reset — the path changed under us the same bytes may succeed later permanent — do not retry INVALID_ARGUMENT — the message is malformed UNAUTHENTICATED — the token is wrong PERMISSION_DENIED — it will stay denied payload too large — it is the same size next time identical bytes, identical verdict a permanent failure retried for two minutes costs two minutes of everything else
The classification matters more than the intervals. A permanently failing batch retried politely is still a two-minute outage for everything behind it.

Choosing the four numbers

The parameters are not independent, and a useful way to choose them is to start from the outage you intend to absorb and work backwards.

Suppose the next hop is a gateway that takes twenty seconds to roll out a new version, and the service produces four hundred spans per second. The queue must hold twenty seconds of production — eight thousand spans — or data is lost regardless of retry settings. The retry deadline must exceed the rollout, so sixty seconds is comfortable and one hundred and twenty is generous. The per-attempt timeout should be short, because during a rollout the connection fails fast rather than hanging, so five seconds only ever applies to the pathological case.

Now change one assumption: the next hop is an external vendor with occasional multi-minute degradations. The queue cannot hold five minutes of production without becoming expensive, so some loss is certain. The question becomes which loss, and the answer from the previous section is to keep the deadline short — sixty seconds — so the pipeline keeps sampling across the outage rather than perfecting its first batch.

A third case makes the trade explicit. A records pipeline carrying billing events, where loss is unacceptable, cannot solve this with retry parameters at all: it needs a persistent queue, so that the deadline governs how long records wait on disk rather than how many are discarded from memory. Recognising which case you are in is more valuable than any specific interval, because the first two are tuning and the third is a different design.

One further interaction is easy to miss. The per-attempt timeout and the batch schedule delay together determine how far behind the exporter can fall before the queue is the binding constraint. If a batch is produced every five seconds and each attempt takes up to ten, the exporter is structurally unable to keep up during any period of slowness, and the queue drains only once the backend is healthy. Keeping the attempt timeout below the batch interval avoids that regime entirely.

The four numbers and what each protects A table of the four settings that govern OTLP export under failure and what each protects. The export timeout, per attempt, protects the export thread from hanging on a stalled connection. The initial retry backoff protects the collector from an immediate retry storm. The maximum backoff caps how long a single wait can grow. The total retry deadline, or maximum elapsed time, bounds how long a batch is held before it is dropped, which protects memory. The note says the deadline must be shorter than the interval at which new batches fill the queue, or retries cause queue overflow. setting typical protects export timeout 10 s the export thread from a stalled connection initial backoff 1–5 s the collector from a retry storm maximum backoff 30 s against unbounded waits total retry deadline 60–300 s memory — how long a batch is held a deadline longer than the queue can absorb turns retries into overflow
Each number bounds a different failure. Together they decide how long an outage can last before data is dropped.

Configuration options

Parameter Local hop Remote hop Notes
per-attempt timeout 5 s 10–15 s below the batch interval where possible
initial_interval 1 s 1 s first wait after a failure
max_interval 30 s 30 s ceiling on backoff growth
max_elapsed_time 60–120 s 60 s shorter keeps sampling across an outage
jitter built in built in do not remove it in custom code
retry on 4xx never never identical bytes, identical verdict
queue size outage × rate outage × rate the real determinant of loss

Verification

Reproduce an outage deliberately and observe both the backoff and the abandonment, because a configuration that has never failed has never been tested.

# stop the next hop for 90 seconds, then bring it back
kubectl scale deploy/otel-gateway --replicas=0
sleep 90 && kubectl scale deploy/otel-gateway --replicas=3

# watch the agent's counters through the window
watch -n5 "curl -s localhost:8888/metrics | grep -E \
  'otelcol_exporter_(send_failed_spans|queue_size|sent_spans)'"

Expected Output: failures rise, the queue fills partway, and recovery drains it without a restart.

otelcol_exporter_send_failed_spans   2048
otelcol_exporter_queue_size          1841
otelcol_exporter_sent_spans        184203   # climbing again after recovery

A queue that reaches its maximum and stays there after recovery means the exporter is not draining faster than production, which is a throughput problem rather than a retry problem and needs a different fix.

Common mistakes

A large default timeout left in place. Error signature: one stalled export blocking all telemetry for a minute. Root cause: the per-attempt timeout is longer than the period in which the queue fills. Remediation: set it explicitly, matched to the hop, as in step 1.

Raising the deadline after an incident. Error signature: the next outage produces complete data for its first minute and nothing afterwards. Root cause: treating retry duration as a reliability control. Remediation: shorten it; increase the queue instead if the goal is to lose less.

Retrying permanent rejections. Error signature: high retry counts with a backend reporting no errors. Root cause: a malformed or oversized payload, or an expired credential, retried to the deadline. Remediation: confirm the status code being retried and stop retrying the permanent classes.

No jitter in a custom retry loop. Error signature: a backend that recovers and immediately fails again. Root cause: every client retrying at the same instant. Remediation: use the SDK or collector's retry rather than a hand-written loop, and randomise if you must write one.

Retrying in two places at once. Error signature: an outage that produces far more attempts than either configuration would explain. Root cause: the application's SDK retries into an agent that is itself retrying outward, so the attempt counts multiply rather than add. Remediation: keep the application's retry minimal because its hop is local and reliable, and let the collector own the policy against the unreliable hop.

Alerting on retries. Error signature: an alert that fires several times a week and is always ignored. Root cause: retries are normal. Remediation: alert on batches abandoned after the deadline, which is the number that means data was actually lost.

Frequently Asked Questions

What is the difference between the exporter timeout and the retry deadline?

The timeout bounds a single attempt; the deadline bounds the whole sequence of attempts for one batch. A five second timeout with a two minute deadline means up to roughly a dozen attempts, each given five seconds, before the batch is abandoned.

Should a 400-class response be retried?

No. A rejection because the payload is malformed, too large or unauthorised will be rejected identically on every attempt, so retrying wastes the exporter's time and delays everything behind it. Only conditions that can change on their own — unavailable, timeout, resource exhausted — are worth retrying.

Why does my collector report retries but the backend looks healthy?

Usually rate limiting. A backend that returns a resource-exhausted status when a client exceeds its quota is healthy and is telling you to slow down. The fix is less volume or a higher quota, not more aggressive retrying.

Does the Python SDK retry by default?

The OTLP exporters implement retry with exponential backoff internally, bounded by an overall deadline. Its parameters are less configurable than the collector's, which is one more reason to keep the application's hop local and short and to do the serious retrying one tier out.

Related Guides