Buffering Telemetry During Backend Outages

Every telemetry buffer is a bet that an outage will be shorter than the buffer. This page covers how to size that bet from the fleet's real numbers, where to place the buffer so the same coverage costs less, and what a persistent queue genuinely buys. It is a task article under backpressure, retries and delivery guarantees, part of the Python telemetry pipelines and delivery section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

The collector needs the file storage extension if a persistent queue is in scope.

Implementation

Step 1 — Measure production rather than estimating it. Buffer arithmetic is only as good as the rate that goes into it, and the rate is easy to get wrong by an order of magnitude because spans per request varies enormously between a service with two spans and one with forty. The SDK's own counters give the real number, and it should be measured at peak rather than at average, since the outage that matters is the one during the busy hour.

# read the production rate straight from the SDK's internal metrics
# spans created per second, per service, over the last five minutes
# rate(otel_sdk_span_processor_processed_spans_total[5m])

Step 2 — Choose a target duration from the backend's history, not from a round number. Managed backends tend to have a bimodal failure distribution: many blips of under a minute, and rare outages of an hour or more. A buffer covering two to five minutes absorbs nearly all of the first category, which is where most of the cumulative data loss lives. Covering the second category requires a buffer two orders of magnitude larger to protect data from an event that happens once a year, which is almost never worth the standing cost.

Step 3 — Do the arithmetic and write it down. Rate multiplied by duration gives the record count; record count multiplied by average size gives the memory. Recording both the inputs and the result next to the configuration means the next person to change it knows what it was meant to cover.

SPANS_PER_SECOND = 4000          # measured at peak, whole fleet
SURVIVE_SECONDS = 120            # from the backend's incident history
AVG_SPAN_BYTES = 1100            # measured from an export payload

records = SPANS_PER_SECOND * SURVIVE_SECONDS      # 480_000
memory_mb = records * AVG_SPAN_BYTES / 1_048_576  # ~503 MiB, fleet-wide
print(f"{records:,} records · {memory_mb:.0f} MiB across the tier holding them")

Expected Output:

480,000 records · 503 MiB across the tier holding them

Step 4 — Place the buffer where it is shared. Five hundred megabytes at a three-replica gateway is one hundred and seventy megabytes per replica, and it is shared: the capacity is available to whichever service needs it. The same coverage bought in two hundred application processes is a reservation in every pod's memory limit, present whether or not there is an outage, and unusable by any process other than its owner. The application's queue therefore only needs to cover what the application can see failing, which is an agent restart — seconds, not minutes.

# gateway: where the fleet's outage coverage lives
exporters:
  otlp/vendor:
    sending_queue:
      enabled: true
      queue_size: 160000          # this replica's share of 480 000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 120s

Step 5 — Persist only where loss is a business problem. A file-backed queue survives a collector restart, which in-memory queues do not. The cost is a disk write per batch, forever, plus a volume to provision and monitor. That is a good trade for a pipeline carrying audit or billing records and a poor one for spans, where the write cost is paid continuously to protect data that is, by construction, from a period when the system was already broken.

extensions:
  file_storage/audit:
    directory: /var/lib/otelcol/audit-queue
    fsync: true                    # the whole point, and the whole cost

exporters:
  otlphttp/audit:
    sending_queue: { enabled: true, queue_size: 20000, storage: file_storage/audit }

Which records to lose when the buffer fills

Every buffer eventually fills, and the component's overflow policy decides which records are lost. This is rarely configured deliberately and it matters more than the size.

The default in most queueing components is to reject the newest arrival when full — the queue is at capacity, so the incoming record is discarded. For telemetry this is usually the wrong way round. During an outage, the records still in the queue describe the beginning of the incident and the ones being rejected describe what is happening now. An engineer investigating afterwards wants continuity up to the present far more than completeness at the start, which argues for a ring-buffer policy that discards the oldest instead.

Very few telemetry components offer that choice, so the practical lever is elsewhere: a shorter retry deadline, as covered in handling OTLP export retries and timeouts, has approximately the same effect. Abandoning old batches quickly frees the exporter to attempt newer ones, so the surviving data is spread across the outage rather than concentrated at its start. Thinking of the retry deadline as an overflow policy in disguise makes the choice of its value considerably easier.

There is a second question underneath: whether all records are equally worth keeping. They are not. An error span, a failed request, a log record at error level — these are the ones an investigation begins from, and a buffer that discards them alongside successful requests is discarding disproportionate value. A filter that drops successful, fast, unremarkable traces before the queue, so that the buffer holds only what is interesting, multiplies the effective coverage without any additional memory. That is the same reasoning as tail sampling and can be applied specifically as an outage response, which is covered in tail sampling in the OpenTelemetry Collector.

What a buffer cannot do

Two limits are worth stating plainly, because a buffer is frequently asked to solve problems it structurally cannot.

A buffer cannot help when the producer is the thing that stops. A pod evicted during a node drain, a worker killed for memory, a process that segfaults — in each case the queue is in that process's memory and goes with it, however large it was. Coverage for that failure comes from flushing on termination and from holding the data one tier out, not from more queue. This is why a fleet with generous application-side buffers can still lose almost everything during a rolling update: every process is terminated in turn, and each takes its buffer along.

A buffer also cannot help when the outage is longer than the buffer and the data keeps being produced, which is the case it is most often asked about. At that point the only remaining levers reduce production: a lower sample rate, a filter that keeps errors and discards successes, or a level change that stops emitting debug records. These can be applied at the collector during an incident without touching any application, which makes "reduce what we produce" a live operational response rather than a design-time decision. Having the configuration ready in advance — a filter that can be switched on when the gateway's queue passes a threshold — turns an hour-long outage from a total loss into a degraded but continuous signal.

Configuration options

Verification

The check is whether the buffer actually covers what it claims. Stop the destination for the target duration and count what arrives afterwards.

# block the backend for exactly the target window, then restore
kubectl scale deploy/backend-proxy --replicas=0
sleep 120
kubectl scale deploy/backend-proxy --replicas=2

# did everything produced during the window eventually arrive?
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_(sent|send_failed)_spans'

Expected Output: a failure count that returns to flat, and a sent count that catches up to what was received.

otelcol_receiver_accepted_spans   1_284_200
otelcol_exporter_sent_spans       1_284_200
otelcol_exporter_send_failed_spans      0

A sent count that stays permanently below the accepted count is the amount the buffer was short by, expressed in records, which is the most direct possible input to resizing it.

Common mistakes

Buffering in the application. Error signature: pod memory limits raised across the fleet to accommodate telemetry queues. Root cause: coverage for a backend outage purchased in the tier with the most instances. Remediation: keep the application's queue small and hold the outage coverage at the gateway.

A target duration chosen as a round number. Error signature: a buffer that is simultaneously too small for real incidents and too large for the budget. Root cause: no reference to the backend's actual failure distribution. Remediation: look at the incident history and size for the common case.

Persistence everywhere. Error signature: collector throughput limited by disk wait. Root cause: fsync on every batch of a full telemetry volume. Remediation: persist the audit pipeline only, and leave spans and debug logs in memory.

Never testing the buffer. Error signature: an outage that loses far more than expected. Root cause: a configured size that was never exercised. Remediation: run the verification above in staging at production rates, at least once per significant change.

Buffering data that will not be useful. Error signature: a large queue full of successful, unremarkable traces. Root cause: no filtering before the buffer. Remediation: drop the uninteresting traffic ahead of the queue so the same memory holds several times more of what matters.

Frequently Asked Questions