Tuning BatchSpanProcessor for Throughput

The default BatchSpanProcessor settings are chosen to be safe on a laptop. On a service producing thousands of spans a second they fill a 2048-span queue in well under a second, and the SDK starts dropping — silently, because dropping is the correct behaviour and nothing about it is an error. This page covers the arithmetic that tells you whether your settings work, and what to do when they do not. It builds on exporters and the OpenTelemetry Collector, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

export OTEL_BSP_MAX_QUEUE_SIZE=2048
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
export OTEL_BSP_SCHEDULE_DELAY=5000
export OTEL_BSP_EXPORT_TIMEOUT=10000

Implementation

Step 1 — Measure spans per request, not requests. This is the number that turns a comfortable-looking queue into a full one. Count from a real trace rather than estimating.

from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult

class CountingExporter(SpanExporter):
    """Wrap the real exporter to count spans in a staging run."""

    def __init__(self, inner: SpanExporter):
        self.inner, self.count = inner, 0

    def export(self, spans) -> SpanExportResult:
        self.count += len(spans)
        return self.inner.export(spans)

    def shutdown(self) -> None:
        print(f"exported {self.count} spans")
        self.inner.shutdown()

A typical instrumented request produces a server span, one per outbound HTTP call, one per database statement, plus manual ones — five to twenty is normal. At 400 requests per second and twelve spans each, the service produces 4 800 spans per second, and the default 2 048-span queue holds 0.4 seconds of that.

Step 2 — Check the drain arithmetic. The scheduled drain is max_export_batch_size / schedule_delay. The processor also exports immediately whenever a full batch has accumulated, so the real ceiling is how fast the exporter completes round trips — but if the scheduled rate is far below production, the configuration is relying entirely on the immediate path and has no headroom.

The middle two rows are the useful ones. If the export round trip is fast — a local Collector, single-digit milliseconds — the effective drain is far above production and the queue never fills. If the Collector is remote and each export takes 200 ms, the effective drain is 2 560 spans per second, below production, and the queue fills at 2 240 spans per second: full in under a second, dropping thereafter.

Step 3 — Size the queue for a blip. A few seconds of peak production is the right target.

from opentelemetry.sdk.trace.export import BatchSpanProcessor

SPANS_PER_SECOND = 4_800
processor = BatchSpanProcessor(
    exporter,
    max_queue_size=SPANS_PER_SECOND * 3,       # ~3 s of headroom
    max_export_batch_size=512,
    schedule_delay_millis=2000,                # tighter cadence, lower peak occupancy
    export_timeout_millis=10000,
)

Beyond that, memory is the constraint: a span with a dozen attributes is on the order of a kilobyte, so a 100 000-span queue is roughly 100 MB of live objects that the garbage collector must also walk.

Step 4 — Export the drop counter. The SDK counts dropped spans internally; publish it so the gap is attributable.

from opentelemetry import metrics

meter = metrics.get_meter("otel.self")
dropped = meter.create_observable_gauge(
    "otel_span_queue_dropped",
    callbacks=[lambda options: [metrics.Observation(processor._dropped_spans)]],
    description="Spans discarded because the export queue was full",
)

Alert on any sustained non-zero value. A trace missing its middle spans reads as "that service did not make the call", and an on-call engineer will act on that reading unless the drop counter says otherwise.

Step 5 — Sample rather than enlarge, when the imbalance is permanent. If occupancy sits at the ceiling, the fix is fewer spans, not more room for them.

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1        # whole traces, 10% of them

Head sampling keeps traces whole, which matters far more than keeping a random 10% of spans. The trade-offs are covered in sampling strategies for distributed tracing.

Configuration options

Verification

Drive a known number of spans through a deliberately slow exporter and confirm the drop count matches the arithmetic.

import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter, SpanExportResult

class SlowExporter(SpanExporter):
    def export(self, spans):
        time.sleep(0.5)                       # a Collector having a bad day
        return SpanExportResult.SUCCESS
    def shutdown(self): pass

processor = BatchSpanProcessor(SlowExporter(), max_queue_size=100, max_export_batch_size=10)
provider = TracerProvider()
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("probe")
for i in range(1000):
    with tracer.start_as_current_span(f"span-{i}"):
        pass

provider.force_flush(timeout_millis=2000)
print("dropped:", processor._dropped_spans)

Expected Output:

dropped: 871

That is the behaviour to internalise: nothing raised, nothing logged at WARNING, the application ran at full speed, and 87% of the spans do not exist. Then repeat with a fast exporter and confirm the count is zero — which is the assertion worth keeping in a staging smoke test.

Common mistakes

The queue was sized against requests

Error signature: a queue set to twice the peak request rate fills within a second. Root cause: each request produces five to twenty spans. Remediation: measure spans per request with a counting exporter, and size against the product.

Drops are invisible

Error signature: traces have gaps that look like missing work, and nothing in the logs mentions telemetry. Root cause: the SDK counts drops internally and reports them nowhere by default. Remediation: export the counter and alert on a sustained non-zero value.

The queue was enlarged instead of sampling

Error signature: memory climbs after raising max_queue_size, and drops continue. Root cause: the export rate is permanently below production, so the queue is a buffer for a deficit that never clears. Remediation: read the occupancy shape — a flat top means sampling, not sizing.

The parameters that interact

The four settings are not independent, and two pairs in particular produce surprising behaviour when tuned separately.

Batch size against export timeout. A larger batch takes longer to serialise and longer to transmit, so raising max_export_batch_size without raising the timeout increases the chance a batch times out — and a timed-out batch is dropped whole, so a larger batch means more spans lost per failure. The pairing to keep in mind is that the timeout must comfortably exceed the observed round trip for a full batch, with headroom for a slow moment.

Schedule delay against queue size. The delay sets how long a span may wait before its batch is sent, so it also sets how many spans accumulate between drains. A five-second delay on a service producing 4 800 spans per second implies up to 24 000 spans arriving between scheduled drains — which is why the immediate-export-on-full-batch path does most of the work at that volume, and why lowering the delay is often more effective than raising the queue.

Queue size against memory. Each queued span holds its attributes, events and links, so a span with twenty attributes and an exception event is several kilobytes rather than a few hundred bytes. A 100 000-span queue is therefore potentially hundreds of megabytes of live objects, all of which the garbage collector walks on every generation-2 pass — which is a second-order cost that shows up as longer GC pauses rather than as anything the tracing metrics report.

A worked starting point

For a service producing roughly 5 000 spans per second per process, with a Collector on the same host answering in single-digit milliseconds:

BatchSpanProcessor(
    OTLPSpanExporter(insecure=True),
    max_queue_size=15_000,          # ~3 s of production
    max_export_batch_size=512,      # small enough to serialise quickly
    schedule_delay_millis=1_000,    # tighter than the default; lowers peak occupancy
    export_timeout_millis=10_000,   # far above the observed round trip
)

The reasoning, in order: the queue covers a three-second blip, which is long enough to ride out a Collector restart and short enough that memory stays modest. The batch stays at the default because a local Collector completes it in milliseconds and a larger batch buys nothing. The delay drops to one second because at this volume the scheduled cadence should be doing real work rather than being bypassed entirely by the full-batch path. And the timeout is generous because its only job is to bound a pathological case.

Then measure. The occupancy shape from the second figure is the check: ripples near the bottom mean the settings are comfortable, a flat top means the arithmetic does not work and sampling is the answer.

Under multiple processes

Each worker in a prefork deployment has its own provider, queue and exporter thread, which means the per-process numbers above multiply by the worker count for memory and for connections to the Collector. Four workers with a 15 000-span queue each is 60 000 spans of headroom and four concurrent exporters, which is usually fine and is worth knowing before choosing a queue size from a per-service span rate rather than a per-process one.

It also means the provider must be built after the fork, for the reason set out in the Flask walkthrough: a BatchSpanProcessor created before forking has an exporter thread that does not survive, leaving each worker with a queue nothing drains.

Related

• Exporters and the OpenTelemetry Collector — the parent guide: where this processor sits in the export path.
• Configuring the OTLP span exporter — the exporter whose round-trip time sets the effective drain rate.
• Running the OpenTelemetry Collector for Python services — making that round trip fast by keeping it local.
• Sampling strategies for distributed tracing — the fix when the arithmetic does not work.
• Non-blocking logging with QueueHandler — the same bounded-queue trade for logs.

Frequently Asked Questions