OpenTelemetry Exporters and the Collector

A span that is recorded but never leaves the process is a span that does not exist. This guide covers the export path in a Python service — the processor, the exporter, the transport, and the Collector that should sit between you and any backend — for backend engineers and SREs who have instrumentation working locally and now need it to survive production. It is part of the distributed tracing and OpenTelemetry in Python section, and it assumes the SDK wiring from OpenTelemetry SDK setup.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-http>=1.27.0,<2.0.0"

export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_BSP_MAX_QUEUE_SIZE=2048
export OTEL_BSP_SCHEDULE_DELAY=5000

Every one of those is read by the SDK without a line of code. A service that configures its exporter entirely through environment variables can move between a local Collector, a staging one, and a vendor endpoint with no rebuild.

Concept and architecture

Three components sit between a finished span and a backend, and they fail differently.

The span processor decides when export happens. SimpleSpanProcessor exports synchronously inside span.end(), which means the backend's latency is added to your request. It exists for tests. BatchSpanProcessor puts the span on a bounded queue and returns; a background thread drains it. Every production configuration uses the second one.

The exporter decides how the batch is serialised and sent. The OTLP exporters — gRPC and HTTP/protobuf — are the ones to use; vendor-specific exporters put a backend's identity into your dependency list. The exporter is also where retry lives, and its retry budget is per batch and bounded, because an exporter that retried forever would fill the queue behind it.

The Collector is a separate process that speaks OTLP in and anything out. Running one locally — as a sidecar container or a node-level agent — turns the application's export into a single local network hop that essentially never fails, and moves every policy decision to a config file that operations owns.

The resource attributes attached to every span come from the provider, not the exporter, and they are how a backend knows which service, version and deployment produced a span. Getting them right once at provider construction matters more than any exporter setting.

Step-by-step implementation

Step 1 — Build the provider with a resource and a batch processor.

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "checkout-api",          # or OTEL_SERVICE_NAME
    "service.version": "2026.8.1",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(),                  # endpoint from OTEL_EXPORTER_OTLP_ENDPOINT
        max_queue_size=2048,
        max_export_batch_size=512,
        schedule_delay_millis=5000,
        export_timeout_millis=30000,
    )
)
trace.set_tracer_provider(provider)

Step 2 — Choose the transport for the network you actually have. gRPC on 4317 is more efficient over a long-lived connection. HTTP/protobuf on 4318 goes through proxies and meshes that mishandle HTTP/2. Both are OTLP; the wire format is the same protobuf.

# gRPC
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

# HTTP/protobuf — note the port and that the SDK appends /v1/traces to a base endpoint
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

The path handling is a genuine trap: OTEL_EXPORTER_OTLP_ENDPOINT is a base and the SDK appends /v1/traces, while OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is the full URL and nothing is appended. Setting the second to a base URL produces a 404 that surfaces only in the SDK's own logs.

Step 3 — Run a Collector next to the service. A minimal pipeline: receive OTLP, batch, redact, export.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: db.statement
        action: hash
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlphttp/primary:
    endpoint: https://ingest.example-backend.com
    headers: { authorization: "Bearer ${env:BACKEND_TOKEN}" }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlphttp/primary]

memory_limiter first in the list is deliberate: it is what stops a Collector from becoming the thing that gets OOM-killed on the node when a backend goes slow.

Step 4 — Flush on shutdown. A container that stops without flushing loses whatever is queued. Wire it into the framework's shutdown event and into a signal handler.

import signal
from opentelemetry import trace

def _flush_and_exit(signum, frame):
    provider = trace.get_tracer_provider()
    provider.force_flush(timeout_millis=5000)     # bounded — never hang the shutdown
    provider.shutdown()
    raise SystemExit(0)

signal.signal(signal.SIGTERM, _flush_and_exit)

Configuration reference

Async and concurrency considerations

The exporter runs on its own thread, which is what keeps it off the event loop — but only if the processor is the batch one. Under asyncio, a SimpleSpanProcessor blocks the loop for the duration of every export, so one slow Collector response delays every other request the loop is serving. That is the same failure described for synchronous log handlers in logging from asyncio tasks without blocking, and it has the same shape.

The queue is shared across all threads and the event loop, and it is bounded. When it fills, new spans are dropped — not the oldest, the newest — and the SDK counts them. Export that counter: a service dropping spans silently produces traces with holes in them, which is worse than no traces because the gaps look like the work did not happen.

Under multiprocessing or a prefork server, each worker has its own provider, its own queue, and its own exporter thread. Provider construction must happen after the fork, in the child, or the workers share a queue object that only one of them drains.

Production code examples

A complete, environment-driven setup with a debug switch, a shutdown flush, and the drop counter exported as a metric.

# observability/tracing.py
import os
import signal
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor, ConsoleSpanExporter, SimpleSpanProcessor,
)

def configure_tracing() -> TracerProvider:
    resource = Resource.create({
        "service.name": os.environ.get("OTEL_SERVICE_NAME", "unknown"),
        "service.version": os.environ.get("SERVICE_VERSION", "0"),
        "deployment.environment": os.environ.get("DEPLOY_ENV", "dev"),
    })
    provider = TracerProvider(resource=resource)

    if os.environ.get("OTEL_DEBUG_CONSOLE") == "1":
        # Prove spans exist before debugging why they do not arrive.
        provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(insecure=True),          # a local Collector, plaintext hop
            max_queue_size=int(os.environ.get("OTEL_BSP_MAX_QUEUE_SIZE", "2048")),
            max_export_batch_size=512,
            schedule_delay_millis=5000,
            export_timeout_millis=10000,
        )
    )

    trace.set_tracer_provider(provider)

    def _on_term(signum, frame):
        provider.force_flush(timeout_millis=5000)
        provider.shutdown()
        raise SystemExit(0)

    signal.signal(signal.SIGTERM, _on_term)
    return provider

Verify the producer side first, with the console exporter:

OTEL_DEBUG_CONSOLE=1 python -c "
from observability.tracing import configure_tracing
from opentelemetry import trace
configure_tracing()
with trace.get_tracer(__name__).start_as_current_span('probe') as s:
    s.set_attribute('probe.kind', 'startup')
"

Expected Output:

{
  "name": "probe",
  "context": {"trace_id": "0x4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "0x00f067aa0ba902b7"},
  "parent_id": null,
  "kind": "SpanKind.INTERNAL",
  "attributes": {"probe.kind": "startup"},
  "resource": {"service.name": "checkout-api", "service.version": "2026.8.1"}
}

Spans exist and carry the resource. Anything missing at the backend from here on is a transport or Collector problem, which is a much smaller search space.

Common mistakes

Using SimpleSpanProcessor in production. It exports inside span.end(), so the backend's latency becomes your latency and a Collector outage becomes a request-timeout incident. It belongs in tests and in the console-debug path only.

Putting backend credentials in every service. OTEL_EXPORTER_OTLP_HEADERS with a vendor token in each deployment means rotating that token is a fleet-wide redeploy. Send to a local Collector without credentials and let the Collector hold them.

Confusing the endpoint variables. OTEL_EXPORTER_OTLP_ENDPOINT is a base and gets /v1/traces appended; OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is complete and does not. Setting the latter to a base URL yields 404s visible only in the SDK's internal logs.

No shutdown flush. Every restart loses the queued batch. On a service that deploys several times a day, that is a systematic hole in the data that correlates suspiciously well with deploys.

An unbounded queue, or an enormous one. The queue exists to survive a blip, not an outage. Sizing it to hold ten minutes of spans converts a Collector outage into a memory-pressure incident in the application.

Ignoring the dropped-span counter. A trace missing its middle spans reads as "that service did not do the work". Export the counter and alert on it, so the gap is attributable to telemetry rather than to the system under investigation.

Failure behaviour, decided in advance

Every component on the export path can fail, and the SDK's answers are opinionated: telemetry must never take down the service it observes. Knowing what each failure produces means the behaviour is a decision rather than a surprise.

The Collector is unreachable. The exporter retries the batch with backoff inside its timeout budget, then drops it. The queue behind it keeps accepting spans until it is full, then drops new ones. Nothing blocks, nothing raises, and the application is unaffected — at the cost of a hole in the trace data whose size is the outage duration minus whatever the queue absorbed. That is the correct trade and it is worth stating explicitly to whoever will be looking at the gap later.

The Collector is slow rather than down. Worse, because it is less obvious. Export round trips lengthen, the effective drain rate falls below the production rate, and the queue fills gradually. The symptom is a rising dropped-span count with no error anywhere, which is why that counter deserves an alert.

The backend rejects the data. Authentication failures and malformed payloads are permanent errors: the exporter does not retry them, because retrying cannot help. With a Collector in place this is the Collector's problem and the application never sees it, which is one of the quieter arguments for having one.

The process exits. Whatever is queued is lost unless something flushed it. On a service that deploys several times a day this is a systematic gap correlated with deploys, and it is entirely avoidable with the shutdown hook.

Multi-signal pipelines

The same Collector usually carries traces, metrics and logs, and the three have different tolerances that are worth configuring separately rather than sharing one policy.

Traces are the most tolerant of loss: a sampled system is already discarding most of them by design, so losing a further few during a blip changes nothing statistically. Metrics are the least tolerant in a different way — losing an export interval leaves a visible gap in a graph, and with cumulative temporality the counter recovers on the next successful export, which is why cumulative is the safer choice when the transport is unreliable. Logs sit in between, and are the signal most likely to be subject to a retention requirement that makes silent loss unacceptable.

The practical consequence is that a Collector's sending_queue and retry_on_failure settings deserve per-pipeline values, and that a persistent queue — the file storage extension — is worth its cost for logs and rarely worth it for traces.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/backend]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/logs]        # its own exporter, with a longer retry budget

Debug exporters, and keeping one available

One capability worth building in from the start: a switch that makes the service print its spans locally. ConsoleSpanExporter behind an environment variable costs nothing while it is off and answers the first question of every export investigation — are spans being produced at all — in about ten seconds.

The value is in the ordering it enforces. Without it, a report of missing traces sends someone to check the Collector, the network policy, the endpoint and the credentials, any of which might be at fault and none of which matters if the sampler is set to always_off or the provider was never registered. With it, the producer side is eliminated first and the search space collapses.

Keep it as a SimpleSpanProcessor rather than a batched one, because in a debug session you want the span printed when it ends rather than five seconds later, and pair it with a note in the runbook so the next person knows the switch exists. It is also the fastest way to check what a new instrumentation actually emits before deciding whether to keep it: turn it on locally, exercise one request, and read the attribute names the library chose rather than guessing them from its documentation.

Starting without a Collector

There is a reasonable first step that skips the Collector entirely: point the exporter at the backend directly, confirm spans arrive, and add the Collector when a second reason for it appears. That is a defensible order for a single service with one backend, and the cost of deferring is small because the change is an endpoint and a credential.

The reasons to stop deferring accumulate quickly, though, and it is worth knowing which one you are waiting for. A second backend, or an evaluation of one, is the most common. A redaction requirement that has to apply identically across languages is the most compelling. Credential rotation across a fleet is the most tedious to do without one. Tail sampling requires it outright. And the moment more than a handful of services export directly, the number of places holding a backend credential is itself the argument.

The migration from direct export to a Collector is one environment variable per service, which is why deferring it is low risk — and why teams that defer it often discover they have been carrying the cost of not having one for longer than the change would have taken.

Sizing the deployment

Two numbers drive Collector capacity, and both are easy to measure before you need them. The first is spans per second across everything feeding one instance, which is request rate multiplied by spans per request multiplied by the number of services — a figure that surprises people the first time they compute it. The second is the resource cost per thousand spans per second, which for a Collector doing batching and a couple of attribute processors is modest, and which grows sharply when tail sampling is involved because tail sampling has to hold whole traces in memory until they are complete.

That last point is the practical reason tail sampling belongs in a separate gateway tier: its resource profile is different enough from an agent's that mixing them makes both harder to size.

Related

• Distributed tracing and OpenTelemetry in Python — the parent section: SDK, propagation, span lifecycle and sampling.
• Configuring the OTLP span exporter — endpoints, TLS, headers and retry in detail.
• Tuning BatchSpanProcessor for throughput — sizing the queue and the batch for a real span rate.
• Running the OpenTelemetry Collector for Python services — deployment shapes and pipeline configuration.
• OpenTelemetry SDK setup — the provider and resource this guide exports from.
• Sampling strategies for distributed tracing — reducing volume before it reaches the queue.

Frequently Asked Questions