Python Observability

Configuring the OTLP Span Exporter in Python

Most "OpenTelemetry is not working" investigations end at one of four things: the wrong exporter package, the wrong endpoint variable, a TLS mismatch, or a timeout that hides the real error. This page covers each of them, and the diagnostics that turn a silent failure into a status code. It builds on exporters and the OpenTelemetry Collector, part of the distributed tracing and OpenTelemetry in Python section.

Two endpoint variables, one of which appends a path The two OTLP endpoint environment variables compared. Setting the base variable, OTEL_EXPORTER_OTLP_ENDPOINT, to a host and port produces a final URL with the signal path appended by the SDK — slash v1 slash traces for spans, slash v1 slash metrics for metrics — which is what a Collector expects. Setting the signal-specific variable, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, is treated as the complete URL and nothing is appended, so it must already include the path. The failure case shown is the combination people actually write: the signal-specific variable set to a bare host and port, producing a request to the root path, which every Collector answers with 404 — and because the exporter's failure is logged only through the SDK's internal logger, the symptom is silence rather than an error. For gRPC the distinction is invisible, since the path is part of the protocol rather than the URL, which is why the same configuration can work on gRPC and fail on HTTP. the same value, two variables, two different URLs OTEL_EXPORTER_OTLP_ENDPOINT = http://collector:4318 the SDK appends the signal path → http://collector:4318/v1/traces a base URL, shared by every signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = http://collector:4318/v1/traces used exactly as given → http://collector:4318/v1/traces a complete URL, for this signal only OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = http://collector:4318 used exactly as given → http://collector:4318/ → 404, silently and on gRPC the same configuration works, because the path is part of the protocol rather than the URL
The third row is the one people write. It works on gRPC and 404s on HTTP, which is why "we switched transports and traces stopped" is such a common report.

Prerequisites

# gRPC on 4317
pip install "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

# or HTTP/protobuf on 4318
pip install "opentelemetry-exporter-otlp-proto-http>=1.27.0,<2.0.0"

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_INSECURE=true
export OTEL_EXPORTER_OTLP_TIMEOUT=10000

The two exporter packages are separate distributions with separate import paths. OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf with only the gRPC package installed raises at startup rather than falling back.

Implementation

Step 1 — Construct the exporter from the environment. With the variables set, the constructor takes no arguments at all — which is the configuration you want, because it moves every endpoint decision out of the image.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter()          # endpoint, protocol, timeout, headers: all from env

Explicit arguments override the environment, which makes them useful for tests and harmful in production images:

exporter = OTLPSpanExporter(
    endpoint="http://otel-collector:4317",
    insecure=True,                      # plaintext to a local Collector
    timeout=10,                         # seconds, not milliseconds, in the constructor
)

Note the unit mismatch: OTEL_EXPORTER_OTLP_TIMEOUT is milliseconds, the constructor's timeout is seconds. Passing 10000 to the constructor sets a timeout of nearly three hours.

Step 2 — Choose the transport and set the matching endpoint.

# gRPC — the path is part of the protocol, so the endpoint is host:port
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

# HTTP/protobuf — the SDK appends /v1/traces to this base
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

gRPC is more efficient over a persistent connection and is the default. HTTP/protobuf survives proxies, meshes and load balancers that mishandle HTTP/2 trailers. Neither is more correct; pick the one your network is friendly to, and be prepared to switch.

Step 3 — Keep TLS and credentials at the Collector. A sidecar or node-local Collector is one hop that never leaves the host, so plaintext is appropriate and keeps certificate rotation out of the application.

exporter = OTLPSpanExporter(insecure=True)      # to a local Collector

When a service genuinely must reach a remote endpoint directly, configure TLS explicitly rather than disabling verification:

export OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/ssl/certs/collector-ca.pem
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${INGEST_TOKEN}"

OTEL_EXPORTER_OTLP_HEADERS is comma-separated key=value pairs, and it is where a vendor token ends up in every service's environment — which is the argument for the Collector, restated as an operational cost.

What the exporter retries, and for how long The exporter's response handling split into three outcomes. A success acknowledges the batch and the queue moves on. A transient failure — unavailable, deadline exceeded, resource exhausted, or an HTTP 429 or 503 — is retried with exponential backoff, and the retries continue until the export timeout budget for that batch is exhausted; the diagram shows three attempts with widening gaps inside a ten-second budget. A permanent failure — an authentication error, a malformed request, an unimplemented endpoint — is not retried at all, because no number of attempts can change the outcome, and the batch is dropped immediately. When the budget runs out on a transient failure, the batch is also dropped, and in both drop cases the spans are gone: nothing is written to disk and nothing is re-queued. The practical note is that the default thirty-second timeout means a single stalled batch occupies the exporter while the bounded queue behind it fills. one batch, one response, three outcomes export(batch) 512 spans OK acknowledged, queue advances transient UNAVAILABLE · 429 · 503 retried with backoff permanent UNAUTHENTICATED · 400 · 404 dropped at once — no retry inside the export timeout budget …then the batch is dropped attempts widen: 1s, 2s, 4s… dropped means gone nothing is written to disk, nothing re-queued the 30 s default holds the exporter
The default thirty-second timeout is the setting worth changing. One stalled batch occupies the exporter for half a minute while the queue behind it runs out of room.

Step 4 — Fail fast. Lower the export timeout so a stalled backend cannot hold the exporter while the queue fills.

export OTEL_EXPORTER_OTLP_TIMEOUT=10000       # ms — down from the 30 s default
export OTEL_BSP_EXPORT_TIMEOUT=10000          # the processor's own budget

Step 5 — Turn on diagnostics while proving the path. The SDK logs through the opentelemetry logger namespace, which most configurations leave at WARNING or suppress entirely.

import logging

logging.getLogger("opentelemetry").setLevel(logging.DEBUG)

export OTEL_PYTHON_LOG_CORRELATION=true       # unrelated, but usually wanted alongside
gRPC or HTTP protobuf — four things that decide it The two OTLP transports compared on four practical properties. On efficiency, gRPC multiplexes over one long-lived HTTP/2 connection and is measurably cheaper per batch, while HTTP protobuf opens a request per batch and is slightly heavier but still negligible next to the work being measured. On network tolerance, gRPC depends on HTTP/2 features such as trailers that some proxies, load balancers and corporate middleboxes mishandle, while HTTP protobuf is an ordinary POST that traverses almost anything. On ports, gRPC uses 4317 and HTTP uses 4318, and mixing them is a common misconfiguration that presents as a connection error rather than a protocol one. On failure presentation, gRPC returns status codes such as UNAVAILABLE and UNIMPLEMENTED that name the problem precisely, while HTTP returns familiar codes where a 404 usually means the path is wrong rather than the service being absent. The summary given is to use gRPC where it already works and switch to HTTP the moment the network is the thing fighting you. the same protobuf payload, two ways to carry it gRPC · 4317 HTTP protobuf · 4318 efficiency one long-lived HTTP/2 connection a request per batch — still negligible networks needs HTTP/2 trailers to survive an ordinary POST — traverses anything failures UNAVAILABLE · UNIMPLEMENTED 404 usually means the path, not the host use gRPC where it already works · switch to HTTP the moment the network is the thing fighting you, not after a week of packet captures
Neither is more correct. The decision is entirely about what your network does to HTTP/2, and it is cheaper to switch than to prove.

Configuration options

Option Env var Constructor Default Recommended
Endpoint (base) OTEL_EXPORTER_OTLP_ENDPOINT endpoint= localhost:4317 the local Collector
Endpoint (signal) OTEL_EXPORTER_OTLP_TRACES_ENDPOINT — unset leave unset unless the path differs
Protocol OTEL_EXPORTER_OTLP_PROTOCOL — grpc match the installed package
Insecure OTEL_EXPORTER_OTLP_INSECURE insecure= false true to a local Collector
Certificate OTEL_EXPORTER_OTLP_CERTIFICATE credentials= system store only for direct remote export
Headers OTEL_EXPORTER_OTLP_HEADERS headers= none none — keep tokens in the Collector
Timeout OTEL_EXPORTER_OTLP_TIMEOUT (ms) timeout= (s) 10 s 10 s — mind the unit difference
Compression OTEL_EXPORTER_OTLP_COMPRESSION compression= none gzip for a remote endpoint

Verification

Start a Collector with a debug exporter, send one span, and read both ends.

docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "$PWD/otel-collector.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest

import logging
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

logging.basicConfig(level=logging.DEBUG)
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(insecure=True)))
trace.set_tracer_provider(provider)

with trace.get_tracer("probe").start_as_current_span("export-check") as span:
    span.set_attribute("probe", True)

provider.force_flush()          # do not wait for the schedule delay
provider.shutdown()

Expected Output (Collector side):

Trace ID       : 4bf92f3577b34da6a3ce929d0e0e4736
Span ID        : 00f067aa0ba902b7
Name           : export-check
Kind           : Internal
Attributes     : probe: true

The force_flush() matters in a probe script: without it the process exits before the five-second schedule delay elapses and nothing is sent, which reads exactly like a broken endpoint.

Failure looks like this, and is the reason to enable the SDK logger:

Expected Output (failure):

DEBUG opentelemetry.exporter.otlp.proto.grpc.exporter Waiting 1s before retrying export of span
DEBUG opentelemetry.exporter.otlp.proto.grpc.exporter Waiting 2s before retrying export of span
WARNING opentelemetry.exporter.otlp.proto.grpc.exporter Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 4s

UNAVAILABLE is a connection problem — wrong port, Collector down, network policy. UNIMPLEMENTED means you reached something that is not an OTLP receiver. A 404 on HTTP means the path is wrong.

Common mistakes

Spans stop when switching from gRPC to HTTP

Error signature: everything worked on 4317; on 4318 the backend receives nothing and the service logs nothing. Root cause: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT was set to a bare host and port, which gRPC ignores and HTTP turns into a request for the root path. Remediation: use the base OTEL_EXPORTER_OTLP_ENDPOINT and let the SDK append /v1/traces.

The timeout is three hours

Error signature: a hung export never times out and the queue fills. Root cause: the constructor's timeout is in seconds and was given a millisecond value. Remediation: set the timeout through the environment variable, which is unambiguously milliseconds, or pass seconds to the constructor.

ImportError at startup after changing the protocol

Error signature: ModuleNotFoundError: opentelemetry.exporter.otlp.proto.http. Root cause: the protocol variable changed but the matching exporter package is not installed. Remediation: install the package for the transport you are using, or the opentelemetry-exporter-otlp meta-package that includes both.

Configuration precedence, and why it matters

Four layers can set the same value, and knowing the order saves the specific confusion of a setting that is present in the deployment and demonstrably not in effect.

A constructor argument wins over everything. Below it, the signal-specific environment variable — OTEL_EXPORTER_OTLP_TRACES_ENDPOINT — wins over the general one. Below that, the general variable, OTEL_EXPORTER_OTLP_ENDPOINT. And below everything, the SDK's default of localhost:4317.

The practical consequence is that a hard-coded constructor argument silently overrides the environment, which is fine in a test and a problem in an image that ships to three environments. A useful rule is that production code passes nothing to the constructor at all: every value comes from the environment, so the deployment manifest is the single description of where telemetry goes.

# in tests, or a script
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)

# in the service
exporter = OTLPSpanExporter()          # everything from the environment
Layer Example Wins over
Constructor argument OTLPSpanExporter(endpoint=…) everything
Signal-specific env OTEL_EXPORTER_OTLP_TRACES_ENDPOINT the general env var
General env OTEL_EXPORTER_OTLP_ENDPOINT the default
SDK default localhost:4317 —

Compression, and when it pays

OTEL_EXPORTER_OTLP_COMPRESSION=gzip reduces payload size substantially — span batches are repetitive protobuf and compress well — at the cost of CPU on the exporter thread. To a Collector on the same host that trade is usually not worth making: the network is a loopback interface and the CPU is real. To a remote endpoint across a network you pay for, it usually is.

That is the same reasoning that puts the Collector next to the service in the first place: the application's hop should be cheap and local, and the expensive, compressed, authenticated hop should happen once from the Collector rather than once per service replica.

Running two exporters

Occasionally a migration needs spans in two places at once — an incumbent backend and a candidate. Two BatchSpanProcessor instances on the same provider does this, each with its own exporter and its own queue.

provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=INCUMBENT)))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=CANDIDATE)))

Both processors see every span, so the cost is two serialisations and two queues, and a failure in one does not affect the other. It works and it is the wrong place to do it in most cases: a Collector with two exporters achieves the same fan-out with one hop from the application, one serialisation, and a change that does not require a redeploy to reverse. Keep the two-processor form for the case where the Collector itself is what you are evaluating.

The diagnostics worth leaving on

The SDK's internal logger is noisy at DEBUG and useful at WARNING, which is a reasonable permanent setting: it stays quiet while exports succeed and produces a record naming the status when they do not. That single line is what turns "no traces" from a search into a diagnosis, and it costs nothing while everything is working.

logging.getLogger("opentelemetry").setLevel(logging.WARNING)

Frequently Asked Questions

Which OTLP exporter package do I install?

opentelemetry-exporter-otlp-proto-grpc for gRPC on port 4317, or opentelemetry-exporter-otlp-proto-http for HTTP protobuf on 4318. There is also an opentelemetry-exporter-otlp meta-package that pulls in both. They are separate import paths, so the OTEL_EXPORTER_OTLP_PROTOCOL variable only takes effect if the corresponding package is installed — otherwise you get an ImportError at startup.

Why does my HTTP exporter return 404?

Because of the two endpoint variables. OTEL_EXPORTER_OTLP_ENDPOINT is a base URL and the SDK appends /v1/traces to it. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is the complete URL and nothing is appended. Setting the second one to a base like http://collector:4318 sends traces to the root path, which every Collector answers with 404.

Do I need TLS between my service and the Collector?

Not when the Collector is a sidecar or a node-local agent — that traffic never leaves the host or the pod, and plaintext keeps the service free of certificate management. TLS belongs on the hop from the Collector to the backend, where it crosses a network you do not control, and that is a Collector configuration rather than an application one.

How does the exporter retry?

Per batch, with exponential backoff, for a bounded period governed by the export timeout. It retries on transient conditions — unavailable, deadline exceeded, resource exhausted — and gives up immediately on permanent ones such as an authentication failure, since retrying those cannot succeed. When it gives up, the batch is dropped and counted; nothing is written to disk.

Can I see why exports are failing?

Yes — the SDK logs through Python's logging module under the opentelemetry namespace, and those records are suppressed by default in most configurations. Set that logger to DEBUG while you are proving the path works, and you get the status code and message for each failed export instead of silence.

Related Guides