Distributed Tracing and OpenTelemetry in Python: Architecture and Implementation Guide

This guide establishes a production-first architecture for distributed tracing and OpenTelemetry in Python, targeting backend engineers, SREs, and platform teams who own latency budgets and incident response. It covers foundational telemetry collection, standardized instrumentation, and strategic observability rollout. Engineers will learn to initialize the OpenTelemetry SDK for reliable data pipelines, manage the span lifecycle and attributes to capture meaningful operational signals, implement context propagation and baggage across service boundaries, apply async tracing patterns so coroutines and worker pools stay correlated, and roll all of it out through instrumentation for Python web frameworks. Tracing rarely lives alone — pair it with Python metrics and instrumentation to turn individual slow traces into aggregate latency and error-rate signals, and with structured logging fundamentals so every log line carries the trace it belongs to.

Key architectural principles:

• Adopt W3C Trace Context as the wire standard so traces correlate across polyglot services without bespoke header parsing.
• Implement progressive disclosure: start with auto-instrumentation for breadth, then advance to manual spans around the code that defines your SLOs.
• Decide sampling once, at the edge, and propagate it — a ParentBased root sampler plus tail sampling in the collector beats per-service guesswork.
• Choose a collector topology deliberately (sidecar, daemonset, or gateway pool) before the first service ships, because retrofitting it means touching every deployment manifest.
• Align telemetry pipelines with SRE error budgets and latency objectives so sampling and retention serve debugging, not vanity dashboards.
• Keep observability close to zero-impact through head sampling, async exporters, and a centralized collector that absorbs backpressure.

Foundational Architecture and Telemetry Standards

The OpenTelemetry data model unifies traces, metrics, and logs through shared resource attributes. Each telemetry signal carries a consistent identity built from attributes like service.name, service.version, deployment.environment, and host.id. Because the three signals share that identity, a backend can pivot deterministically from a slow span to the metrics and logs emitted by the same process during the same window. This cross-signal correlation is the entire payoff of standardizing on one SDK rather than stitching together vendor agents.

The architecture separates concerns cleanly into four parts you configure independently. The API surface is what your code and instrumented libraries call; it is intentionally a no-op until an SDK is installed, so importing it can never force a tracing runtime on a consumer. The SDK supplies the real implementation: providers, processors, samplers, and exporters. The wire protocol, OTLP, is a vendor-neutral encoding over gRPC or HTTP that every compliant backend understands. And the collector is a separate process that receives, processes, and routes that data. Because these layers are decoupled, you can change your backend, add a second one, or re-tune sampling without touching application code — the property that makes OpenTelemetry an investment rather than a lock-in.

A trace is a tree of spans. The root span represents the entry point — an inbound HTTP request, a consumed Kafka message, a scheduled job — and every downstream operation becomes a child span carrying the same trace_id but a fresh span_id. The parent-child relationship is reconstructed by the backend from the parent_span_id field, which is why losing context across an await or a thread boundary silently fragments a trace into disconnected pieces. Maintaining that linkage end to end is the single most important correctness property of an instrumentation layer.

Each span carries a name, a start and end timestamp, a kind, a status, a flat set of attributes, an ordered list of timestamped events (used for exceptions and milestones), and optional links to spans in other traces. Attributes are the queryable dimensions, so their cardinality is a design decision rather than an afterthought: bounded values such as HTTP route templates and status codes belong on spans, while unbounded values such as raw user IDs belong in logs that the span links to. Span events, by contrast, are perfect for the things that happen at a point in time within an operation — a cache miss, a retry, a recorded exception — because they carry their own timestamp without inflating attribute cardinality. How to set names, attributes, status, and events correctly is the subject of the guide on span lifecycle and attributes.

Span links deserve a mention because they solve a problem parenting cannot. When one operation is caused by many upstream traces — a batch job that drains fifty queued messages, or a fan-in aggregation step — there is no single parent to point at. A link records the causal relationship to another trace without making the span its child, so the batch keeps one coherent trace of its own while every contributing message trace remains navigable from it. Reach for links whenever the relationship is "these caused me" rather than "this contains me".

Semantic conventions are the other half of the data model, and they are what make telemetry portable between backends. Attribute names such as http.request.method, http.route, url.path, db.system, and messaging.destination.name are specified centrally, so a dashboard or alert written against them keeps working when you swap FastAPI for Django or Postgres for MySQL. The conventions are versioned, and the SDK stamps a schema_url alongside resource attributes so a backend can tell which revision produced a given span. Follow the conventions for anything that has one, and namespace your own attributes under a company prefix (acme.tenant.tier) so they can never collide with a future standard name.

W3C Trace Context compliance ensures interoperability across mixed-language environments. The specification standardizes the traceparent header, which packs the version, trace ID, parent span ID, and sampling flag into a single ASCII string, plus the tracestate header for vendor-specific routing data. Standardizing on these headers means a Python service, a Go gateway, and a Java worker reconstruct one coherent trace without translation shims.

The traceparent value has a fixed shape — 00-<32-hex-trace-id>-<16-hex-span-id>-<2-hex-flags> — and the last byte is the sampling decision. That single bit is what makes consistent head sampling possible across services: a downstream service reads the flag and either records or drops its spans in agreement with the root. The tracestate header is an ordered list of vendor entries that survives alongside traceparent, letting backends carry proprietary correlation data without breaking the standard contract. Because both headers are plain text, they cross HTTP, gRPC, and most message brokers unchanged, which is the practical reason the W3C format displaced the older B3 and Jaeger encodings as the default.

Collector topology directly dictates scalability and data durability. A sidecar collector deployed per pod isolates network overhead and gives each workload a local, low-latency export target; the application's exporter talks to localhost, and the collector owns the noisy, retry-prone connection to the backend. A daemonset consolidates egress at the node level, reducing connection count to the central tier at the cost of coarser per-pod isolation. Direct export from the SDK to a SaaS backend is the simplest topology but sacrifices buffering and tail sampling, so production systems route through at least one collector tier to enforce schema validation, attribute scrubbing, and routing policy before data leaves the VPC.

A common production layout uses two tiers: an agent collector close to the workload (sidecar or daemonset) that does only batching and lightweight processing, feeding a gateway collector pool that owns tail sampling and fan-out to multiple backends. The agent tier keeps the export hop cheap and reliable; the gateway tier centralizes the expensive, stateful decisions in a place you can scale independently of the services. This separation is what lets you change sampling policy or add a new backend without redeploying a single application. One constraint shapes the gateway tier specifically: tail sampling requires every span of a trace to reach the same collector instance, so the gateway pool must be load-balanced by trace ID rather than round-robin, or traces arrive split across replicas and no instance ever sees a complete one.

Resource attributes deserve special discipline because they are the join key for everything downstream. Set them once, at process start, from a combination of code defaults and the OTEL_RESOURCE_ATTRIBUTES environment variable, and resist the temptation to compute them per-request. A stable service.name and service.namespace pair lets the backend group instances into a logical service; service.instance.id distinguishes replicas; deployment.environment keeps staging traffic out of production dashboards. When these are consistent across traces, metrics, and logs, the three signals line up automatically — which is exactly the cross-signal correlation that makes Python metrics and instrumentation worth wiring to the same resource.

Instrumentation Strategy and SDK Configuration

Resource detection must precede provider initialization, because the Resource object is immutable once attached to a TracerProvider. Build it from environment-aware defaults so a single codebase produces different identities in staging and production without code changes. Semantic conventions standardize attribute naming across frameworks, which is what lets a backend recognize that http.request.method means the same thing whether it came from FastAPI, Django, or aiohttp.

There are three layers of instrumentation, and a mature service uses all of them. Zero-code instrumentation via the opentelemetry-instrument launcher patches libraries at startup with no source changes — fast to roll out, but coarse. Library instrumentation packages (the opentelemetry-instrumentation-* contrib distributions) hook specific frameworks, HTTP clients, and database drivers, producing spans that already follow semantic conventions. Manual instrumentation, where you call tracer.start_as_current_span() yourself, is reserved for the business operations that define your SLOs and that no library can name meaningfully on your behalf. Start at the top for breadth, then add manual spans where the auto-generated trace is too shallow to debug an incident.

Span processors dictate export behavior, and the choice between them is the difference between a healthy service and one that stalls under load. The BatchSpanProcessor queues finished spans in memory and flushes them on a background daemon thread, decoupling export latency from request latency. The SimpleSpanProcessor exports synchronously on span end and is strictly for local debugging, because in production it turns every span completion into a blocking network call. The end-to-end mechanics of provider lifecycle, processor tuning, and exporter selection are covered in depth in the OpenTelemetry SDK setup guide.

Environment variables such as OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, and OTEL_RESOURCE_ATTRIBUTES should drive runtime configuration. The SDK reads them automatically, so deployment manifests rather than source code become the source of truth for where telemetry goes. Hardcoded endpoints introduce deployment friction and leak environment topology into your repository; keep configuration immutable at runtime to guarantee pipeline stability across rolling restarts. Two more are worth standardizing early: OTEL_TRACES_SAMPLER with OTEL_TRACES_SAMPLER_ARG lets you change the sampling ratio with a restart instead of a release, and OTEL_SDK_DISABLED=true gives operators a kill switch that turns the whole SDK into a no-op if telemetry ever becomes the incident.

The batch processor's tuning parameters are worth understanding rather than copying blindly, because their interaction determines both memory ceiling and data loss under stress. max_queue_size caps how many finished spans can wait in memory; once full, new spans are dropped rather than blocking the application, which is the correct failure mode but means an undersized queue silently loses data during a spike. max_export_batch_size bounds how many spans go out in one OTLP request, trading request overhead against per-request latency. schedule_delay_millis sets the maximum time a span waits before a flush, so it bounds export staleness. The right values follow from your peak span rate: size the queue to roughly twice the spans you expect in flight, keep batches large enough to amortize the network round trip, and keep the delay short enough that a crash loses only a few seconds of telemetry. The full reference and a worked initialization sequence live in the guide on OpenTelemetry SDK setup.

Shutdown is the half of the provider lifecycle teams forget. Spans sitting in the batch queue when a process exits are simply lost, and those are disproportionately the spans covering the deploy, the crash, or the OOM you are trying to explain. Register a termination hook — a SIGTERM handler, an ASGI lifespan shutdown, or Gunicorn's worker_exit — that calls force_flush() with a bounded timeout and then shutdown() on the provider. Keep the timeout shorter than your orchestrator's grace period so a slow backend delays termination instead of causing a hard kill mid-flush.

In multi-process servers — Gunicorn with several workers, or any pre-fork model — provider initialization must happen after the fork, not before. A provider created in the master process shares its exporter's gRPC connection and background thread across forked children, which produces duplicate spans, deadlocks, and corrupted batches. Use the framework's post-fork hook (Gunicorn's post_fork, or a worker startup callback) so each worker builds its own provider, exporter connection, and batch queue. The same rule applies to any code that spawns processes with multiprocessing: initialize telemetry inside the child, after the fork completes.

When you move from a standalone service to a web application, framework-specific instrumentation becomes the fastest path to coverage. The guide on instrumenting Python web frameworks shows how the contrib packages wrap ASGI and WSGI request handling so every route produces a properly parented server span without touching your handlers. The focused walkthroughs cover the two most common stacks end to end: setting up OpenTelemetry in FastAPI for an async application, and instrumenting Django with OpenTelemetry for a WSGI one where middleware ordering and the ORM instrumentation both matter.

Span kind is part of instrumentation strategy and is easy to get wrong. SERVER and CLIENT spans mark the two ends of a remote call and let the backend reconstruct the network edge between services; PRODUCER and CONSUMER do the same for messaging; INTERNAL is for in-process work. Auto-instrumentation sets kind correctly for the libraries it wraps, but manual spans default to INTERNAL, so set the kind explicitly whenever a span represents a boundary you want drawn on the service map.

Finally, treat instrumentation coverage as something you tune down as well as up. Chatty libraries — health-check endpoints, per-row ORM calls, a Redis client used as a hot cache — can triple span volume without adding diagnostic value. The launcher honours OTEL_PYTHON_DISABLED_INSTRUMENTATIONS for wholesale exclusions, and most contrib packages accept request hooks or URL filters so you can drop /healthz and /metrics spans while keeping the rest of the framework instrumented.

Async and Concurrency Patterns

Python's asyncio relies on contextvars to maintain execution context across coroutine suspension points. The OpenTelemetry SDK stores the active span in a context variable, so within a single coroutine the current span is always correct across await. The trouble starts at boundaries the runtime does not bridge automatically: tasks scheduled with asyncio.create_task, callbacks run via loop.run_in_executor, and threads in a ThreadPoolExecutor do not inherit the caller's context unless you copy it explicitly.

Background work therefore requires deliberate context propagation. Capturing the current context with contextvars.copy_context() and running the spawned callable inside it ensures the child inherits the parent span, so the resulting spans attach to the right trace instead of becoming new roots. Avoid sharing a single mutable span across concurrent tasks; spans are not designed for concurrent mutation, and doing so produces interleaved attributes and races on the end timestamp. For the full set of techniques across task groups, executors, and async database drivers, work through the guide on async tracing patterns.

# Propagate the active span into a thread-pool task safely.
import contextvars
from concurrent.futures import ThreadPoolExecutor
from opentelemetry import trace

tracer = trace.get_tracer(__name__)


def enrich_record(record_id: str) -> str:
    # Runs on a worker thread but stays inside the parent's trace.
    with tracer.start_as_current_span("enrich_record") as span:
        span.set_attribute("record.id", record_id)
        return f"enriched:{record_id}"


def submit_with_context(pool: ThreadPoolExecutor, record_id: str):
    ctx = contextvars.copy_context()           # snapshot the active OTel context
    return pool.submit(ctx.run, enrich_record, record_id)

Expected Output:

# The enrich_record span shares the caller's trace_id and lists the
# caller's span as its parent, instead of starting a disconnected trace.
span name=enrich_record trace_id=4bf92f3577b34da6a3ce929d0e0e4736 parent=00f067aa0ba902b7

The same hazard appears with asyncio.create_task: the task captures the context at creation time, so a span opened after the task is scheduled will not be its parent. Open the parent span first, then create the task inside it, or pass an explicit context. Background workers consuming from a queue have the opposite problem — there is no ambient context to inherit, so each message must reconstruct its context from the propagated headers before any span is opened.

Long-lived concurrency needs one more rule: a span's lifetime should match one logical operation, not one object. Fire-and-forget tasks that outlive the request that spawned them will keep the parent span open if you hold it across the boundary, producing spans whose duration is the lifetime of the task rather than the work. Close the request span when the response is sent and let the background task open its own span linked to the original trace. The same applies to async generators and streaming responses, where the natural instinct is to wrap the whole generator in one span: prefer a span for setup plus events (or child spans) per meaningful chunk, so a stalled stream shows up as a gap you can see rather than one long opaque bar.

Async I/O libraries each have their own context pitfalls, which is why the per-library walkthroughs matter more here than general advice. Two of the most common in production Python are instrumenting aiohttp client requests, where a shared session must not pin a stale context, and tracing SQLAlchemy async queries, where the engine's connection pool runs work on threads the caller never sees. Both are collected alongside task-group and executor recipes in the guide on async tracing patterns.

Network and Protocol Integration

Distributed systems require standardized header propagation on every hop. HTTP/1.1 and HTTP/2 transports carry the W3C traceparent and tracestate headers, which instrumented clients inject on outbound requests and instrumented servers extract on inbound ones. gRPC relies on metadata interceptors to inject and extract the same context during RPC calls, so the trace survives the transition between protocols without manual plumbing.

The injection lifecycle follows a strict order on each service: extract incoming headers into a context, attach that context so new spans parent correctly, do the work, then inject the current context into outgoing requests. Registering the right propagators globally is what makes this automatic — a composite of TraceContextTextMapPropagator and W3CBaggagePropagator covers both trace linkage and the user-defined key/value pairs that ride alongside it. The OTEL_PROPAGATORS environment variable configures the same composite without code, which is the cleanest way to roll a change across a fleet. Legacy systems emitting B3 or Jaeger headers need a translating propagator during migration so old and new services interoperate; register both formats for extraction and emit only W3C on the way out, and the fleet converges without a flag day.

Trust boundaries matter here and are routinely overlooked. An internet-facing service that blindly extracts traceparent from client requests lets anyone inject arbitrary trace IDs, poison your sampling decisions, or graft spans onto someone else's trace. Strip or ignore inbound trace headers at the public edge and start a fresh root span there, then propagate normally between internal services. If you genuinely want client-side traces to join server traces — for real user monitoring — accept the header only from your own instrumented front end and keep the sampling decision server-side.

Message queues are the hardest case because the producer and consumer are separated in time, not just across the network. The fix is to inject the current context into the message envelope — headers on a Kafka record, attributes on an SQS message, a dedicated field in a Celery task payload — and extract it on the consuming side before opening the CONSUMER span. Skipping this step is the most common reason an async pipeline shows two disconnected traces where the engineer expected one; propagating context across Celery tasks is covered specifically in the guide on propagating trace context across Celery tasks. Treat baggage as a small, bounded channel: it is replicated onto every downstream request and is visible to every service in the path, so keep it to a few low-cardinality, non-sensitive keys such as tenant or feature-flag cohort, never tokens or PII. The extraction and injection mechanics, including baggage size limits, are detailed in context propagation and baggage.

Queued work also breaks the assumption that a parent is still running when its child starts. A task that sits in a queue for ten minutes and then executes as a child of a long-finished request span produces a trace whose root duration is meaningless. When the delay is significant, open a new root span for the task and attach a link back to the enqueueing trace, so each trace measures a real unit of work while remaining navigable to the other.

Data Volume Control and Cost Management

Unfiltered telemetry inflates storage cost and query latency faster than most teams expect, because a single busy endpoint can generate tens of thousands of spans per second. Head-based sampling executes at the SDK before export: parent-based sampling honors the upstream decision so a trace is kept or dropped consistently across services, while ratio-based sampling keeps a fixed percentage of new root traces. Rate-limited sampling caps spans per time window to protect the collector during traffic spikes.

The cost model has three independent levers, and conflating them is a frequent mistake. The first is trace count, controlled by head sampling at the edge. The second is spans-per-trace, controlled by how aggressively you instrument — every wrapped library multiplies span volume, so disabling instrumentation for chatty internal calls is often a bigger win than lowering the sample rate. The third is attribute weight, controlled by what you put on each span; a span with twenty string attributes costs far more to store and query than one with five. Tune all three deliberately. A service that drops to a 1% head sample but still attaches request bodies to every span has solved the cheap problem and left the expensive one in place.

A back-of-the-envelope estimate is worth doing before you pick a rate. Multiply requests per second by spans per trace by average serialized span size, then by the sampling ratio, and you have bytes per second leaving the fleet — a service at 500 rps with 25 spans per trace and 1 KB spans produces roughly 12 MB/s unsampled, and 120 KB/s at a 1% head sample. Do that arithmetic per service and the argument about sampling rates stops being a matter of taste. The SDK also enforces hard ceilings you should set explicitly: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, OTEL_SPAN_EVENT_COUNT_LIMIT, and OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT truncate pathological spans before they ever reach the queue, which turns a runaway loop that adds attributes into a bounded cost rather than an outage.

Tail-based sampling operates downstream in the collector, where it can see a complete trace before deciding. This lets you retain 100% of traces that contain an error or exceed a latency threshold while sampling the boring successful traffic aggressively — the hybrid that gives you cheap baseline volume and complete diagnostic fidelity for the traces that matter. The trade-off is that the collector must buffer every span of an in-flight trace until it ends, which costs memory and forces a decision latency window; size the gateway tier for your slowest expected trace, not your median. Attribute-based processors strip high-cardinality or sensitive fields (raw payloads, full URLs with tokens) before storage, which controls both cost and your data-protection surface. The full decision matrix is covered in the guide on sampling strategies for distributed tracing.

The decision that makes head sampling safe across a distributed system is ParentBased: a service honors the sampling flag in the inbound traceparent rather than rolling its own dice. Without it, an upstream service might keep a trace while a downstream one drops it, leaving you with half a trace that is worse than none. Set the root sampling ratio at the edge service, wrap it in ParentBased, and let every downstream service inherit the decision. Reserve absolute volume control for the collector's rate limiter, which protects the pipeline during incidents when error traces — the ones you most want — spike hardest.

Sampling has one consequence that must be designed around rather than tolerated: sampled traces cannot be used to count things. Any question of the form "how many requests failed" has to be answered by a signal that sees every request, which is why pairing trace sampling with aggregate signals from Python metrics and instrumentation is not optional at scale — the metrics still count every request even when only one trace in a hundred is stored. Exemplars close the loop by attaching a trace_id to a sampled histogram observation, so a latency spike on a dashboard links straight to a stored trace that produced it. Logs play the same role at the other end: keeping the trace_id on every record via trace IDs in log records means even a dropped trace leaves an evidence trail you can search.

Production Code Examples

Production-Ready SDK Initialization

This configures deferred export via the batch processor so network I/O never blocks request handling, drives every endpoint from the environment, and flushes cleanly on shutdown so the last few seconds of telemetry survive a deploy or a SIGTERM.

import atexit
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
#   "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"

# 1. Identity shared across traces, metrics, and logs.
resource = Resource.create({
    "service.name": os.getenv("OTEL_SERVICE_NAME", "order-service"),
    "service.version": os.getenv("SERVICE_VERSION", "2.4.1"),
    "deployment.environment": os.getenv("DEPLOY_ENV", "production"),
})

# 2. Provider plus async-safe batch export to the local collector.
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "otel-collector:4317")),
    max_queue_size=2048,
    max_export_batch_size=512,
    schedule_delay_millis=5000,
))
trace.set_tracer_provider(provider)


# 3. Drain the queue on termination; keep the timeout under the grace period.
@atexit.register
def _flush_telemetry() -> None:
    provider.force_flush(timeout_millis=5000)
    provider.shutdown()

Expected Output: No console output. Spans queue in memory and flush asynchronously to the OTLP endpoint on a background daemon thread, so the event loop and request path stay unblocked during export. On exit, the buffered spans are pushed before the process terminates instead of being dropped.

Manual Span Creation with Error Handling

This shows explicit span boundaries, attribute enrichment, and standardized error recording so failed operations are visible to SLO tracking.

import asyncio
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)


async def process_order(order_id: str) -> dict:
    # start_as_current_span attaches the span to the active context,
    # so any child operations parent correctly across awaits.
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        try:
            await asyncio.sleep(0.05)               # stand-in for real I/O
            return {"status": "completed", "order_id": order_id}
        except Exception as exc:
            span.record_exception(exc)              # attach stack trace as a span event
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise

Expected Output: Returns {"status": "completed", "order_id": "ORD-123"} on success. On failure the span status becomes ERROR, the exception is recorded as a span event with its stack trace, and the span still closes cleanly so the trace is never left open.

Correlating Logs with the Active Trace

This injects the active trace_id and span_id into every log record so a slow span links directly to the lines it emitted, the cross-signal correlation the unified resource model is designed for.

import logging
from opentelemetry import trace
# pip install "opentelemetry-sdk>=1.30.0,<2.0.0"


class TraceContextFilter(logging.Filter):
    # Stamp each record with the current span's identifiers.
    def filter(self, record: logging.LogRecord) -> bool:
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, "032x") if ctx.is_valid else "-"
        record.span_id = format(ctx.span_id, "016x") if ctx.is_valid else "-"
        return True


handler = logging.StreamHandler()
handler.addFilter(TraceContextFilter())
handler.setFormatter(logging.Formatter(
    "%(asctime)s %(levelname)s trace_id=%(trace_id)s span_id=%(span_id)s %(message)s"
))
logging.basicConfig(level=logging.INFO, handlers=[handler])

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("checkout"):
    logging.getLogger("shop").info("charged card")

Expected Output:

2026-06-19 10:15:02,481 INFO trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7 charged card

Common Mistakes

• Using synchronous exporters in high-throughput async applications: The SimpleSpanProcessor exports on every span end, turning each completion into a blocking network call that stalls the event loop and exhausts worker threads under load. Use the BatchSpanProcessor everywhere except local debugging.
• Omitting resource attributes during SDK initialization: Without service.name, service.version, and deployment.environment, traces collapse into an unknown_service bucket, breaking service maps and cross-environment filtering. Build the Resource explicitly before constructing the provider.
• Over-instrumenting with high-cardinality span attributes: Putting user IDs, raw request bodies, or UUIDs on every span explodes storage and slows queries. Keep unbounded values in logs and reserve span attributes for bounded, queryable dimensions.
• Failing to propagate context across thread or process boundaries: Executors, multiprocessing, and background tasks do not inherit the active context automatically, so spans fragment into orphan roots. Copy the context with contextvars.copy_context() before scheduling work.
• Initializing the provider after instrumentation attaches: If auto-instrumentation registers before set_tracer_provider, spans go to the default no-op provider and vanish. Bootstrap the SDK at the very start of process startup, before any framework hooks run.
• Treating the collector as optional in production: Exporting straight to a SaaS backend gives up retries, tail sampling, and scrubbing, so a backend outage becomes data loss. Route through at least one collector tier.
• Mixing sampling decisions across services without ParentBased: When each service samples independently, a trace can be kept upstream and dropped downstream, leaving you with partial traces that are harder to reason about than complete ones. Wrap the root sampler in ParentBased so the edge decision propagates.
• Never flushing on shutdown: A process that exits without calling force_flush drops whatever spans are still buffered, so the requests immediately before a deploy or crash vanish — exactly the ones you often need. Flush and shut down the provider in a graceful-termination hook.
• Load-balancing a gateway collector round-robin while running tail sampling: Spans from one trace land on different replicas, no replica ever assembles the whole trace, and the sampling policy silently makes decisions on fragments. Route by trace ID instead.

Taken together, these decisions form a single pipeline: a correctly initialized SDK produces well-named spans with bounded attributes, a global propagator keeps them linked across every protocol and queue, head sampling decided once at the edge controls volume, and a collector tier enforces the expensive policies before data lands in a backend. Each guide in this section drills into one stage of that pipeline — provider setup, span lifecycle, propagation, async patterns, and framework integration — and they are designed to compose into a coherent, low-overhead observability layer for production Python services.

Related Reading

• OpenTelemetry SDK Setup for Python — provider lifecycle, processors, exporters, and environment-driven configuration.
• Span Lifecycle and Attributes — naming, kinds, status, events, and attribute cardinality discipline.
• Context Propagation and Baggage — extract/inject mechanics across HTTP, gRPC, and message brokers.
• Async Tracing Patterns in Python — contextvars, task groups, executors, and async clients.
• Instrumenting Python Web Frameworks — ASGI and WSGI coverage for FastAPI, Django, and Flask.
• Exporters and the OpenTelemetry Collector — how spans leave the process, and why a Collector belongs between you and any backend.
• Correlating Logs, Traces and Metrics — the shared resource, the trace IDs on log records, and the exemplars that link a metric to a request.
• Python Metrics and Instrumentation and Python Logging Fundamentals — the two signals that pair with tracing, both reachable from the Python observability guides.

Frequently Asked Questions