Instrumenting Python Web Frameworks with OpenTelemetry
Auto-instrumentation is the fastest path from a blank trace view to a populated service map: a single instrumentor call wraps every inbound request in a span carrying the route template, HTTP method, and status code. This guide details how the opentelemetry-instrumentation-* contrib packages hook FastAPI, Starlette, Django, Flask, and raw WSGI/ASGI applications, how server spans differ from client spans, and how to constrain span volume with excluded_urls before traces overwhelm your collector. It is part of the Distributed Tracing and OpenTelemetry in Python guide and assumes you have already completed the OpenTelemetry SDK setup for Python. For framework-specific deep dives, see setting up OpenTelemetry in FastAPI and the dedicated walkthrough for instrumenting Django with OpenTelemetry.
Prerequisites
Auto-instrumentation packages depend on a configured SDK. Pin the API, SDK, exporter, and the framework instrumentor together so a contrib release never drags in an incompatible core. Contrib packages version on the 0.x pre-release track and must align with the 1.x SDK release they were built against.
# Core pipeline (shared across every framework)
pip install "opentelemetry-api>=1.30.0,<2.0.0" \
"opentelemetry-sdk>=1.30.0,<2.0.0" \
"opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"
# Framework instrumentors — install only what you run
pip install "opentelemetry-instrumentation-fastapi>=0.51b0,<1.0.0" # FastAPI / Starlette
pip install "opentelemetry-instrumentation-django>=0.51b0,<1.0.0" # Django
pip install "opentelemetry-instrumentation-flask>=0.51b0,<1.0.0" # Flask
pip install "opentelemetry-instrumentation-wsgi>=0.51b0,<1.0.0" # raw WSGI
pip install "opentelemetry-instrumentation-asgi>=0.51b0,<1.0.0" # raw ASGI
# Client-side instrumentors for outbound CLIENT spans
pip install "opentelemetry-instrumentation-requests>=0.51b0,<1.0.0" \
"opentelemetry-instrumentation-httpx>=0.51b0,<1.0.0"
Set the exporter target through environment variables so the same image runs in every environment:
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_SERVICE_NAME="checkout-api"
export OTEL_PYTHON_EXCLUDED_URLS="healthz,readyz,metrics"
Concept & Architecture
Every framework instrumentor works the same way: it injects a middleware (WSGI or ASGI) at the outermost layer of the request stack. When a request arrives, that middleware extracts incoming W3C trace headers, starts a SERVER span, sets it as the active context for the duration of the request, and closes the span when the response is written. Because the span is the active context, anything you trace inside the view — a manual start_as_current_span, a database query, an outbound HTTP call — automatically becomes a child without explicit wiring.
The span kind matters for backend topology. A SERVER span tells the backend "this service received a request," while a CLIENT span on the calling side tells it "this service sent a request." A backend stitches the two halves of a network hop together by matching the traceparent header that the client instrumentor injects and the server instrumentor extracts. This is the same mechanism described in context propagation and baggage: the instrumentor does extraction and injection for you, but it relies on the global propagators you registered during SDK setup.
Auto-instrumentation captures three high-value attributes by default: the route template (for example /orders/{order_id} rather than /orders/4815), the HTTP method, and the response status code. Using the route template instead of the resolved URL is the single most important behavior for cardinality control — it keeps span names bounded regardless of how many distinct IDs hit the endpoint. The trade-off is that the template only becomes available once the framework's router has matched the request, which is why instrumentor ordering matters.
WSGI and ASGI are the two substrates beneath these frameworks. Flask and classic Django run on WSGI, a synchronous request/response protocol. FastAPI, Starlette, and ASGI-mode Django run on ASGI, the async protocol. The opentelemetry-instrumentation-wsgi and opentelemetry-instrumentation-asgi packages provide the low-level middleware; the framework-specific packages are thin wrappers that know how to extract the route template and integrate with the framework's lifecycle.
Two hooks let you shape every captured span without touching view code. A server_request_hook fires once per inbound request with the freshly created server span and the framework's request object, which is the right place to copy a tenant identifier, a feature-flag cohort, or an authenticated user id onto the span. A client_request_hook and client_response_hook fire for the nested send/receive events that ASGI exposes, letting you annotate streaming responses or websocket frames. Hooks run synchronously inside the request path, so keep them allocation-light: read an attribute, set it on the span, return. Heavy work in a hook becomes per-request latency on every traced route.
Header capture deserves a deliberate policy rather than a blanket allow-list. The OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST and ..._SERVER_RESPONSE variables record named headers as span attributes, which is invaluable for debugging routing and content negotiation. But headers frequently carry secrets, so always pair capture with OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS set to at least authorization and cookie. Sanitized values are replaced with a redaction marker before export, so the attribute key still appears in traces while the secret never leaves the process.
Step-by-Step Implementation
Step 1 — Bootstrap the SDK first
Provider initialization must happen before any instrumentor attaches, exactly as covered in the OpenTelemetry SDK setup. The snippet below is the shared bootstrap every framework example reuses.
# otel_bootstrap.py — import this once at process start
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
def init_tracing() -> None:
resource = Resource.create({
ResourceAttributes.SERVICE_NAME: os.getenv("OTEL_SERVICE_NAME", "checkout-api"),
ResourceAttributes.SERVICE_VERSION: os.getenv("SERVICE_VERSION", "1.0.0"),
ResourceAttributes.DEPLOYMENT_ENVIRONMENT: os.getenv("DEPLOYMENT_ENV", "production"),
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) # endpoint via env var
trace.set_tracer_provider(provider)
Step 2 — Instrument FastAPI and Starlette
The FastAPI instrumentor accepts the application instance. Pass excluded_urls to skip probes and server_request_hook to enrich the span with request-specific attributes. Starlette uses the identical pattern via StarletteInstrumentor.
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from otel_bootstrap import init_tracing
init_tracing() # SDK before instrumentation
app = FastAPI()
def server_request_hook(span, scope):
# Attach a tenant attribute pulled from the ASGI scope headers
if span and span.is_recording():
headers = dict(scope.get("headers") or [])
tenant = headers.get(b"x-tenant-id", b"unknown").decode()
span.set_attribute("tenant.id", tenant)
FastAPIInstrumentor.instrument_app(
app,
excluded_urls="healthz,readyz",
server_request_hook=server_request_hook,
)
@app.get("/orders/{order_id}")
async def get_order(order_id: str):
return {"order_id": order_id}
Step 3 — Instrument Flask
The Flask instrumentor wraps the Flask app object. It records the route rule (/orders/<order_id>) as the span name. Capture extra request headers with OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST.
from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from otel_bootstrap import init_tracing
init_tracing()
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app, excluded_urls="healthz,readyz")
@app.get("/orders/<order_id>")
def get_order(order_id):
return {"order_id": order_id}
Step 4 — Instrument Django
Django is configured globally rather than per-app because the framework owns its own settings and middleware registry. Call DjangoInstrumentor().instrument() from a startup hook. The full middleware-ordering and database-instrumentation details live in the dedicated Django and OpenTelemetry guide.
from opentelemetry.instrumentation.django import DjangoInstrumentor
from otel_bootstrap import init_tracing
init_tracing()
# Reads DJANGO_SETTINGS_MODULE and injects OpenTelemetryMiddleware
DjangoInstrumentor().instrument(is_sql_commentor_enabled=True)
Step 5 — Instrument a raw WSGI or ASGI app
When you run a framework with no dedicated instrumentor, wrap the application object directly with the WSGI or ASGI middleware. This produces server spans but cannot resolve route templates, since the generic middleware has no router to consult.
# Raw ASGI middleware wrapping any ASGI callable
from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware
from otel_bootstrap import init_tracing
init_tracing()
from my_app import application # any ASGI app
application = OpenTelemetryMiddleware(application) # adds SERVER spans
Step 6 — Add client instrumentation for outbound spans
Framework instrumentors only cover inbound traffic. To turn outbound calls into CLIENT spans that join the same trace, install and activate the matching client instrumentor once at startup.
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
RequestsInstrumentor().instrument() # requests -> CLIENT spans
HTTPXClientInstrumentor().instrument() # httpx -> CLIENT spans
Configuration Reference
| Option / Environment variable | Scope | Effect |
|---|---|---|
excluded_urls (kwarg) / OTEL_PYTHON_EXCLUDED_URLS |
All HTTP instrumentors | Comma-separated path patterns that are never traced. Use for healthz, readyz, metrics. |
server_request_hook |
FastAPI, Starlette, Django, Flask | Callback (span, scope/environ) to add attributes from the inbound request. |
client_request_hook |
FastAPI, Starlette, ASGI | Callback fired for nested ASGI send/receive events to enrich sub-spans. |
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST |
All HTTP instrumentors | Comma-separated request header names to record as http.request.header.* attributes. |
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE |
All HTTP instrumentors | Response header names to record as http.response.header.* attributes. |
OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS |
All HTTP instrumentors | Header names whose values are redacted before export (set for authorization, cookie). |
is_sql_commentor_enabled (kwarg) |
Django | Appends the active trace context as a SQL comment for database-side correlation. |
meter_provider / tracer_provider (kwarg) |
All instrumentors | Override the global provider; useful in tests to isolate state. |
Async & Concurrency Considerations
ASGI frameworks run on an event loop, so the request span lives in a contextvars context that the SDK attaches and detaches automatically across await boundaries. The FastAPI and Starlette instrumentors are built on this guarantee: an async def view that awaits a database driver or an httpx call keeps the server span active for the whole coroutine, so child spans nest correctly without manual context copying.
The hazard appears when you spawn background work. Calling asyncio.create_task or handing work to a thread pool does not propagate the active span unless the new task inherits the context. Copy the context explicitly with contextvars.copy_context() before scheduling, or use asyncio.create_task inside the request scope so it captures the current context at creation time. Fire-and-forget tasks created after the response is sent will start a fresh, parentless trace. These patterns are covered in depth in async tracing patterns.
For WSGI frameworks the model is simpler: each request owns a worker thread, and the instrumentor stores the span in thread-local context. The risk there is thread pools inside a view — an executor.submit call runs on a worker that has no active span, so wrap submitted callables to re-attach context if you need their spans to nest.
A subtle but common failure is mixing sync and async carelessly. Calling a blocking driver inside an async def view stalls the event loop, and while the span still records correctly, the latency it captures will be dominated by event-loop starvation rather than the operation itself, producing misleading traces. Run blocking work through asyncio.to_thread or a properly instrumented async driver so the recorded duration reflects real I/O. Likewise, the FastAPI and Starlette instrumentors emit one server span per request, but middleware that swallows exceptions can leave the span status as unset; attach a server_request_hook or rely on the framework's exception handlers so failed requests are marked with an error status code rather than appearing as silent successes.
Production Code Examples
End-to-end: FastAPI service with manual spans and an outbound call
This service combines auto-instrumentation, a client instrumentor, and a hand-written child span. The manual span nests under the auto-generated server span with zero extra context plumbing.
import httpx
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from otel_bootstrap import init_tracing
init_tracing()
HTTPXClientInstrumentor().instrument() # outbound CLIENT spans
app = FastAPI()
FastAPIInstrumentor.instrument_app(app, excluded_urls="healthz")
tracer = trace.get_tracer(__name__)
client = httpx.AsyncClient(base_url="http://pricing-svc:8080")
@app.get("/orders/{order_id}/total")
async def order_total(order_id: str):
# Manual INTERNAL span nests under the FastAPI SERVER span automatically
with tracer.start_as_current_span("compute_total") as span:
span.set_attribute("order.id", order_id)
resp = await client.get(f"/price/{order_id}") # becomes a CLIENT span
return {"order_id": order_id, "total": resp.json()["amount"]}
Expected Output:
{
"resourceSpans": [{
"resource": {"attributes": [
{"key": "service.name", "value": {"stringValue": "checkout-api"}}
]},
"scopeSpans": [{
"spans": [
{
"name": "GET /orders/{order_id}/total",
"kind": "SPAN_KIND_SERVER",
"attributes": [
{"key": "http.request.method", "value": {"stringValue": "GET"}},
{"key": "http.route", "value": {"stringValue": "/orders/{order_id}/total"}},
{"key": "http.response.status_code", "value": {"intValue": "200"}}
]
},
{"name": "compute_total", "kind": "SPAN_KIND_INTERNAL"},
{
"name": "GET",
"kind": "SPAN_KIND_CLIENT",
"attributes": [
{"key": "http.request.method", "value": {"stringValue": "GET"}},
{"key": "server.address", "value": {"stringValue": "pricing-svc"}}
]
}
]
}]
}]
}
All three spans share one trace_id; compute_total and the client span list the server span's span_id as their parent_span_id.
End-to-end: Flask service with header capture and exclusions
import os
os.environ["OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST"] = "x-tenant-id"
os.environ["OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS"] = "authorization,cookie"
from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from otel_bootstrap import init_tracing
init_tracing()
RequestsInstrumentor().instrument()
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app, excluded_urls="healthz,metrics")
@app.get("/orders/<order_id>")
def get_order(order_id):
return {"order_id": order_id}
Expected Output:
# A request carrying `x-tenant-id: acme` produces a server span with:
name: GET /orders/<order_id>
kind: SPAN_KIND_SERVER
attributes:
http.request.method = GET
http.route = /orders/<order_id>
http.response.status_code = 200
http.request.header.x_tenant_id = ["acme"]
# Requests to /healthz and /metrics produce no spans.
Common Mistakes
Instrumenting after the app is already serving. Error signature: server spans appear for some routes but not others, or stop entirely after a reload. Root cause: the instrumentor must inject its middleware before the first request is handled; attaching it lazily in a request handler or after app.run() misses the middleware stack. Remediation: call the instrumentor at module import time, immediately after init_tracing() and before the server binds.
Expecting outbound calls to appear without a client instrumentor. Error signature: server spans are present but downstream services start brand-new traces, breaking the service map. Root cause: framework instrumentors only cover inbound requests; outbound requests/httpx/aiohttp calls are untraced until their own instrumentor is active. Remediation: call RequestsInstrumentor().instrument() (and the matching client instrumentors) at startup, and confirm the propagators are registered as in the SDK setup.
Letting health checks dominate trace volume. Error signature: the backend is flooded with thousands of identical /healthz spans, inflating cost and burying real traffic. Root cause: probes hit the service every few seconds and are traced by default. Remediation: set excluded_urls or OTEL_PYTHON_EXCLUDED_URLS to skip probe paths; pair with sampling strategies for distributed tracing for ratio-based control.
High-cardinality span names from raw URLs. Error signature: the backend shows one unique operation per request ID, making aggregation impossible. Root cause: a generic WSGI/ASGI wrapper was used, or custom middleware ran before the router resolved the template, so the resolved URL became the span name. Remediation: use the framework-specific instrumentor and ensure it sits at the outermost layer so http.route carries the template, not the path.
Frequently Asked Questions
Does auto-instrumentation create both server and client spans?
The framework instrumentor creates SERVER spans for inbound requests. Outbound HTTP calls become CLIENT spans only when you also install the matching client instrumentation, such as the requests or httpx instrumentor.
How do I stop health checks from flooding my traces?
Set the excluded_urls option on the instrumentor or the OTEL_PYTHON_EXCLUDED_URLS environment variable to a comma-separated list of path patterns. Matching requests are never sampled.
Can I mix auto-instrumentation with manual spans?
Yes. The instrumentor sets the request span as the active context, so any tracer.start_as_current_span call inside a view automatically becomes a child of the server span without extra wiring.
Why does my route attribute show the raw URL instead of the template?
Some frameworks resolve the route template late in the request cycle. Ensure the instrumentor runs before custom middleware that short-circuits requests, and confirm the framework version exposes the route pattern to the instrumentation hook.