Python Observability

Instrumenting Starlette and ASGI Middleware

ASGI applications — Starlette, and frameworks built on it such as FastAPI — are instrumented by middleware that wraps the whole application and creates a server span per request. Getting a useful trace depends on where that middleware sits relative to everything else, on giving the span a route template rather than a raw path, and on enriching the one server span rather than creating a second. Streaming responses and WebSockets add their own behaviour. This page covers all of it. It is a task article under instrumenting Python web frameworks, part of the distributed tracing and OpenTelemetry in Python section, and it complements setting up OpenTelemetry in FastAPI.

Outermost means everything is measured An ASGI middleware stack is drawn as nested layers. With the OpenTelemetry middleware outermost, its server span contains the authentication middleware, the CORS middleware, the error handling middleware, the router and the handler; the span's duration is what the client experienced, and a slow token verification in authentication shows up inside it. With the OpenTelemetry middleware placed after authentication instead, authentication runs outside any span: its latency is invisible to the trace, requests it rejects produce no span at all, and the server span's duration understates what the client saw. The note records that the second arrangement is common because middleware order is easy to get backwards, and that the symptom is traces faster than the client-side measurements. the middleware stack, two orderings OpenTelemetry outermost server span auth — token check timed CORS error handling router → handler duration = what the client saw OpenTelemetry after auth auth — outside any span, rejections invisible server span CORS · error handling router → handler duration understates the client's symptom of the wrong order: traces consistently faster than client-side timings
Middleware inside the span is measured; middleware outside it is invisible. Outermost is the only position that makes the span describe the whole request.

Prerequisites

pip install "opentelemetry-instrumentation-starlette>=0.48b0,<1.0.0" \
            "opentelemetry-instrumentation-asgi>=0.48b0,<1.0.0" \
            "starlette>=0.37.0,<1.0.0"

Implementation

Step 1 — Instrument the application so the middleware is outermost. StarletteInstrumentor.instrument_app adds the OpenTelemetry ASGI middleware to the application. Called after the application is created and before other middleware is added, it ends up outermost, so its span covers every other layer. For a plain ASGI application without Starlette, wrapping the application object directly with the ASGI middleware achieves the same.

from starlette.applications import Starlette
from opentelemetry.instrumentation.starlette import StarletteInstrumentor

app = Starlette(routes=routes)
StarletteInstrumentor.instrument_app(app)       # 1. outermost: covers everything below

app.add_middleware(AuthMiddleware)              # runs inside the server span
app.add_middleware(CORSMiddleware, allow_origins=["https://app.example"])

Step 2 — Let Starlette supply the route template. The generic ASGI middleware sees only the raw path when the request arrives. Starlette's instrumentation renames the span once the router has matched, so the span is named GET /orders/{order_id} rather than with the identifier. Unmatched requests are named by method alone, so random paths from scanners form one group rather than thousands, as described in naming spans and using semantic conventions.

Step 3 — Enrich the server span from your own middleware. Application middleware often knows things the instrumentation does not: the authenticated tenant, the API version requested, a feature flag. Adding them as attributes on the current span puts them on the server span. Starting a new span for "the request" inside middleware produces a redundant nested server span that doubles the span count and confuses every aggregate.

from opentelemetry import trace
from starlette.middleware.base import BaseHTTPMiddleware

class RequestContextMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        span = trace.get_current_span()                       # 2. the server span
        if span.is_recording():
            span.set_attribute("app.api_version", request.headers.get("x-api-version", "1"))
            if (tenant := getattr(request.state, "tenant", None)):
                span.set_attribute("app.tenant.id", tenant.id)
        response = await call_next(request)
        return response

Step 4 — Create child spans only for distinct steps. A child span is worth its cost only when it measures something an engineer would want to see on its own. Where middleware does something costly and separable — a remote token verification, a rate limit check against a shared store — a child span around that step shows its cost. Everything else about the request belongs on the server span.

tracer = trace.get_tracer("auth")

class AuthMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        with tracer.start_as_current_span("verify token"):   # 3. a distinct, costly step
            request.state.user = await verify(request.headers.get("authorization"))
        return await call_next(request)

Step 5 — Exclude health and metrics endpoints. Probes and scrapes arrive constantly — every few seconds from every load balancer and every scraper — carry no diagnostic value and consume sampling budget. Excluding them by URL keeps the traces about real traffic.

export OTEL_PYTHON_STARLETTE_EXCLUDED_URLS="healthz,readyz,metrics"

Expected Output: one server span per request, named by template, with application attributes and a child span for the token check.

GET /orders/{order_id}   SERVER  214 ms  app.tenant.id=acme app.api_version=2
  verify token           INTERNAL 31 ms
  SELECT orders          CLIENT   41 ms
When the server span ends Three kinds of ASGI interaction are drawn against time. A normal response: the handler runs, returns a body, and the server span ends when the body has been sent, shortly after the handler returns. A streaming response: the handler returns a streaming response object almost immediately, and the server span stays open while the body is generated and sent, ending only when the stream completes several minutes later. A WebSocket connection: the span begins at the handshake and ends when the connection closes, which may be hours later, with individual messages exchanged inside it. The note records that all three durations are accurate, and that mixing them in one latency aggregate makes the aggregate meaningless, so long-lived endpoints should be excluded from request latency views or distinguished by attribute. three interactions, three span lifetimes normal response ends when the body is sent · ms streaming response span open while the stream runs minutes WebSocket handshake → close · messages inside · hours all three durations are accurate — mixing them in one aggregate is not exclude long-lived endpoints from request latency views, or mark them with an attribute
A streaming or WebSocket span is correct and very long. Leaving it in a request latency aggregate makes the p99 a measurement of connection lifetime.

Streaming, WebSockets and lifespan

Three ASGI features behave differently from ordinary requests, and each needs a decision.

Streaming responses. The server span ends when the last chunk of the body is sent, not when the handler returns. A handler that returns a streaming response immediately and then generates data for five minutes produces a five-minute span, which is accurate and distorts any latency aggregate it is included in. The options are to exclude streaming endpoints from latency views, to mark them with an attribute so views can filter them, or — where time-to-first-byte is the meaningful measure — to record it as an attribute or an event on the span when the first chunk goes out.

WebSockets. A WebSocket connection is one long-lived span from handshake to close, which may be hours. Individual messages are better represented as span events on the connection span, or as their own short spans when message handling is substantial. A span per message that is a child of the connection span produces traces with thousands of children, which few viewers handle well; linking message spans to the connection, as in adding span links for batch work, is often more readable.

Lifespan. ASGI lifespan events — startup and shutdown — are not requests and are not traced by the middleware. Startup work that is worth measuring, such as warming a cache or loading a model, can be wrapped in a manual span in the startup handler. That span is the root of its own trace, which is correct: startup belongs to no request. The shutdown handler is where providers should be flushed, as covered in graceful shutdown and telemetry flush.

BaseHTTPMiddleware and context

Starlette's BaseHTTPMiddleware is convenient and has a property that matters for tracing: it runs the downstream application in a separate task. Context variables are copied into that task when it is created, so the server span is visible inside the handler, and spans started in the handler are correctly parented. What does not flow is the reverse direction — context variables set inside the handler are not visible to the middleware after call_next returns, because they were set in the child task's copy.

For tracing this is usually harmless, because the middleware reads the server span, which it did not create and which exists in both contexts. It becomes relevant when middleware expects to read something the handler set in a context variable — a tenant resolved deep in the handler, for example — to add it to the span after the response. That value will not be there. The workaround is to put such values on request.state, which is a shared object rather than a context variable, and read them from there.

Pure ASGI middleware — a class implementing the ASGI interface directly — does not create a separate task and does not have this property. It is more verbose to write and slightly faster, and for middleware whose only job is enriching the current span, the difference rarely matters. Where middleware does heavy work per request, or where the separate-task behaviour causes confusion, writing it as pure ASGI middleware removes both the overhead and the surprise.

What each ASGI connection type records A table of the three ASGI scope types and what the instrumentation records for each. An HTTP request produces one server span from the first byte received until the final body message, named with the route template once Starlette has matched it. A WebSocket connection produces one span for the whole connection, which may last hours, so its duration says nothing about latency and messages are better recorded as events or separate spans. The lifespan scope, which covers startup and shutdown, produces no request spans and should be excluded. The note says the HTTP span is the one latency dashboards and SLOs should read. ASGI scope span created what to watch for http one server span per request route template set after routing websocket one span for the connection may last hours — not a latency lifespan none startup work needs its own span latency dashboards and SLOs should read the http spans only filter websocket spans out of duration panels, or record messages as events
Only the HTTP scope produces request-shaped spans. A WebSocket span measures how long someone stayed connected.

Configuration options

Setting Value Why
Middleware position outermost the span covers every layer
Route naming Starlette instrumentation template names, not raw paths
Application context attributes on the current span no redundant server spans
Child spans in middleware only for costly, distinct steps shows their cost separately
Excluded URLs health, readiness, metrics sampling spent on real traffic
Streaming endpoints marked or excluded from latency views accurate but long spans
WebSocket messages events or linked spans readable connection traces
Lifespan manual spans at startup; flush at shutdown not covered by the middleware

Verification

Assert that each request produces exactly one server span, named by template.

from starlette.testclient import TestClient

def test_one_server_span_per_request(exporter):
    TestClient(app).get("/orders/8812")
    servers = [s for s in exporter.get_finished_spans()
               if s.kind == trace.SpanKind.SERVER]
    assert len(servers) == 1
    assert servers[0].name == "GET /orders/{order_id}"

Expected Output:

.
1 passed

Two server spans per request means both the ASGI and Starlette instrumentations were applied independently, or application middleware is starting its own request span.

Common mistakes

Instrumentation added after other middleware. Error signature: traces faster than client-side timings, and no spans for requests rejected by authentication. Root cause: middleware outside the server span. Remediation: instrument first so it is outermost.

Raw paths as span names. Error signature: thousands of operations, one per identifier. Root cause: generic ASGI middleware without Starlette's route renaming. Remediation: use the Starlette instrumentation.

A second request span in middleware. Error signature: nested server spans and doubled span counts. Root cause: middleware starting its own span for the request. Remediation: add attributes to the current span.

Streaming endpoints in latency aggregates. Error signature: a p99 of several minutes. Root cause: long-lived spans mixed with ordinary requests. Remediation: exclude or mark them.

Reading handler-set context variables in middleware after the response. Error signature: an attribute the middleware adds after call_next is always empty. Root cause: the handler ran in a copied context. Remediation: pass such values through request.state.

Instrumenting twice. Error signature: duplicate spans after enabling automatic instrumentation on an already instrumented app. Root cause: both the launcher and instrument_app applied. Remediation: choose one mechanism.

Frequently Asked Questions

What is the difference between the ASGI and Starlette instrumentations?

The ASGI instrumentation is framework-agnostic middleware that creates a server span for every ASGI request. The Starlette instrumentation builds on it and names the span with the matched route template, which the generic middleware cannot know.

Where should the OpenTelemetry middleware go in the stack?

Outermost. Middleware added after it runs inside its span, so authentication, CORS and error handling are timed as part of the request. Middleware outside it is invisible to the trace, including any latency it adds.

Should my own middleware create spans?

Usually not for the request itself — that would create a redundant nested server span. Add attributes to the current span instead. Create a child span only for a distinct, costly step the middleware performs, such as a token verification call.

Why do streaming responses show very long spans?

Because the server span ends when the response body finishes sending, not when the handler returns. For a stream that lasts minutes, the span lasts minutes. That is accurate, and it means such endpoints should be excluded from latency aggregates or named distinctly.

Related Guides