Instrumenting aiohttp Client Requests with OpenTelemetry

An outbound aiohttp call that carries no trace headers makes the downstream service start a brand-new trace, so the call chain breaks at every hop and a slow request looks like two unrelated traces instead of one. This page is for engineers running asyncio services that fan out over HTTP and want those hops stitched together; it is part of the async tracing patterns guide within Distributed Tracing and OpenTelemetry in Python, and it shows exactly how AioHttpClientInstrumentor produces client spans and propagates W3C trace context.

The instrumentation patches aiohttp.ClientSession so every request opens a CLIENT span and writes the active trace context into the outgoing headers as a traceparent value. The receiving service, if it extracts that header, attaches its SERVER span to the same trace. This handoff is the client-side half of context propagation and baggage, and it is what turns a pile of independent spans into one connected call graph.

Prerequisites

Pin the instrumentation against the SDK. The client instrumentation tracks the unstable instrumentation channel, so a bounded range keeps builds reproducible.

pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
            "opentelemetry-instrumentation-aiohttp-client>=0.51b0,<1.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0" \
            "aiohttp>=3.9.0,<4.0.0"

Configure the exporter endpoint and service name so client spans are attributed correctly. OTEL_SERVICE_NAME becomes the service.name resource attribute that your backend uses to place these spans on the caller's side of the service map.

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="checkout-gateway"

Implementation

Step 1 — install the tracer provider before anything else. Set up the TracerProvider and a BatchSpanProcessor at import time, before any ClientSession is created. Instrumentation resolves its tracer at patch time; if the provider is installed afterwards, requests bind to a NoOpTracer and no spans are ever exported.

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# 1. Provider first so instrumentation binds to a real tracer.
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"))
    )
)
trace.set_tracer_provider(provider)

Step 2 — instrument the aiohttp client once. The single instrument() call patches ClientSession process-wide, so it must run before the sessions you want traced are constructed. Calling it twice is a no-op; calling it from inside a worker after fork() is the usual fix for pre-fork servers that create their event loop late.

from opentelemetry.instrumentation.aiohttp_client import AioHttpClientInstrumentor

# 2. Patch ClientSession so every request opens a CLIENT span and injects context.
AioHttpClientInstrumentor().instrument()

Step 3 — issue requests inside an active span. The instrumentation injects traceparent from whatever span is current in the asyncio context, so the outbound call must run beneath a parent span to propagate a meaningful chain. In a web service that parent normally comes from the inbound server instrumentation described in instrumenting Python web frameworks; in a worker or script, open one yourself.

import asyncio
import aiohttp
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def fetch_inventory(sku: str) -> dict:
    # The CLIENT span and traceparent header are created under this parent span.
    with tracer.start_as_current_span("check-inventory"):
        async with aiohttp.ClientSession() as session:
            async with session.get(
                f"http://inventory:8080/items/{sku}"
            ) as resp:
                resp.raise_for_status()
                return await resp.json()

asyncio.run(fetch_inventory("SKU-9931"))

Step 4 — confirm the header reaches the wire. The downstream service receives a request whose headers carry the injected context.

Expected Output: the outgoing request includes a W3C traceparent header that the downstream service can extract.

GET /items/SKU-9931 HTTP/1.1
Host: inventory:8080
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

How the traceparent header is injected

The instrumentation patches ClientSession._request through an aiohttp.TraceConfig. When a request begins, it opens the CLIENT span, makes that span current in a copy of the active context, and then calls the configured global propagator's inject method against the request's headers mapping. The default propagator is TraceContextTextMapPropagator, which writes the traceparent field; if you have also registered W3C baggage or B3 through a CompositePropagator, each one writes its own header in the same pass. Injection uses a setter that mutates the outgoing CIMultiDict in place, so it appends to whatever headers you supplied rather than replacing them.

The traceparent value encodes four hyphen-separated fields: the version (00), the 32-hex-character trace id, the 16-hex-character parent span id — which is this client span's id, not the request's logical parent — and a two-character trace-flags byte whose low bit is the sampled flag. A downstream service that extracts a traceparent with the sampled bit clear should honour that decision under a parent-based sampler, which is how a head-sampling choice made at the edge propagates intact through every hop; see sampling strategies for distributed tracing for how that decision is made. Because the injected span id is the client span's id, the downstream SERVER span's parent_id equals that value, and that equality is the join key your backend uses to draw the edge between the two services.

Injection reads the current context at request time, so if you mutate context between opening your span and issuing the request — by entering a nested span, for example — the header reflects the innermost active span. That is usually correct, but it explains why a request issued from inside a helper coroutine links to the helper rather than to the request handler.

Enriching spans with request and response hooks

The default CLIENT span carries http.method, http.url, and http.status_code, but production traces usually need more: the upstream service name, a request id, a payload size, or a categorised error. Hooks run inside the span's lifetime, so anything they set lands on the right span without manual context juggling — the same span attribute discipline that applies to spans you create by hand. The request_hook fires after the span opens but before the bytes go out; the response_hook fires once headers return.

from aiohttp import TraceRequestStartParams, TraceRequestEndParams
from opentelemetry.trace import Span

def request_hook(span: Span, params: TraceRequestStartParams) -> None:
    # Add a stable peer name and the target host for easier filtering.
    if span and span.is_recording():
        span.set_attribute("peer.service", "inventory")
        span.set_attribute("http.request.host", params.url.host)

def response_hook(
    span: Span, params: TraceRequestEndParams
) -> None:
    # Record the upstream's content length and flag server errors.
    if span and span.is_recording():
        length = params.response.headers.get("Content-Length")
        if length is not None:
            span.set_attribute("http.response.body.size", int(length))
        if params.response.status >= 500:
            span.set_attribute("error.type", "upstream_5xx")

AioHttpClientInstrumentor().instrument(
    request_hook=request_hook,
    response_hook=response_hook,
)

Expected Output: the enriched CLIENT span now carries the hook-added attributes.

{
  "name": "GET",
  "kind": "SpanKind.CLIENT",
  "attributes": {
    "http.method": "GET",
    "http.url": "http://inventory:8080/items/SKU-9931",
    "http.status_code": 200,
    "peer.service": "inventory",
    "http.request.host": "inventory",
    "http.response.body.size": 184
  }
}

Always guard hook bodies with span.is_recording(). Under a sampler that dropped the trace, the span is a non-recording stub; calling set_attribute on it is harmless, but the surrounding work — parsing headers, computing sizes, formatting strings — is wasted on every request, and the guard makes the cost-free path explicit. Keep hooks synchronous and allocation-light: they execute on the event loop, so an accidental blocking call there stalls every other coroutine in the process.

Connection pooling and span timing

A ClientSession owns a TCPConnector whose pool is bounded by limit (total) and limit_per_host. This matters for trace interpretation because the CLIENT span covers the whole request, including any time spent waiting for a free connection from the pool. Under saturation — more concurrent requests than limit_per_host allows — a span's duration includes queueing latency that is invisible in the status code, so a slow span with a 200 result and a healthy upstream is a strong signal that the connector, not the remote service, is the bottleneck. Create one long-lived session per upstream and reuse it: constructing a session per request defeats keep-alive, forces a fresh TCP and TLS handshake on every call, and inflates every span with connection-setup time that should have been amortised.

import aiohttp
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

# One shared, long-lived session; the connector pools and reuses sockets.
_connector = aiohttp.TCPConnector(limit=100, limit_per_host=20)
_session: aiohttp.ClientSession | None = None

async def get_session() -> aiohttp.ClientSession:
    global _session
    if _session is None or _session.closed:
        _session = aiohttp.ClientSession(connector=_connector)
    return _session

async def fetch_price(sku: str) -> dict:
    session = await get_session()
    with tracer.start_as_current_span("fetch-price"):
        async with session.get(f"http://pricing:8080/p/{sku}") as resp:
            resp.raise_for_status()
            return await resp.json()

Expected Output: reused sockets keep span durations dominated by upstream time, not handshakes.

GET   http.url=http://pricing:8080/p/SKU-9931   duration=12ms   (warm pool)
GET   http.url=http://pricing:8080/p/SKU-4410   duration=94ms   (cold session per call)

Retries and per-attempt spans

Transient upstream failures are normal, and you usually want each retry attempt to appear as its own CLIENT span so the trace shows the attempt count and the backoff gaps. Wrap the retry loop in an application span and re-enter session.get on each iteration — that re-enters the instrumented request path, producing a fresh span per attempt, each carrying its own traceparent with a new client span id, so the downstream sees genuinely distinct requests rather than one retried mystery.

import asyncio

async def fetch_with_retry(sku: str, attempts: int = 3) -> dict:
    session = await get_session()
    with tracer.start_as_current_span("fetch-inventory-retrying") as parent:
        for attempt in range(1, attempts + 1):
            parent.set_attribute("retry.attempt", attempt)
            try:
                # Each iteration opens a new instrumented CLIENT span.
                async with session.get(
                    f"http://inventory:8080/items/{sku}"
                ) as resp:
                    resp.raise_for_status()
                    return await resp.json()
            except aiohttp.ClientError:
                if attempt == attempts:
                    raise
                await asyncio.sleep(0.2 * attempt)  # linear backoff

Expected Output: three sibling CLIENT spans under one parent, each with its own span id.

fetch-inventory-retrying   parent_id=null
GET   parent_id=<fetch-inventory-retrying>   http.status_code=503
GET   parent_id=<fetch-inventory-retrying>   http.status_code=503
GET   parent_id=<fetch-inventory-retrying>   http.status_code=200

Because parent.set_attribute overwrites retry.attempt on each pass, the parent span records the final attempt count while the children preserve the per-attempt detail. If your backoff is long enough to matter, add a span event per failure instead of relying on the attribute alone.

Configuration options

AioHttpClientInstrumentor().instrument accepts hooks and filters that shape the emitted spans; the same keywords are accepted by create_trace_config() when you want to trace one session rather than the whole process.

A url_filter is the standard way to keep high-cardinality path segments or sensitive query strings out of the http.url attribute, and it runs before the attribute is ever set, so nothing unredacted reaches the exporter.

from yarl import URL

def url_filter(url: URL) -> str:
    # Collapse ids to a template and drop the query string entirely.
    parts = [p if not p.isdigit() else "{id}" for p in url.path.split("/")]
    return str(url.with_path("/".join(parts)).with_query(None))

Two environment variables round out the configuration surface: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=aiohttp-client turns the integration off without a code change, and OTEL_SEMCONV_STABILITY_OPT_IN=http switches the emitted attributes from the legacy http.method / http.url / http.status_code names to the stable http.request.method / url.full / http.response.status_code set. Flip that opt-in deliberately, since dashboards and alerts keyed on the old names go quiet the moment it changes.

Verification

Attach a ConsoleSpanExporter locally and confirm a CLIENT span is produced with the expected HTTP attributes. The trace_id must match the SERVER span recorded by the downstream service, and that span's parent_id must equal the client span's span_id.

{
  "name": "GET",
  "kind": "SpanKind.CLIENT",
  "attributes": {
    "http.method": "GET",
    "http.url": "http://inventory:8080/items/SKU-9931",
    "http.status_code": 200
  },
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

For a check that does not need a running collector, assert on the headers directly with an InMemorySpanExporter and a stub endpoint: the request handler should see a traceparent whose middle field equals the exported client span's span_id. That single assertion catches both halves of the failure mode — a missing header and a header built from the wrong context.

If the downstream SERVER span carries a different trace_id, the request either ran outside any active span or the receiving service is not extracting the header. Confirm the client call sits under a parent span and that the server runs matching instrumentation — the extraction half of context propagation. When the trace ids do line up but your logs still cannot be correlated with them, the missing piece is usually log enrichment rather than propagation; see adding trace ids to log records.

Common mistakes

• Error signature: the console exporter prints nothing, or spans appear with trace_id 00000000000000000000000000000000. Root cause: instrument() ran before set_tracer_provider(), so the instrumentation captured the default no-op tracer, or sessions were built before the patch was applied. Remediation: initialise the provider and call instrument() at process start, ahead of any ClientSession construction — and in pre-fork servers, do both inside the worker rather than at import time in the master.

• Error signature: the downstream SERVER span opens a new trace with parent_id null even though the client span exists. Root cause: the request was issued outside any active span, so injection had nothing meaningful to serialise. Remediation: wrap outbound calls in start_as_current_span, or let inbound server instrumentation establish the parent, the same discipline applied when setting up OpenTelemetry in FastAPI.

• Error signature: CLIENT span durations sit an order of magnitude above the upstream's own server spans, and the log fills with "Unclosed client session" warnings. Root cause: a ClientSession is constructed per request, so every call pays a fresh DNS lookup plus TCP and TLS handshake, and unawaited sessions leak sockets. Remediation: build one long-lived session per upstream, share it through a module-level accessor, and close it in your shutdown hook — the same shared-client rule covered across the async tracing patterns guide.

• Error signature: trace search slows to a crawl and http.url shows millions of distinct values, some containing tokens. Root cause: raw URLs with embedded identifiers and query strings are recorded verbatim on every span. Remediation: supply a url_filter that templates id segments and drops the query, keeping both the cardinality and the secrets out of your backend.

Related

• Async tracing patterns — the parent guide covering contextvars, task boundaries, and non-blocking export.
• Tracing SQLAlchemy async queries — the database half of the same asyncio request.
• Context propagation and baggage — how the extraction side works and what else can ride alongside traceparent.
• Propagating trace context across Celery tasks — the same handoff over a broker instead of HTTP.
• OpenTelemetry SDK setup — provider lifecycle and processor tuning that these client spans depend on.

Frequently Asked Questions