Tracing asyncio gather and TaskGroups

Concurrent fan-out with asyncio.gather or a TaskGroup is one of the places where tracing earns its keep: the parent's duration is the slowest child's, and only per-branch spans say which child that was. The basic nesting works automatically, because tasks copy the context that was current when they were created. What needs deliberate handling is everything around failure — the branch that raised, the siblings a TaskGroup cancels in response, and the exception group that arrives at the parent. This page covers span structure for fan-out and recording partial failure and cancellation honestly. It is a task article under async tracing patterns, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "opentelemetry-api>=1.27.0,<2.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

TaskGroup requires Python 3.11 or later.

Implementation

Step 1 — Wrap the fan-out in a parent span. A span around the concurrent section gives the branches a common parent and measures the fan-out as a whole. Because tasks copy the context at creation, starting the parent before creating the tasks is all that is needed for the children to nest under it.

import asyncio
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("quotes")

async def build_quote(order):
    with tracer.start_as_current_span("build quote") as parent:
        async with asyncio.TaskGroup() as tg:
            inv = tg.create_task(traced("inventory", fetch_inventory, order))
            pri = tg.create_task(traced("pricing", fetch_pricing, order))
            shp = tg.create_task(traced("shipping", fetch_shipping, order))
        return combine(inv.result(), pri.result(), shp.result())

Step 2 — Give each branch its own span. Each concurrent unit starts its own span, so its duration and outcome are visible separately. The parent's duration is roughly the slowest child's; only the children's spans say which one that was.

Step 3 — Record failures on the branch that failed. When a branch raises, its span records the exception and sets error status before the exception propagates. That puts the failure on the operation that caused it rather than only on the parent, where it arrives wrapped in an exception group.

Step 4 — Mark cancellation distinctly. When one branch fails, a TaskGroup cancels the others, and each receives CancelledError at its current await. A span that simply ends as that exception passes through looks like either a success or — if the instrumentation records every exception as an error — a failure. Catching CancelledError, marking the span, and re-raising records what actually happened.

async def traced(name: str, fn, *args):
    with tracer.start_as_current_span(name, record_exception=False,
                                      set_status_on_exception=False) as span:
        try:
            return await fn(*args)
        except asyncio.CancelledError:
            span.set_attribute("async.cancelled", True)     # 1. stopped, not failed
            span.set_status(Status(StatusCode.UNSET))
            raise                                            # cancellation must propagate
        except Exception as exc:
            span.record_exception(exc)                       # 2. the real failure
            span.set_status(Status(StatusCode.ERROR, type(exc).__name__))
            raise

Step 5 — Summarise the outcome on the parent. The parent receives an ExceptionGroup containing the failures, not the cancellations. Recording counts on the parent — succeeded, failed, cancelled — gives a one-line summary of the fan-out visible without opening the children, and the parent's own status reflects that the operation as a whole failed.

        # inside build_quote, around the TaskGroup
        try:
            async with asyncio.TaskGroup() as tg:
                tasks = {n: tg.create_task(traced(n, f, order)) for n, f in BRANCHES.items()}
        except* Exception as eg:
            parent.set_attribute("fanout.failed", len(eg.exceptions))
            parent.set_attribute("fanout.cancelled",
                                 sum(1 for t in tasks.values() if t.cancelled()))
            parent.set_status(Status(StatusCode.ERROR, "fan-out failed"))
            raise
        parent.set_attribute("fanout.succeeded", len(tasks))

Expected Output: the trace as exported, with each span's status telling the true story.

build quote   1.02s  ERROR   fanout.failed=1 fanout.cancelled=2
  inventory   0.21s  UNSET   async.cancelled=true
  pricing     0.20s  ERROR   exception: ConnectionError pricing unreachable
  shipping    0.21s  UNSET   async.cancelled=true

Reading a fan-out trace

Once each branch has its own span with an honest status, a fan-out trace answers three questions quickly, and it is worth knowing where to look for each.

Which branch set the duration? The parent's duration is the slowest child's plus a little overhead. The child whose end aligns with the parent's end is the one that determined latency, and it is the only one whose optimisation would make the parent faster. Improving any other branch changes nothing the caller can see, which is a useful thing to know before spending effort on it.

Did the branches actually run concurrently? Children whose start times are staggered, each beginning as the previous one ends, are running sequentially despite the fan-out — usually because something inside them is synchronous and blocking the loop, or because they share a resource with a concurrency limit of one. The trace shows it immediately as a staircase where a column was expected. The causes and their diagnosis are in diagnosing blocked event loops in production.

What was lost when it failed? Cancelled branches carry their cancellation marker and their partial duration. A branch cancelled after doing most of its work — a write that completed but whose confirmation was never processed — is a correctness question as well as a performance one, and the trace is the evidence of what state was left behind.

These readings depend entirely on each branch having its own span. A fan-out traced only at the parent level shows a duration and an outcome and nothing about why.

Large fan-outs

The patterns above assume a handful of branches. A fan-out over hundreds or thousands of items — processing a batch concurrently, querying many shards — needs a different span structure, because a span per item produces traces no viewer can open and export volumes that dominate the service's telemetry.

Bound concurrency and span the batches. Large fan-outs usually run through a semaphore or a bounded pool anyway, to avoid overwhelming downstream services. A span per concurrent slot or per chunk of items, rather than per item, keeps the span count proportional to the concurrency rather than to the input size.

Record counts and outliers on the parent. How many items succeeded, failed and were retried, the slowest item's duration, and the identifiers of the items that failed — capped to a small number — carry most of what a per-item span would have shown, as attributes on one span.

Keep per-item spans for failures only. Starting a span only when an item fails, or recording a span event on the parent for each failure, keeps the detail where it is needed. Successful items rarely need to be seen individually.

Consider links instead of children. When the fan-out processes items that belong to different upstream requests — a consumer processing a batch of messages, for instance — each item relates to a different trace, and links from the batch span to each originating context describe that far better than children would. Adding span links for batch work covers the mechanism.

The underlying principle matches the one for batch jobs in telemetry from serverless and batch Python: spans describe units of work worth looking at individually, and counts describe the rest.

Configuration options

Verification

Run a fan-out with one failing branch in a test and assert on each span's status.

def test_cancelled_siblings_are_not_errors(exporter):
    with pytest.raises(ExceptionGroup):
        asyncio.run(build_quote(order_with_failing_pricing()))
    by_name = {s.name: s for s in exporter.get_finished_spans()}
    assert by_name["pricing"].status.status_code is StatusCode.ERROR
    for sibling in ("inventory", "shipping"):
        assert by_name[sibling].attributes.get("async.cancelled") is True
        assert by_name[sibling].status.status_code is StatusCode.UNSET

Expected Output:

.
1 passed

Common mistakes

No span per branch. Error signature: a slow fan-out with no indication of which branch was slow. Root cause: only the parent is instrumented. Remediation: a span inside each concurrent unit.

Cancellation recorded as an error. Error signature: inflated error rates on operations that were merely cancelled. Root cause: instrumentation that records every exception as a failure. Remediation: catch CancelledError, mark it, re-raise.

Swallowing CancelledError. Error signature: a TaskGroup that hangs or tasks that ignore shutdown. Root cause: catching cancellation without re-raising. Remediation: always re-raise it.

Plain gather with independent failures. Error signature: child spans ending after their parent; background work continuing after a request failed. Root cause: siblings not cancelled on the first exception. Remediation: TaskGroup, or return_exceptions=True.

Relying on the parent's exception group for diagnosis. Error signature: an investigation that has to parse exception text to learn which branch failed. Root cause: failures recorded only where they arrive. Remediation: record each failure on its own branch span.

Parent started after task creation. Error signature: branch spans parented to the request instead of the fan-out. Root cause: tasks copied the context before the parent span existed. Remediation: start the parent first.

Frequently Asked Questions