Adding Span Links for Batch Work

A span has exactly one parent, which is the right model for work contained in a request and the wrong one for work that serves many. A consumer that processes a batch of fifty messages, each produced by a different request, cannot be the child of fifty spans; choosing one as its parent misattributes the batch to that request and leaves the other forty-nine traces with no idea their work ever happened. Span links solve this by connecting the batch span to every trace it served without claiming to belong to any of them. This page covers capturing contexts, creating linked spans, and keeping links useful at scale. It is a task article under span lifecycle and attributes, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "opentelemetry-api>=1.27.0,<2.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Implementation

Step 1 — Capture each item's span context when it is enqueued. The producer's span is current only while the request runs. Its context must be captured then and stored with the item — in the message headers for a broker, in a column for a database-backed queue, in the object itself for an in-memory buffer. Injecting through the propagator gives a portable, serialisable form.

from opentelemetry import trace
from opentelemetry.propagate import inject

tracer = trace.get_tracer("orders")

def enqueue_for_indexing(order_id: str) -> None:
    carrier: dict[str, str] = {}
    inject(carrier)                               # 1. this request's context, as headers
    queue.put({"order_id": order_id, "otel": carrier})

Step 2 — Start the batch span as a new root, with a link per item. The worker extracts each item's context and builds a Link from its span context. The batch span is started with those links and no parent — it is the root of its own trace — so it does not claim to belong to any of the requests.

from opentelemetry.propagate import extract
from opentelemetry.trace import Link

def process_batch(items: list[dict]) -> None:
    links = []
    for position, item in enumerate(items[:MAX_LINKS]):
        span_ctx = trace.get_current_span(extract(item["otel"])).get_span_context()
        if span_ctx.is_valid:
            links.append(Link(span_ctx, attributes={
                "app.order.id": item["order_id"],       # 2. what this link refers to
                "app.batch.position": position,
            }))

    with tracer.start_as_current_span(
        "index orders",
        context=trace.set_span_in_context(trace.INVALID_SPAN),   # 3. a new root
        links=links,
    ) as span:
        span.set_attribute("app.batch.size", len(items))
        span.set_attribute("app.batch.links_recorded", len(links))
        index(items)

Step 3 — Put attributes on each link. A link on its own says only that two spans are related, and in a batch of fifty that is almost no information at all. An attribute naming the item — the order identifier, its position in the batch — makes the relationship interpretable when someone opens the batch span and sees fifty links.

Step 4 — Cap the links. Each link carries a trace identifier, a span identifier and its attributes, so a thousand links add real weight to one span. The SDK limits links per span, and trace viewers stop rendering them usefully well before that limit. Linking up to a few dozen items and recording the total batch size as an attribute keeps the span readable; for very large batches, linking a sample — the first N, or every Kth — preserves navigability without the cost.

export OTEL_SPAN_LINK_COUNT_LIMIT=64

Step 5 — Make it navigable from the request side too. A link lives on the batch span, so a viewer showing request B's trace does not know the batch exists unless it indexes links in reverse. Some backends do; where they do not, recording the batch's trace identifier back on the item — for example, in a log record the worker emits per item with the item's own trace identifier attached — gives the request side a way to find its processing.

Expected Output: the batch span with its links.

{
  "name": "index orders",
  "parentSpanId": "",
  "attributes": {"app.batch.size": 3, "app.batch.links_recorded": 3},
  "links": [
    {"traceId": "9f2a71c4…", "spanId": "4b1e77a2…", "attributes": {"app.order.id": "ord_1", "app.batch.position": 0}},
    {"traceId": "31c0e8b9…", "spanId": "7a0d14ce…", "attributes": {"app.order.id": "ord_2", "app.batch.position": 1}},
    {"traceId": "e44f2a07…", "spanId": "b90c3e11…", "attributes": {"app.order.id": "ord_3", "app.batch.position": 2}}
  ]
}

Sampling and links

Links interact with sampling in a way that affects whether they are useful, and it is worth planning for.

A head sampler decides at the root of each trace whether to record it. The batch span is the root of its own trace, so it is sampled independently of the requests it links to. With a ten percent sample rate, the batch trace is kept one time in ten, and each linked request is kept one time in ten independently — so the chance that a given request and the batch processing it are both recorded is one in a hundred. A link to an unrecorded span is a dangling reference, pointing at a trace that does not exist in the store.

Three arrangements reduce this. The simplest is to keep batch spans at a higher rate than request spans, since there are far fewer of them — one batch serves many requests — and their cost is correspondingly lower. Samplers can inspect links at span creation, so a sampler that keeps the batch span whenever any linked context was sampled ensures the processing of every recorded request is also recorded. And at the collector, a tail sampling policy that keeps a trace when a linked trace was kept achieves the same effect after the fact, as described in tail sampling in the OpenTelemetry Collector.

The sampled flag in each linked context records whether that originating trace was sampled, which is what makes the second arrangement possible: the batch sampler can see, per link, whether the link's target will exist.

Links beyond batches

Batches are the clearest case, and the same pattern applies in several other places where work relates to earlier work without being contained in it.

Message consumers. A consumer processing a message produced minutes earlier can be a child of the producer's context by messaging convention, which extends the producer's trace across the whole delay. Many teams prefer a new root for the consumer with a link to the producer, keeping each trace's duration honest. Either is defensible; consistency across the fleet matters more than the choice. The propagation side is covered in propagating trace context across Celery tasks.

Retries of an earlier failure. A scheduled retry of a failed operation can link to the original attempt's span, so the retry's trace shows what it is retrying without the original trace growing by hours.

Fan-in aggregation. A job that aggregates the results of many earlier requests — a nightly report over the day's orders — can link to a sample of the requests it covers, which is occasionally invaluable when a number in the report looks wrong.

In each case the distinction is the same as for batches: a parent means "this work is part of that one", and a link means "this work exists because of that one". Choosing the right one keeps trace durations meaningful and relationships navigable.

Configuration options

Verification

Check that the batch span has no parent and a link per item.

spans = exporter.get_finished_spans()
batch = next(s for s in spans if s.name == "index orders")
assert batch.parent is None
assert len(batch.links) == 3
assert {l.attributes["app.order.id"] for l in batch.links} == {"ord_1", "ord_2", "ord_3"}

Expected Output:

.
1 passed

Common mistakes

First item as parent. Error signature: one request's trace stretched by seconds of batch processing, and other requests with no trace of their items. Root cause: choosing a parent where none is correct. Remediation: a new root with links.

Context captured at dequeue. Error signature: links pointing at the worker's own context. Root cause: capturing after the producer's span ended. Remediation: capture at enqueue and store it with the item.

Unbounded links. Error signature: huge spans that viewers cannot render. Root cause: a link per item for large batches. Remediation: cap links and record the total.

Independent sampling of batches and requests. Error signature: links that point to traces not in the store. Root cause: both sampled at a low rate independently. Remediation: sample batches higher, or use a link-aware sampler.

Links without attributes. Error signature: a batch span with fifty links and no way to tell which item each one represents. Root cause: bare links. Remediation: an identifier and position on every link.

Links added after start where the sampler needs them. Error signature: link-aware sampling that never triggers. Root cause: samplers see only links present at creation. Remediation: pass links when starting the span.

Frequently Asked Questions