Linking Profiles to Traces with Span Context

A profile says what code a process spent its time in. A trace says which operation within a request was slow. Neither alone connects "this span took nine hundred milliseconds" to "because of this line", and that connection is where most of the value in continuous profiling lies. This page covers recording span context with each sample, the two query directions it enables, and the statistical limit that affects short spans. It is a task article under continuous profiling in production, part of the Python profiling and performance observability section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-api>=1.27.0,<2.0.0"

The profiling agent must support attaching labels to samples; most in-process agents do, and the mechanism below shows what they are doing.

Implementation

Step 1 — Read the span for the sampled thread, not the sampler's. The sampling thread's own context is empty; the context that matters belongs to the thread whose stack is being captured. OpenTelemetry stores the current span in a context variable, and the value for another thread is not directly readable from outside it. In-process agents solve this by having each thread publish its current span identifiers into a structure the sampler can read — typically by hooking span start and end.

# span_registry.py — each thread publishes what it is doing; the sampler reads it.
import threading
from opentelemetry.sdk.trace import SpanProcessor

ACTIVE: dict[int, tuple[str, str, str]] = {}   # thread id -> (trace_id, span_id, route)
_lock = threading.Lock()


class ActiveSpanPublisher(SpanProcessor):
    """Maintain a per-thread record of the innermost active span."""

    def on_start(self, span, parent_context=None):
        ctx = span.get_span_context()
        route = span.attributes.get("http.route", "") if span.attributes else ""
        with _lock:
            ACTIVE[threading.get_ident()] = (
                f"{ctx.trace_id:032x}", f"{ctx.span_id:016x}", route)

    def on_end(self, span):
        parent = span.parent
        with _lock:
            if parent is not None:
                prev = ACTIVE.get(threading.get_ident())
                if prev:
                    ACTIVE[threading.get_ident()] = (prev[0], f"{parent.span_id:016x}", prev[2])
            else:
                ACTIVE.pop(threading.get_ident(), None)

Step 2 — Attach the identifiers to each sample. The sampler looks up the thread it is sampling in the published record and labels the stack with whatever it finds. Samples from threads with no active span — background workers, the exporter, idle pool threads — are labelled as such, which is itself useful: it separates request work from everything else.

import sys, threading
from collections import Counter
from span_registry import ACTIVE

SAMPLES: Counter = Counter()

def sample_once():
    frames = sys._current_frames()
    for tid, frame in frames.items():
        if tid == threading.get_ident():
            continue
        trace_id, span_id, route = ACTIVE.get(tid, ("-", "-", "background"))
        stack = []
        while frame is not None and len(stack) < 64:
            stack.append(frame.f_code.co_name)
            frame = frame.f_back
        SAMPLES[(route, trace_id, span_id, ";".join(reversed(stack)))] += 1

Step 3 — Add the route template as a low-cardinality label. Trace and span identifiers are unique per request, which makes them ideal for drilling into one request and poor as a grouping dimension. The route template — /orders/{id} rather than /orders/8812 — has a small, bounded value set, and it is what makes "which endpoint is responsible for this frame" a cheap aggregate query rather than a scan over individual traces.

Step 4 — Link from the trace viewer. The correlation becomes routine only when it is one click. A link from a span to a profile query filtered by that span's identifier, over the span's time range, is the integration that turns continuous profiling from a specialist tool into something every engineer reading a slow trace uses.

def profile_link_for_span(trace_id: str, span_id: str, start_ns: int, end_ns: int) -> str:
    return ("https://profiles.internal/explore"
            f"?trace_id={trace_id}&span_id={span_id}"
            f"&from={start_ns // 1_000_000}&to={end_ns // 1_000_000}")

Expected Output: the stacks from inside one slow span.

span pricing.calculate  720 ms  samples 72
   56  dateutil.parser:_parse
    9  pricing.rules:apply_discount
    4  decimal:quantize
    3  <other>

Step 5 — Read short spans through aggregates. At a hundred hertz a sample is taken every ten milliseconds, so a four millisecond span usually has none. This is not a gap in the integration; it is how sampling works. Short spans are understood by aggregating over many of them — every cache.get span in the last hour, for example — rather than by opening one.

What this changes about investigations

The practical effect of the link is to shorten the path from symptom to code, and it is worth being concrete about how much.

Without it, a slow span leads to a hypothesis: the pricing calculation is slow, perhaps because of the discount rules. Testing that hypothesis means reproducing the request, adding timing, or taking a profile of the whole process and hoping the relevant frames stand out. Each step takes time and several of them require the problem to still be happening.

With it, the slow span leads directly to the stacks sampled inside it, from the moment it happened, on the instance where it happened. Seventy-eight percent of the span's samples in date parsing is not a hypothesis; it is the answer, and it arrives from a click rather than from an afternoon.

The reverse direction is equally valuable and less often used. Starting from an expensive frame in a fleet-wide profile and asking which endpoints produce it turns a vague optimisation target into a specific one: date parsing costs nine percent of the fleet's CPU, and ninety-one percent of that comes from one endpoint. That is a change with an owner and a measurable outcome, rather than a general sense that date parsing should be faster.

Both directions depend on the same thing — the span identifiers recorded with each sample — which is why the setting that enables it is the one worth insisting on when configuring an agent. The machinery underneath is the same context propagation the tracing SDK already maintains, so the cost of adding it is small.

Async services need this most

In a threaded service the stack itself goes a long way towards attributing a sample: each thread handles one request at a time, and the handler's frames identify it. In an asyncio service that attribution disappears. Every coroutine runs on the single loop thread, so a sample of that thread shows whichever coroutine happened to be executing, beneath the loop's own frames, and nothing in the stack says which request it belonged to.

Span context restores the attribution. The current span is held in a context variable, and context variables in asyncio are per task, so reading the variable at the moment of sampling gives the span of exactly the coroutine that was executing. The publishing mechanism in step 1 works unchanged, with one adjustment: the key must be the task rather than the thread, since many tasks share one thread over time. An agent that understands tasks does this internally.

The payoff is correspondingly larger. In an async service, a whole-process profile often shows a relatively flat picture across many coroutines and is hard to act on. The same profile filtered by route shows each endpoint's own shape, and filtered by a single slow span it shows what that one request's coroutine was doing while it held the loop — which, when the problem is a blocking call, is the call itself.

Configuration options

Verification

Confirm the labels are present and that they match real spans.

# pick a recent slow span and confirm the profile has samples for it
span = find_recent_span(service="checkout", min_duration_ms=500)
samples = query_profile(trace_id=span.trace_id, span_id=span.span_id)
print(f"span {span.name} {span.duration_ms} ms -> {samples.total} samples")
assert samples.total >= span.duration_ms // 10 * 0.7, "coverage lower than expected"

Expected Output: a sample count roughly equal to the span's duration divided by the sampling interval.

span pricing.calculate 720 ms -> 72 samples

Zero samples for a long span means the identifiers are not being recorded, or are recorded from the wrong thread's context — both of which produce profiles that look normal and cannot be joined to anything.

Common mistakes

Reading the sampler's own context. Error signature: every sample labelled with no span, or with the same span. Root cause: the context was read on the sampling thread rather than for the sampled one. Remediation: publish per-thread span identifiers and read those, as in step 1.

Using the trace identifier as a grouping label. Error signature: a profile store whose index grows with traffic. Root cause: a per-request value used as a dimension. Remediation: group by route template; filter by trace identifier.

Expecting samples for every span. Error signature: a conclusion that the integration is broken because fast spans are empty. Root cause: sampling interval longer than the span. Remediation: read fast spans in aggregate.

No link from the trace viewer. Error signature: an integration that exists and is never used. Root cause: the correlation requires a manual query. Remediation: add a link from each span to its profile, so the path is one click.

Route labels from rendered paths. Error signature: an endpoint label with thousands of values. Root cause: using the request path rather than the route template. Remediation: take the template from the framework's routing, which is what the span attribute should already carry.

Frequently Asked Questions