Python Observability

Propagating Context Across Thread and Process Pools

OpenTelemetry keeps the current span in a context variable, and context variables are per thread and per task. Work submitted to a ThreadPoolExecutor runs on a worker thread whose context is empty, so any span it starts has no parent and becomes the root of an unrelated trace. Work sent to a process pool is further away still: the context object cannot cross the process boundary at all. This page covers carrying context into both. It is a task article under context propagation and baggage, part of the distributed tracing and OpenTelemetry in Python section.

Connected trace or orphan roots A request handler runs inside a server span and submits three pieces of work to a thread pool. Without context propagation, each worker thread starts with an empty context, so the spans the workers create have no parent: the trace for the request shows only the server span with an unexplained gap where the work happened, and three separate single-span traces appear elsewhere, each a root with no connection to the request. With propagation, the current context is captured at submission and attached inside each worker, so the three worker spans are children of the server span and the trace shows the whole request, including the parallel work and how long each piece took. The note records that the unpropagated version loses both the timing breakdown and the ability to find the work from the request. one request, three pieces of pooled work plain submit() GET /report — with an unexplained gap root: fetch A root: fetch B root: fetch C three orphan traces nobody will find from the request propagated GET /report fetch A fetch B fetch C — the slow one, now visible propagation restores both the timing breakdown and the path from request to work
Without propagation the request's trace has a hole and the work appears as disconnected roots. With it, the slow piece of parallel work is visible inside the request that waited for it.

Prerequisites

pip install "opentelemetry-api>=1.27.0,<2.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Implementation

Step 1 — Capture the context where the span is current. The submitting thread is the only place the request's span is the current span. context.get_current() returns an immutable snapshot of the whole context — span, baggage and anything else stored there — which can be carried to another thread safely.

from opentelemetry import context, trace

tracer = trace.get_tracer("reports")

def submit_with_context(pool, fn, *args, **kwargs):
    ctx = context.get_current()                    # 1. captured on the caller's thread
    return pool.submit(_run_in_context, ctx, fn, *args, **kwargs)

Step 2 — Attach inside the worker, detach afterwards. context.attach makes the captured context current on the worker thread and returns a token; context.detach restores what was there before. Detaching in a finally block matters because pool threads are reused: a context left attached would become the parent of whatever the thread runs next.

def _run_in_context(ctx, fn, *args, **kwargs):
    token = context.attach(ctx)                    # 2. the request's span is now current
    try:
        return fn(*args, **kwargs)
    finally:
        context.detach(token)                      # 3. clean for the next task

Step 3 — Prefer asyncio.to_thread from async code. asyncio.to_thread copies the current context into the call automatically, so no wrapper is needed. loop.run_in_executor does not, which is a frequent source of orphan spans in async services that offload blocking calls.

import asyncio

async def build_report(ids):
    with tracer.start_as_current_span("build report"):
        # context carried automatically
        rows = await asyncio.to_thread(load_rows, ids)
        return render(rows)

Step 4 — Serialise context for process pools. A process has its own memory, and context objects cannot be pickled. The propagator used for HTTP headers works equally well here: inject the context into a dictionary, send the dictionary with the work, and extract it in the worker. The worker's span then has the right trace identifier and parent.

from concurrent.futures import ProcessPoolExecutor
from opentelemetry.propagate import inject, extract

def submit_to_process(pool: ProcessPoolExecutor, fn, *args):
    carrier: dict[str, str] = {}
    inject(carrier)                                # traceparent (+ baggage) as strings
    return pool.submit(_process_entry, carrier, fn, *args)

def _process_entry(carrier, fn, *args):
    ctx = extract(carrier)
    with tracer.start_as_current_span(fn.__name__, context=ctx):
        return fn(*args)

The worker process also needs its own tracer provider, initialised in the pool's initialiser, because providers do not travel across a spawn and a forked provider's export thread does not survive the fork.

Step 5 — Wrap the executor once. Relying on every call site to use the helper fails the first time someone writes pool.submit directly. An executor subclass that propagates context on every submission makes it the default.

from concurrent.futures import ThreadPoolExecutor

class ContextThreadPoolExecutor(ThreadPoolExecutor):
    def submit(self, fn, /, *args, **kwargs):
        ctx = context.get_current()
        return super().submit(_run_in_context, ctx, fn, *args, **kwargs)

Expected Output: worker spans sharing the request's trace identifier, parented to its span.

trace 9f2a71c4…  GET /report          span 4b1e77a2  parent -
trace 9f2a71c4…  fetch A              span 7c30a1d9  parent 4b1e77a2
trace 9f2a71c4…  fetch B              span 1e84b6f0  parent 4b1e77a2
trace 9f2a71c4…  fetch C              span a9d2c741  parent 4b1e77a2
Why detach matters A single pool thread runs a task for request A and then, later, a task for request B. In the version without detach, the thread attaches A's context at the start of A's task and never restores the previous state. When B's task arrives through a helper that does not attach anything — a code path that forgot the wrapper — the thread's current context is still A's, so B's work creates spans parented to A's request span. A's trace gains spans that happened seconds after A finished, and B's trace is missing them. In the version with detach in a finally block, the thread returns to an empty context after A's task, and anything run later either attaches its own context or has none, but never inherits A's. The note records that the leak is intermittent and depends on which thread picks up which task, which makes it very hard to diagnose from the traces alone. one pool thread, two requests attach without detach task for A — attaches A later task for B — still has A B's spans land in A's trace, seconds after A finished — intermittently detach in finally task for A — attach, detach task for B — clean context each task sees only the context it was given the leak depends on which thread picks up which task so it appears in some traces and not others, which makes it very hard to find from traces alone
Pool threads outlive tasks. A context attached and never detached becomes the parent of whatever that thread does next.

Once context reaches the worker, there is a choice about how the worker's span relates to the originating one, and it is worth making deliberately.

A child span says the work is part of the originating operation, and the operation's duration includes it. This is right for fan-out within a request — the report handler waiting on three fetches — where the parent genuinely cannot finish until the children do, and the trace's timeline should show that.

A link says the work is related to the originating operation without being inside it. This is right when the work outlives the request, when one piece of work serves many requests, or when the work is batched: a background flush triggered by a request, a worker processing a batch of items submitted by many different requests. Making such work a child produces traces whose root finishes long before its children, or a single batch span with a hundred parents, neither of which a trace viewer renders sensibly. The mechanics of links are covered in adding span links for batch work.

The deciding question is whether the originating operation waits for the work. If it does, child; if it does not, link.

What else travels in the context

The context object carries more than the current span, which is part of why propagating it whole is better than passing a trace identifier by hand.

Baggage — key-value pairs such as a tenant or a feature flag, described in using baggage for tenant and feature context — lives in the context. A worker that receives the whole context sees the baggage too, so tenant-aware sampling and attribute enrichment work unchanged inside the pool.

Suppression flags that instrumentation uses to avoid recording its own internal calls live there as well. Carrying the whole context keeps that behaviour consistent in the worker.

Anything else stored in context variables — structlog's bound fields, a request deadline — is not part of the OpenTelemetry context and needs its own propagation, or a contextvars.copy_context().run wrapper that copies all context variables at once. For thread pools, running the task inside a copied contextvars context carries everything, including the OpenTelemetry context, in one step.

import contextvars

def submit_all_context(pool, fn, *args):
    ctx = contextvars.copy_context()           # every context variable, not just OTel's
    return pool.submit(ctx.run, fn, *args)

Finding orphans in an existing service

Services that have been running for a while often have orphan spans already, and they are easy to find once you know what to look for.

The signature is a root span whose name is not an entry point. A service's legitimate roots are server spans for incoming requests, consumer spans for messages, and spans for scheduled jobs. A root named after an internal function — fetch_prices, render_pdf, load_rows — almost always started on a thread where context was not propagated. A query over the trace store for root spans grouped by name, filtered to exclude the known entry points, lists them directly, usually with an obvious culprit per name.

The second signature is a parent span with an unexplained gap: a request that takes eight hundred milliseconds with children accounting for fifty. The missing time is frequently spent in pooled work whose spans became orphans. Comparing the gap's timing with the start times of the orphan roots from the first query often matches them up exactly.

Fixing each one is a matter of finding the submission site and using the propagating executor, and replacing the service's executors with the subclass above fixes all of them at once. Adding a test that fails when any root span has a non-entry-point name keeps them from coming back — a cheap guard that belongs alongside the checks in testing spans with the in-memory span exporter.

What carries context across each boundary A table of four concurrency boundaries and how trace context crosses each. A ThreadPoolExecutor submit does not copy context variables by default, so the task runs with an empty context; wrapping the callable with contextvars.copy_context().run, or using asyncio's to_thread which copies automatically, carries it. asyncio.to_thread copies the current context itself, so no extra work is needed. A ProcessPoolExecutor starts a separate interpreter where context variables do not exist; the parent must inject the context into a carrier dictionary with the propagator and pass it as an argument, and the child extracts it. A multiprocessing Pool behaves the same way as a process pool. The note says threads need the context copied, processes need it serialised. boundary context by default what carries it ThreadPoolExecutor.submit lost — empty context copy_context().run wrapper asyncio.to_thread copied automatically nothing extra needed ProcessPoolExecutor absent — new interpreter inject to a dict, extract in child multiprocessing.Pool absent — new interpreter inject to a dict, extract in child threads need the context copied; processes need it serialised a span started without either becomes a new root — an orphan trace
Threads share memory but not context variables; processes share neither. Each boundary needs its own carrier.

Configuration options

Situation Mechanism Note
Async code offloading a call asyncio.to_thread copies context automatically
run_in_executor wrap with captured context does not copy by default
ThreadPoolExecutor attach/detach wrapper or subclass detach in finally
All context variables contextvars.copy_context().run includes structlog fields
ProcessPoolExecutor inject into a dict, extract in worker objects cannot be pickled
Worker provider initialise in the pool initialiser not inherited under spawn
Relationship child if awaited, link if not keeps traces readable

Verification

Assert in a test that pooled spans share the parent's trace.

from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

def test_pool_spans_join_the_request_trace(tracer, exporter: InMemorySpanExporter):
    with ContextThreadPoolExecutor(4) as pool, tracer.start_as_current_span("request") as root:
        futures = [pool.submit(lambda: tracer.start_span("work").end()) for _ in range(3)]
        [f.result() for f in futures]
    spans = exporter.get_finished_spans()
    work = [s for s in spans if s.name == "work"]
    assert all(s.context.trace_id == root.get_span_context().trace_id for s in work)
    assert all(s.parent.span_id == root.get_span_context().span_id for s in work)

Expected Output:

.
1 passed

Common mistakes

Plain submit. Error signature: worker spans as separate root traces. Root cause: the worker thread's context is empty. Remediation: capture and attach, or use the subclass.

run_in_executor assumed to propagate. Error signature: orphan spans from offloaded blocking calls in async code. Root cause: it does not copy context. Remediation: asyncio.to_thread, or wrap the call.

No detach. Error signature: spans appearing in the wrong request's trace, intermittently. Root cause: a pool thread keeping an old context. Remediation: detach in finally.

Passing a Context to a process. Error signature: a pickling error, or a trace identifier passed by hand with no parent span. Root cause: context objects cannot cross processes. Remediation: inject into a carrier and extract.

Children for work that outlives the request. Error signature: traces whose root ends long before its children. Root cause: background work modelled as part of the request. Remediation: use a link.

Frequently Asked Questions

Why do spans from my thread pool appear as separate traces?

Because the worker thread has its own context, which is empty. ThreadPoolExecutor.submit does not copy the caller's context, so a span started in the worker has no parent and becomes the root of a new trace.

Does asyncio.to_thread propagate context?

Yes. It copies the current context into the call it runs on the thread, so spans and structlog bound fields are present. loop.run_in_executor does not do this by default, which is a common source of confusion.

Can I pass a Context object to a process pool?

No. Context objects are not picklable and would be meaningless in another process anyway. Inject the context into a dictionary with the configured propagator, send the dictionary, and extract it in the worker.

Should the worker's span be a child or a link?

A child when the work is part of the request and the request waits for it. A link when the work is batched or outlives the request, so it is related to the originating trace without pretending to be inside it.

Related Guides