Propagating Trace Context Across Celery Tasks

When a web request fans out work to a Celery worker, the worker's span becomes a disconnected root unless the producer's traceparent rides along in the message headers, and this page shows the exact instrumentation that keeps trace_id continuity intact across the broker. It is written for backend engineers and SREs who already export spans from their web tier and now watch the trace stop dead at .delay(). It is a focused task within the context propagation and baggage guide, part of the Distributed Tracing and OpenTelemetry in Python guide.

Reliable continuity depends on four things being true at once. The instrumentation package must patch Celery's signals so headers are written and read automatically. The global propagator must serialize W3C TraceContext into the message payload. Every worker process — not just the parent that runs celery -A — must own a live span processor and exporter. And the worker must restore the parent context before your task body runs so child spans attach correctly.

Prerequisites

Pin the instrumentation and exporter so the producer and worker agree on the wire format. A version skew between opentelemetry-api and the instrumentation package is the most common source of silent propagation failures: the instrumentation is published on a 0.x track that must match the 1.x API release it was built against.

pip install \
  "opentelemetry-sdk>=1.30.0,<2.0.0" \
  "opentelemetry-instrumentation-celery>=0.51b0,<1.0.0" \
  "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0" \
  "celery>=5.3.0,<6.0.0"

Set the export endpoint and service name through environment variables so the same module runs unchanged in the producer and the worker. Define OTEL_SERVICE_NAME distinctly per role so producer and consumer spans are attributable, and export the same values in both deployments so the two halves of the trace land in one backend.

# Producer (web tier)
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="checkout-api"

# Worker
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="payments-worker"

Implementation

Establish a baseline SDK configuration before instrumenting Celery. Initializing the TracerProvider after worker startup triggers NoOpTracer fallbacks and severs the trace chain, so the ordering below is not stylistic — it is what makes the signals bind to a real tracer.

1. Register the W3C propagator globally. Call set_global_textmap with a composite of TraceContextTextMapPropagator and W3CBaggagePropagator before the Celery app is instantiated. This is the same propagator the rest of your services use, which is what keeps trace_id consistent end to end; it mirrors the standard context propagation and baggage mechanics used over HTTP.

2. Instrument the Celery app once. Call CeleryInstrumentor().instrument() in the main process, after the Celery() object exists. The instrumentation hooks before_task_publish to inject traceparent, tracestate, and baggage into the message headers, and hooks task_prerun to extract them and start a child span. Manual header manipulation is unnecessary and actively breaks extraction.

3. Create the provider per worker process. Attach a TracerProvider with a BatchSpanProcessor inside a worker_process_init handler rather than at module import. The prefork pool forks after the module is loaded, and background threads do not survive os.fork(), so a provider built in the parent leaves each child queuing spans into a processor nobody drains.

4. Set baggage before dispatch. Any baggage you set inside an active span before calling .delay() is serialized into the baggage header and restored on the worker. This is how request-scoped values such as a tenant identifier travel to the worker without being smuggled through task arguments, where they would pollute the task signature and every retry payload.

5. Read the restored context inside the task. By the time your function body executes, task_prerun has already attached the extracted context, so trace.get_current_span() returns the worker span and baggage.get_all() returns what the producer set. Add task-specific attributes here rather than re-creating a root span.

# worker_tasks.py
import os
from celery import Celery
from celery.signals import worker_process_init
from opentelemetry import trace, baggage
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.instrumentation.celery import CeleryInstrumentor

# 1. Global propagator: trace context + baggage, registered BEFORE Celery()
set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    W3CBaggagePropagator(),
]))

# 2. Instrument once in the main process, after the app object exists
app = Celery("worker_tasks", broker="redis://localhost:6379/0")
CeleryInstrumentor().instrument()


# 3. Build the provider in each forked child (threads do not survive fork)
@worker_process_init.connect(weak=False)
def init_tracing(*args, **kwargs):
    provider = TracerProvider()
    exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
    )
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)


tracer = trace.get_tracer(__name__)


@app.task(bind=True, max_retries=3)
def process_payment(self, order_id: str):
    # 5. Parent context already restored by CeleryInstrumentor on task_prerun
    span = trace.get_current_span()
    span.set_attribute("payment.order_id", order_id)
    span.set_attribute("messaging.message.id", self.request.id)
    current_baggage = baggage.get_all()
    print(f"Processing {order_id} with baggage: {current_baggage}")
    return {"status": "completed", "order_id": order_id}

The producer process needs the same propagator registration and its own provider, but it does not fork, so a module-level provider is fine there. Set baggage inside a span and dispatch the task; the instrumentation injects the headers on publish without any further work.

# producer.py
from opentelemetry import baggage, context
from worker_tasks import process_payment, tracer

with tracer.start_as_current_span("checkout"):
    # 4. Baggage attached to the active context is serialized into headers
    ctx = baggage.set_baggage("tenant_id", "acme_corp")
    ctx = baggage.set_baggage("request_source", "api_gateway", context=ctx)
    token = context.attach(ctx)
    try:
        process_payment.delay("order_12345")
    finally:
        context.detach(token)

Expected Output (producer console, ConsoleSpanExporter):

{"name": "checkout", "trace_id": "0x4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "0x00f067aa0ba902b7", "parent_id": null}
{"name": "apply_async/worker_tasks.process_payment", "trace_id": "0x4bf92f3577b34da6a3ce929d0e0e4736", "parent_id": "0x00f067aa0ba902b7"}

Pool type changes where the provider must live

The worker_process_init handler is what makes the default prefork pool work, and it is harmless on the other pools. With --pool=threads, the worker never forks, so the signal fires once and the shared provider serves every thread; OpenTelemetry's context is stored in contextvars, which are copied per thread by the pool, so each task still sees only its own parent. With --pool=gevent or --pool=eventlet, monkey-patching must run before the OTLP exporter's gRPC channel is created, otherwise the exporter's socket blocks the hub — the same event-loop discipline covered in the async tracing patterns guide. Whichever pool you choose, add a worker_shutdown handler that calls provider.shutdown() so the batch processor flushes its final queue instead of dropping the last few seconds of spans on a rolling deploy.

Retries, countdowns, and canvas workflows

A retry is a fresh publish of the same message, so the original headers travel with it and every attempt lands under the producer's trace_id as a new child span. That is usually what you want: one trace shows the failed attempt, the backoff gap, and the eventual success. It also means a task retried against a sampled-out trace stays sampled out — the sampling flag lives in the traceparent flags byte, so head sampling decided at the web tier propagates to every downstream attempt. If worker failures are the thing you most need to see, weigh that against the sampling strategy you run at the edge.

Canvas primitives follow the same rule, one hop at a time. In a chain(a.s(), b.s()), task b is published by the worker running a, so b's span parents to a's span rather than to the original request. A group fans out from whichever process dispatched it and produces a sibling span per task; a chord callback is published by the last member to finish. The result is a trace whose shape mirrors the actual causality, which is exactly what you want when diagnosing where a pipeline stalled.

Long countdown or eta schedules are the one case where parent-child linkage is a poor fit. A task scheduled six hours out inherits a traceparent whose producer span ended long ago, and most backends will render a six-hour trace with one span at each end. For anything beyond a short retry window, prefer starting a new trace on the worker and recording the producer context as a span link, so the two are still navigable without distorting duration percentiles.

from opentelemetry import trace
from opentelemetry.trace import Link, SpanKind

@app.task(bind=True)
def nightly_reconcile(self, batch_id: str):
    # Detach from the (long-stale) producer parent, but keep a link to it
    producer_ctx = trace.get_current_span().get_span_context()
    with tracer.start_as_current_span(
        "nightly_reconcile",
        kind=SpanKind.CONSUMER,
        context=trace.set_span_in_context(trace.INVALID_SPAN),
        links=[Link(producer_ctx)],
    ) as span:
        span.set_attribute("batch.id", batch_id)

Configuration options

Verification

Run the worker and dispatch a task, then confirm the worker restores the baggage that the producer set. The presence of the baggage values proves the headers crossed the broker and were extracted before the task body ran.

Expected Output (Worker Execution):

Processing order_12345 with baggage: {'tenant_id': 'acme_corp', 'request_source': 'api_gateway'}

For wire-level confirmation, run the worker with OTEL_LOG_LEVEL=debug and look for the inject and extract events in stdout. Absence of these lines confirms a signal-hook failure rather than a broker problem.

Expected Output (debug log):

[celery] Injecting trace context into headers traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
[celery] Extracting trace context from headers traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The authoritative check is the exported payload. Both spans must carry the same trace_id, and the consumer span's parentSpanId must equal the producer span's spanId.

Expected Output (OTLP collector payload, trimmed):

[
  {
    "name": "apply_async/worker_tasks.process_payment",
    "kind": "SPAN_KIND_PRODUCER",
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "00f067aa0ba902b7",
    "parentSpanId": "8c4f1e0b2a9d7c31",
    "attributes": {"service.name": "checkout-api", "messaging.system": "celery"}
  },
  {
    "name": "run/worker_tasks.process_payment",
    "kind": "SPAN_KIND_CONSUMER",
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
    "spanId": "b7ad6b7169203331",
    "parentSpanId": "00f067aa0ba902b7",
    "attributes": {"service.name": "payments-worker", "celery.state": "SUCCESS"}
  }
]

Finally, turn the check into a standing query rather than a one-off. Filtering the worker service for parent_id = null surfaces every orphaned span, which is the fastest way to catch a deploy that shipped a worker image without the instrumentation. Pair that with trace IDs in your log records so a broken hop is visible from the logs alone, without opening the trace UI.

Common mistakes

• Error signature: worker logs emit NoOpTracer warnings and every process_payment span appears as an isolated root. Root cause: CeleryInstrumentor().instrument() never ran in the worker process, so task_prerun was never patched and nothing extracted the headers. Remediation: call instrument() at module scope immediately after Celery(), in the module celery -A imports, so the signals are patched before any task is registered or consumed.

• Error signature: spans are silently missing from the worker only when running the default pool, and reappear under --pool=solo. Root cause: the TracerProvider and its BatchSpanProcessor were created at import time in the parent, and the exporter's background thread did not survive the prefork os.fork(), so each child enqueues spans that are never flushed. Remediation: move provider construction into a worker_process_init handler, and shut the provider down on worker_shutdown so the final batch is flushed — the same per-process ownership rule that governs logging across multiple processes.

• Error signature: KeyError: 'traceparent' during extraction, or worker spans with a parent_id that matches nothing in the backend. Root cause: application code is stamping its own headers or writing kwargs['trace_id'], overwriting or shadowing what the propagator serialized. Remediation: remove every manual header assignment and let the composite propagator own the headers dict; if you need extra fields, put them in baggage rather than alongside traceparent.

• Error signature: the producer's HTTP span and the worker span share no trace_id, and the producer trace ends at the route handler. Root cause: the dispatching process configured a provider but never registered the global propagator, so inject() had no format to write with. Remediation: call set_global_textmap in both deployments before the app object is built — the same registration that makes web framework instrumentation emit usable traceparent headers on outbound calls.

Related

• Context propagation and baggage — the parent guide covering inject, extract, attach, and detach in detail.
• Distributed Tracing and OpenTelemetry in Python — the full tracing guide this task belongs to.
• Instrumenting aiohttp client requests — the same context handoff over HTTP instead of a broker.
• OpenTelemetry SDK setup — provider lifecycle and processor tuning that the worker export path depends on.
• Sampling strategies for distributed tracing — why the sampling flag in traceparent decides whether worker spans are kept.
• Adding trace IDs to log records — correlate worker logs with the trace that dispatched the task.

Frequently Asked Questions