Python Observability

Debugging Missing Spans in Python

"The span isn't there" has many causes and one symptom, which is why missing-span investigations so often go in circles. A span that should exist and does not was either never created, never exported, or never stored — and each of those breaks down further. The productive approach is to check them in order, starting inside the process, because most causes live there and the checks are cheap. This page gives that order and the check for each step. It is a task article under testing and validating instrumentation, part of the distributed tracing and OpenTelemetry in Python section.

Five checks, in order Five checks are drawn as a sequence, each eliminating a class of cause. First, was the span created: instrumentation not installed, a library imported before patching, or code on a path that is not instrumented. Second, was it sampled: a low ratio, an unsampled parent from upstream, or a tail policy. Third, does the process have a working provider: a prefork worker whose export thread was lost, a second provider created somewhere, or a provider shut down early. Fourth, did export succeed: exporter errors, queue drops, or collector refusals and drops. Fifth, is the query right: wrong service name, wrong time window, or a filter on an attribute with a different name. The first three are inside the process and account for most cases; the note records that starting at the collector or the backend, as investigations commonly do, looks in the least likely place first. check in this order — the cheapest and most likely first 1 · created? not installed imported early uninstrumented path 2 · sampled? low ratio unsampled parent tail policy 3 · provider? thread lost to fork second provider shut down early 4 · exported? exporter errors queue drops collector drops 5 · query? wrong name wrong window wrong attribute inside the process — most causes, cheapest checks pipeline and store investigations usually start at the backend, which is the least likely place starting in the process eliminates most of the search space in a few minutes
Each check eliminates a class of causes. Working left to right starts where the causes usually are and where checking costs least.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

Access to the process — a shell in the container, or the ability to set environment variables and restart one instance — makes every step below faster.

Implementation

Step 1 — Check whether the span was created. The quickest test is to add a console exporter alongside the real one on a single instance and make the request. Spans printed to standard output exist; the problem is downstream. No output means the span was never created — the instrumentation package is missing, the library was imported before the launcher patched it, or the code path is not instrumented at all.

from opentelemetry import trace
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

# temporarily, on one instance only
trace.get_tracer_provider().add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

Step 2 — Check the sampling decision. A span that is not sampled is created as a non-recording span and never exported — by design. A low ratio makes a particular trace missing most of the time. A parent-based sampler inherits an unsampled decision from upstream, so a service that samples at one hundred percent still drops traces that arrive marked unsampled. Checking the sampled flag of the current span context answers the question directly.

span = trace.get_current_span()
ctx = span.get_span_context()
print("recording:", span.is_recording(), "sampled:", ctx.trace_flags.sampled)

Expected Output: a trace that was not sampled, which ends the investigation.

recording: False sampled: False

Step 3 — Check the provider in the process that handled the request. Under a prefork server the SDK configured in the master may be the only one, and the workers that actually handle requests inherited its queue without its export thread. Spans are created, sampled, queued and never exported, silently. Counting live threads in a worker, or checking that a post-fork hook initialised the provider, settles it. The fix is described in zero-code instrumentation with opentelemetry-instrument.

import threading
print([t.name for t in threading.enumerate()])
# a healthy worker lists an export thread for the batch span processor;
# a worker that lost it after fork lists only MainThread

Step 4 — Check the export path. The SDK logs exporter failures at warning level and counts queue drops; the collector counts accepted, refused, sent and dropped spans per pipeline. Reading them in order locates the tier where spans stopped, as set out in detecting dropped spans and metrics.

kubectl logs deploy/checkout | grep -i "opentelemetry.*\(export\|dropp\)" | tail -5
curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted|exporter_sent|processor_refused)_spans'

Step 5 — Check the query. A span that exists in the store and is not found by a query looks identical to a span that does not exist. Searching by trace identifier — printed from the request, or taken from a log line — bypasses every filter. If the trace is found that way, the original query was wrong: a different service name, a time window in the wrong timezone, or an attribute filter using a name the spans do not carry.

A trace that stops partway A trace shows the gateway and the orders service but not the inventory service that orders called. Two explanations are possible and they need different fixes. In the first, propagation broke: orders did not inject the trace context, or inventory did not extract it, so inventory's spans exist in the store under a different trace identifier as the root of their own trace. Searching for inventory spans in the same time window, without filtering by trace, finds them, each a root with no parent. In the second, inventory's spans were created in the right trace but dropped by inventory's own pipeline, so searching the time window finds nothing from inventory at all. The note records that one query — inventory spans in the window, unfiltered by trace — distinguishes the two in seconds. the trace shows gateway and orders — inventory is missing gateway orders → calls inventory inventory — ? propagation broke inventory spans exist as roots of other traces fix: inject / extract spans dropped no inventory spans in the window at all fix: inventory's pipeline one query separates them inventory spans in the same time window, not filtered by trace identifier roots with no parent mean propagation; nothing at all means the pipeline
A trace that ends at a boundary is either a propagation failure or a delivery failure downstream. The downstream service's spans in the same window say which.

Patterns worth recognising

Certain symptoms map to specific causes often enough that recognising them shortens the investigation considerably.

Spans from the web tier, none from workers. A prefork or task-queue deployment where the export thread did not survive the fork. Nearly always step 3.

Spans in development, none in production. An exporter endpoint, a sampler, or a network policy that differs between environments. The environment variables in the running production container, compared with development, usually show it immediately.

Some traces complete, others missing a service. Parent-based sampling combined with a service that samples differently, or propagation through a path — a queue, a pool — that only some requests take. Checking whether the missing service's spans exist as separate roots distinguishes the two.

Spans present, attributes missing. Not a missing-span problem at all but an instrumentation version change that renamed attributes, or a query using an old name. Covered in naming spans and using semantic conventions.

Spans missing only during high traffic. Queue drops in the SDK or refusals in the collector, which are load-dependent by nature. The drop counters confirm it, and the fix is queue sizing or sampling, not instrumentation.

The last few seconds before a restart are always missing. No flush on shutdown, the subject of graceful shutdown and telemetry flush.

Turning the investigation into a permanent check

Every missing-span investigation reveals a gap that will recur if nothing watches for it. Three small investments turn a one-off diagnosis into ongoing protection.

A debug switch rather than a code change. The console exporter in step 1 is most useful when it can be enabled on one instance without a deploy. An environment variable read at startup — or a protected internal endpoint that attaches a console processor at runtime — makes the first check a matter of seconds during the next incident rather than a build and rollout.

Export-thread health as a metric. The prefork failure from step 3 is the most common cause and the least visible. A gauge that reports whether the batch span processor's worker thread is alive, per process, turns it into a dashboard line; a process reporting zero is a process whose spans are going nowhere. The same gauge catches a provider shut down too early, which produces the same silence for a different reason.

A synthetic trace per deploy. The end-to-end check described in testing and validating instrumentation, run after every deploy, catches the delivery failures in steps 3 and 4 before anyone needs a real trace. It costs one request and one query, and it would have prevented most missing-span investigations before they started.

The combination changes the character of the problem. Instead of an engineer discovering during an incident that the trace they need does not exist, a dashboard shows the gap the moment it opens, and the deploy that caused it is the one that just finished.

Missing spans versus missing context

It is worth separating two problems that are often reported the same way. A missing span means the span does not exist in the store at all. Missing context means the span exists but is disconnected from the trace it should belong to — an orphan root, a downstream service in a separate trace. The symptom, "I can't see this in the trace", is identical.

The distinction matters because the fixes are unrelated. Missing spans are fixed in configuration, sampling and the export path, as above. Missing context is fixed in propagation: injecting and extracting at boundaries, carrying context into pools and tasks, as covered in propagating context across thread and process pools. A search for the expected span by name and time window, without filtering by trace, settles which problem it is in one query, and is worth running before either kind of fix.

Symptom to first check A table mapping what is observed to the first thing to check. No spans at all from a service: the exporter endpoint and whether a tracer provider was ever set. Spans in staging but not production: the sampler configuration and environment variables that differ. Spans from some workers but not others: whether the provider was created before a fork. Spans for requests but none for background jobs: whether the job entry point runs the telemetry setup. Some spans in each trace missing: whether the batch processor is dropping because its queue is full. The note says starting from the symptom skips most of the checklist. what you see check first no spans at all endpoint · was a provider set? staging yes, production no sampler and env var differences some workers, not others provider created before fork requests yes, jobs no job entry point skips setup some spans in each trace batch queue full and dropping the shape of what is missing points at the cause start from the symptom and most of the checklist can be skipped
Which spans are missing, and where, narrows the cause before any code is read.

Configuration options

Check Tool Tells you
Created console exporter on one instance instrumentation or pipeline
Sampled trace_flags.sampled by-design drop or not
Provider alive thread list, post-fork hook export thread present
Exported SDK warnings, drop counters process-side delivery
Collector accepted vs sent counters which tier dropped
Stored search by trace identifier query or data
Boundary downstream spans, unfiltered propagation or delivery

Verification

The investigation is complete when a single request can be followed from creation to storage. Record its trace identifier at the source and find it in the backend.

with trace.get_tracer("probe").start_as_current_span("missing-span probe") as span:
    print(f"trace_id={span.get_span_context().trace_id:032x}")

Expected Output: an identifier that, searched in the backend within a minute, returns the span.

trace_id=5d2e81c0b9f44a7e8c3a1f9b0d6e2c47

Common mistakes

Starting at the backend. Error signature: hours spent on queries and collector configuration for a span that was never created. Root cause: checking the least likely place first. Remediation: start inside the process.

Forgetting the sampler. Error signature: an investigation into a trace that was dropped by design. Root cause: not checking the sampled flag. Remediation: check it second, before the pipeline.

Assuming the master's configuration reaches workers. Error signature: no spans from any worker. Root cause: export thread lost at fork. Remediation: initialise the provider after fork.

Querying by attribute. Error signature: a span "missing" that is present under a renamed attribute. Root cause: a filter that no longer matches. Remediation: search by trace identifier first.

Leaving the console exporter on. Error signature: standard output flooded with span dumps after the investigation. Root cause: a debugging change never reverted. Remediation: add it through an environment flag, and remove it when done.

Frequently Asked Questions

What is the most common cause of missing spans?

In prefork servers, the export thread. The SDK is configured in the master, the master forks, and workers inherit a queue with no thread draining it. Spans are created and never exported, with no error anywhere.

How do I tell whether a span was created?

Add a console exporter temporarily, or check the SDK's span processor counters. If the span appears on the console it was created and the problem is downstream; if it does not, the problem is instrumentation, sampling or configuration.

Could the sampler be responsible?

Very often. A ratio sampler with a low rate, a parent-based sampler receiving an unsampled parent from upstream, or a tail sampler at the collector will all drop traces by design. Check the sampled flag on the trace context before looking anywhere else.

Why does a trace show some services and not others?

Either propagation broke at the boundary where it stops — the downstream service started a new trace — or the missing service's spans were dropped by its own pipeline. Searching for the downstream service's spans in the same time window, without filtering by trace identifier, tells the two apart.

Related Guides