Testing and Validating OpenTelemetry Instrumentation in Python

Instrumentation is code, and it regresses like code. A refactor moves a call onto a thread pool and its spans become orphan roots; a dependency upgrade renames an attribute and a dashboard goes blank; a new middleware starts its own span and every request now has two server spans. The service keeps working throughout, which is why these regressions reach production and are discovered by the engineer who needed the trace. This guide covers testing instrumentation in-process, validating it in a running environment, and diagnosing it when spans go missing. It is part of the distributed tracing and OpenTelemetry in Python section, and its child pages cover testing spans with the in-memory span exporter and debugging missing spans in Python.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "pytest>=8.0.0,<9.0.0" \
            "pytest-asyncio>=0.23.0,<1.0.0"

Concept and architecture

Tracing instrumentation has three properties that can each break independently, and a useful test suite checks each one.

Existence. The spans that should be created are created, with the expected names and kinds. This breaks when code moves, when an instrumentation package is removed or disabled, and when a sampler configured in tests differs from what was intended.

Structure. Spans form the expected tree: the database span is a child of the handler's span, the downstream call carries the same trace identifier, a fan-out's branches share a parent. This breaks at every boundary where context can be lost — thread pools, process pools, callbacks, message queues — and it is the property most often broken by refactoring, because moving a call onto a pool changes nothing about its behaviour and everything about its span's parent.

Content. The attributes that downstream consumers depend on are present with the right names, values and types, and errors are recorded on the right spans. This breaks with instrumentation upgrades that move to newer semantic convention names, with refactors that stop setting an attribute, and with code paths that raise without the span recording the exception.

The three break at different rates. Existence breaks occasionally, usually visibly. Content breaks with upgrades, predictably, and a changelog usually warns of it. Structure breaks constantly and silently, with ordinary refactoring, which is why structural assertions carry most of the value in a tracing test suite.

All three can be tested in-process, cheaply, with an in-memory exporter. A fourth property — delivery, that the trace arrives in the backend — can only be checked in a running environment, because the failures that break it are properties of the deployment: a prefork server that lost the export thread, a collector that drops, a network policy that blocks the exporter.

Step-by-step implementation

Step 1 — Install an in-memory exporter for tests. A tracer provider with a simple span processor and an in-memory exporter gives tests synchronous access to every finished span. The simple processor exports on span end, so assertions run immediately after the code under test without waiting for a batch.

# conftest.py
import pytest
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

_exporter = InMemorySpanExporter()
_provider = TracerProvider()
_provider.add_span_processor(SimpleSpanProcessor(_exporter))
trace.set_tracer_provider(_provider)              # once per test session


@pytest.fixture
def spans():
    _exporter.clear()
    yield _exporter
    _exporter.clear()

Step 2 — Assert on structure. A helper that builds the span tree from finished spans makes structural assertions readable: this span is the child of that one, these spans share a trace.

def tree(finished) -> dict[str, list[str]]:
    by_id = {s.context.span_id: s for s in finished}
    children: dict[str, list[str]] = {}
    for s in finished:
        parent = by_id.get(s.parent.span_id).name if s.parent and s.parent.span_id in by_id else "<root>"
        children.setdefault(parent, []).append(s.name)
    return children


def test_order_handler_span_tree(client, spans):
    client.post("/orders", json={"sku": "A1"})
    t = tree(spans.get_finished_spans())
    assert t["<root>"] == ["POST /orders"]
    assert set(t["POST /orders"]) >= {"reserve stock", "charge payment"}

Step 3 — Assert on the attributes that matter. Dashboards, alerts and sampling policies key on specific attributes. A test that checks them — name and type — catches the upgrade that renames them before it reaches production.

def test_server_span_attributes(client, spans):
    client.get("/orders/8812")
    server = next(s for s in spans.get_finished_spans() if s.kind == trace.SpanKind.SERVER)
    assert server.attributes["http.route"] == "/orders/{order_id}"
    assert isinstance(server.attributes["http.response.status_code"], int)

Step 4 — Test every boundary that carries context. Each place context can be lost deserves a test: an outbound HTTP call injects headers, a thread pool submission keeps its parent, a queued message carries its context. These tests are the ones that catch refactoring regressions, and the patterns they check are described in propagating context across thread and process pools.

Step 5 — Validate the deployed service end to end. A synthetic request through the deployed service, with a known header value, followed by a query to the backend for the resulting trace, checks the one property unit tests cannot: that spans actually leave the process and arrive. Running it after every deploy catches the prefork export problem and collector configuration changes.

import time, uuid, requests

def validate_deployment(base_url: str, backend_query) -> None:
    probe = uuid.uuid4().hex
    requests.get(f"{base_url}/orders/probe", headers={"x-probe-id": probe}, timeout=5)
    time.sleep(15)                                   # batch delay + pipeline
    trace_doc = backend_query(attribute="app.probe_id", value=probe)
    services = {s["service"] for s in trace_doc["spans"]}
    assert {"checkout", "inventory"} <= services, f"trace incomplete: {services}"

Step 6 — Fail the build on span-name cardinality. A test that exercises several requests with different identifiers and asserts that the number of distinct span names does not grow catches the regression where an identifier creeps into a span name — the failure described in naming spans and using semantic conventions.

Configuration reference

Async and concurrency considerations

Testing instrumentation in async code has one recurring difficulty: spans from concurrent tasks finish in whatever order the tasks complete, so assertions on span order are flaky. Asserting on the tree — parent relationships and sets of children — rather than on the order spans appear in the exporter makes async tests deterministic.

A second consideration is that background tasks started by a handler may still be running when the test's request returns. Their spans are not yet finished, and assertions that expect them will intermittently fail. Awaiting the application's background work explicitly in the test, or flushing a task registry, makes the test wait for exactly what it asserts on.

Thread pools behave similarly: work submitted by the handler may finish after the handler returns. Tests that exercise pooled work should wait for the pool — by shutting it down, or by awaiting the futures — before reading spans.

Finally, the global tracer provider can only be set once per process. Tests that need different providers — one with a different sampler, for instance — should use a provider passed explicitly to the tracer rather than the global one, or reset global state between tests with care. Most suites settle on one test-wide provider with a simple processor and in-memory exporter, which covers nearly every case.

Production code examples

A reusable assertion helper that makes instrumentation tests read like specifications:

# tests/tracing_asserts.py
from dataclasses import dataclass
from opentelemetry.trace import StatusCode


@dataclass
class SpanView:
    spans: list

    def named(self, name: str):
        matches = [s for s in self.spans if s.name == name]
        assert matches, f"no span named {name!r}; have {sorted({s.name for s in self.spans})}"
        return matches[0]

    def child_of(self, child: str, parent: str) -> None:
        c, p = self.named(child), self.named(parent)
        assert c.parent is not None and c.parent.span_id == p.context.span_id, \
            f"{child!r} is not a child of {parent!r}"

    def same_trace(self) -> None:
        traces = {s.context.trace_id for s in self.spans}
        assert len(traces) == 1, f"spans split across {len(traces)} traces"

    def errored(self, name: str, exc_type: str) -> None:
        s = self.named(name)
        assert s.status.status_code is StatusCode.ERROR
        assert any(e.attributes.get("exception.type", "").endswith(exc_type) for e in s.events)

def test_payment_failure_is_recorded_on_the_right_span(client, spans, gateway):
    gateway.fail_with(TimeoutError("read timed out"))
    client.post("/orders", json={"sku": "A1"})
    v = SpanView(spans.get_finished_spans())
    v.same_trace()
    v.child_of("charge payment", "POST /orders")
    v.errored("charge payment", "TimeoutError")

Expected Output: failures that say exactly what broke.

FAILED test_orders_tracing.py::test_payment_failure_is_recorded_on_the_right_span
AssertionError: spans split across 2 traces

Testing propagation between services

The boundary between two services is where tracing is most valuable and most often broken, and it can be tested without deploying both.

The approach is to run the downstream service's application in-process through a test client, and to make the upstream service's HTTP client call that test client instead of the network. The upstream code runs with its real instrumentation, which injects the trace context into the outgoing request's headers; the downstream test client receives those headers, and its instrumentation extracts them. Both services' spans land in the same in-memory exporter, and a single assertion checks that they share a trace and that the downstream server span is a child of the upstream client span.

def test_checkout_to_inventory_propagation(checkout_client, inventory_app, spans, monkeypatch):
    from starlette.testclient import TestClient
    inventory = TestClient(inventory_app)
    # route checkout's outbound call to the in-process inventory app
    monkeypatch.setattr("checkout.clients.inventory_base_url", "http://testserver")
    monkeypatch.setattr("checkout.clients.http", inventory)

    checkout_client.post("/orders", json={"sku": "A1"})

    v = SpanView(spans.get_finished_spans())
    v.same_trace()
    v.child_of("GET /stock/{sku}", "GET")      # inventory server under checkout client

The same pattern applies to queues: publish through the real producer instrumentation to an in-memory broker, consume through the real consumer instrumentation, and assert the consumer's span carries the producer's trace. It is the cheapest way to catch the class of regression where a library upgrade stops injecting headers, or a custom client wrapper drops them — failures that otherwise surface as traces that end abruptly at a service boundary, the symptom described in W3C Trace Context versus B3 propagation.

Instrumentation as part of code review

Tests catch regressions after they are written. A few review habits catch instrumentation problems as they are written, and they are cheap to adopt.

Look at every new pool submission, callback and background task. Each is a place context can be lost. A reviewer asking "does this propagate context?" at each one catches most orphan-span bugs before they merge, and pointing to the context-propagating executor makes the fix a one-word change.

Look at every new span name. Anything computed from input is a cardinality bug in waiting. A span name that is a string literal, or a small enumerated set, passes; one built with an f-string needs a second look.

Look at every new attribute name. Semantic convention names where one exists, a namespace prefix otherwise. An attribute that duplicates an existing concept under a new name splits every query that uses it.

Look at every new exception handler. A handler that catches, logs and returns a default hides the failure from the span unless it records it. Whether the span should show an error is a decision the author should make deliberately, and review is where that decision is visible.

These four questions take a minute per pull request and prevent most of the regressions the tests above are designed to catch — which is the right order: prevention in review, detection in tests, verification in the deployed environment.

Checks that belong in production

Some properties of instrumentation cannot be tested before deployment at all, because they depend on real traffic, real configuration and real infrastructure. Three continuous checks cover them.

Orphan root rate. The proportion of root spans whose names are not legitimate entry points. A healthy service's roots are server spans, consumer spans and scheduled jobs; anything else is a span that lost its parent. Tracking this rate per service turns context-propagation regressions into an alert rather than a discovery.

Span name cardinality. The number of distinct span names per service per day. It should be flat; growth means an identifier has entered a name, and the query in the naming article lists the offender.

Trace completeness for a synthetic flow. The end-to-end check from step 5, run on a schedule rather than only at deploy, catches delivery problems that appear between deploys — a collector configuration change, a certificate expiry, a network policy update. It is the tracing equivalent of the synthetic producer in detecting dropped spans and metrics, and it pairs naturally with it.

What not to test

As with logging, not every span deserves a test, and a suite that pins every attribute of every span becomes a maintenance burden that teaches the team to update tests without reading them.

Test what something depends on. The server span's route and status, because dashboards group by them. The spans for business operations, because engineers navigate by them. The attributes sampling policies read, because a rename silently changes what is kept. The propagation at every boundary, because it is the most fragile property and the most valuable.

Do not test library internals. Automatic instrumentation for a database driver or HTTP client produces spans whose exact attribute set is the instrumentation's business. Asserting on all of it couples the suite to the instrumentation's version. Asserting that a database span exists as a child of the right parent is enough.

Do not test timing. Span durations vary with the machine, the continuous integration runner, the load and the phase of the moon. Assertions on durations are flaky by construction; the ordering and nesting of spans carry the structural information without the timing.

Keep one end-to-end check, not many. The delivery check is valuable, slow and dependent on shared infrastructure. One synthetic request per deploy, through the path that crosses the most services, catches almost every delivery problem; a battery of them catches little more and slows every deploy.

Common mistakes

A batch processor in tests. Error signature: tests that need sleeps and still flake. Root cause: spans exported asynchronously. Remediation: a simple processor with an in-memory exporter.

Asserting only that spans exist. Error signature: a green suite and orphan spans in production. Root cause: no structural assertions. Remediation: assert parent relationships and shared trace identifiers.

Asserting span order in async tests. Error signature: tests that fail one run in ten. Root cause: concurrent tasks finishing in varying order. Remediation: assert on the tree, not the order.

No end-to-end check. Error signature: perfect unit tests and no spans from production workers. Root cause: the export thread lost to a fork, invisible in-process. Remediation: a synthetic request and a backend query after each deploy.

Pinning every attribute of automatic spans. Error signature: tests updated with every instrumentation upgrade. Root cause: coupling to library internals. Remediation: assert only the attributes something depends on.

Testing only the happy path. Error signature: exceptions that never appear on spans during real incidents. Root cause: no test exercises a failure. Remediation: at least one test per critical operation that forces a failure and asserts it is recorded on the right span.

Sampler set to drop in the test environment. Error signature: tests that find no spans at all, intermittently or always. Root cause: the test provider inherited a ratio sampler from environment variables. Remediation: configure the test provider with an always-on sampler explicitly.

Global provider reconfigured per test. Error signature: warnings about overriding the provider, and tests affecting each other. Root cause: the global provider can only be set once. Remediation: one session-wide test provider, cleared between tests.

Frequently Asked Questions