Python Observability

Testing Spans with the In-Memory Span Exporter

The OpenTelemetry SDK ships an exporter that keeps finished spans in a list, and combined with a processor that exports synchronously, it lets a test read every span the code under test produced the moment that code returns. This page covers a pytest setup that does this reliably — one provider for the session, isolation between tests, lookups by name rather than position — and the assertions worth writing. It is a task article under testing and validating instrumentation, part of the distributed tracing and OpenTelemetry in Python section.

What the test sees, and in what order The test-time pipeline is drawn: application code creates spans through a tracer, which uses the session-wide test provider with an always-on sampler; each span is passed on end to a simple span processor, which exports it immediately to the in-memory exporter's list. Below, a request producing a server span with two children is traced through time: the first child ends, then the second, then the server span. The exporter's list therefore holds child, child, parent — end order, not start order. A test that asserts the first element is the server span fails. A test that looks spans up by name and checks parent identifiers passes regardless of order. The note records that export order is the one thing about the exporter tests should never depend on. synchronous export, in end order code under test test provideralways_on simple processorexport on end in-memory list POST /orders — ends third reserve stock — first charge payment — second exporter list: [reserve, charge, POST] spans[0].name == "POST /orders" fails — index is end order named("charge").parent == named("POST") passes, whatever the order
Children usually end before parents, so the exporter's list is in end order. Look spans up by name and relationship, never by index.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "pytest>=8.0.0,<9.0.0"

Implementation

Step 1 — Configure one provider for the whole session. The global tracer provider can be set only once per process. Setting it in conftest.py, before application modules are imported, means every tracer — including ones created at import time — routes to it. The sampler is set explicitly to always-on, so an environment variable intended for production cannot make tests drop spans.

# tests/conftest.py
import pytest
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.sampling import ALWAYS_ON

EXPORTER = InMemorySpanExporter()
_provider = TracerProvider(sampler=ALWAYS_ON)
_provider.add_span_processor(SimpleSpanProcessor(EXPORTER))
trace.set_tracer_provider(_provider)

Step 2 — Isolate tests with a clearing fixture. The exporter is shared across the session, and nothing empties it on its own. Clearing it before and after each test means each test sees only its own spans, and a failure leaves nothing behind for the next one.

@pytest.fixture
def spans():
    EXPORTER.clear()
    yield EXPORTER
    EXPORTER.clear()

Step 3 — Look spans up by what they are. Because export happens on span end, the list is in end order. Helpers that select by name, kind or attribute keep tests independent of order and produce clear failures when a span is missing.

from opentelemetry.trace import SpanKind

def by_name(exporter, name):
    found = [s for s in exporter.get_finished_spans() if s.name == name]
    assert found, f"no span {name!r}; saw {sorted({s.name for s in exporter.get_finished_spans()})}"
    return found[0]

def server_span(exporter):
    (s,) = [s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER]
    return s

Step 4 — Assert on the properties that matter. The tree, the attributes consumers use, recorded exceptions and status. Each assertion targets a property that something outside the test depends on.

from opentelemetry.trace import StatusCode

def test_charge_failure_is_recorded(client, spans, gateway):
    gateway.fail_with(TimeoutError("read timed out"))
    client.post("/orders", json={"sku": "A1"})

    server = server_span(spans)
    charge = by_name(spans, "charge payment")

    assert charge.parent.span_id == server.context.span_id          # tree
    assert charge.context.trace_id == server.context.trace_id
    assert server.attributes["http.route"] == "/orders"              # attributes
    assert charge.status.status_code is StatusCode.ERROR             # status
    (exc,) = [e for e in charge.events if e.name == "exception"]     # exception event
    assert exc.attributes["exception.type"].endswith("TimeoutError")

Step 5 — Wait for background work. If the code under test starts tasks or submits pool work that continues after the request returns, their spans are not finished when the test asserts. Waiting for that work explicitly — awaiting the tasks, shutting down the pool — makes the assertion see exactly what the test intends.

async def test_background_indexing_is_traced(app, spans):
    await app.handle_order({"sku": "A1"})
    await app.background.drain()                      # the indexing task finishes here
    assert by_name(spans, "index order").parent is not None

Expected Output: a passing suite that fails with a specific message when instrumentation regresses.

tests/test_order_tracing.py::test_charge_failure_is_recorded PASSED
tests/test_order_tracing.py::test_background_indexing_is_traced PASSED
Why isolation needs both clearing and waiting Two consecutive tests are drawn on a timeline. The first test's code starts a background task and returns; the test asserts and finishes while the background task is still running. The second test begins, and during it the first test's background task ends, exporting its span into the shared exporter. Without clearing, the second test sees both its own spans and the first test's leftovers from the start. With clearing at the start of each test but no waiting, the second test still sees the late span that arrived during it. With both clearing and the first test waiting for its own background work before finishing, the late span is exported during the first test and cleared before the second begins, so each test sees only its own spans. The note records that a flaky tracing test is almost always one of these two gaps. test 1 starts background work; test 2 follows test 1 test 2 test 1's background task ends inside test 2 clear only test 2 still sees test 1's late span clear + wait test 1 drains its own work before finishing; test 2 sees only its own spans a flaky tracing test is almost always one of these two gaps spans left over from before, or spans arriving late from work the previous test did not wait for
Clearing removes what came before. Waiting stops a test's own work from arriving during the next one. Both are needed for tests that never see each other's spans.

The same exporter gives access to everything recorded on a span, and three less obvious properties are worth testing where they matter.

Span events. Exceptions are recorded as events named exception, with the type, message and stack in attributes. Other events — a cache miss, a retry, a state transition — are recorded the same way. Tests that assert an event exists, with the right attributes, protect the detail that engineers rely on when reading a single trace, as described in recording exceptions and span events.

Links. A batch span's links can be asserted like any other property: count, target trace identifiers, and the attributes on each link. This is the test that catches a batch processor quietly reverting to parenting on the first item.

Resource. Every finished span carries the provider's resource. In tests the resource is whatever the test provider was given, which is usually not what production uses — so asserting on production resource attributes in unit tests is meaningless. Resource correctness belongs to the deployment check, where the real provider is running.

A practical point about attributes: the SDK applies limits to attribute counts and value lengths, and tests run against the same limits. A test that asserts on a long attribute value may see it truncated, which is correct behaviour and occasionally surprising. Setting the limits explicitly in the test provider to match production makes truncation visible in tests before it is discovered in the store.

Testing async code

Async tests need the same harness and two adjustments.

The first is the event loop. With a plugin such as pytest-asyncio, each async test runs on a loop the plugin manages, and spans created in coroutines are exported by the same simple processor, synchronously, on the loop's thread. Nothing about the exporter changes. What changes is that tasks created by the code under test may still be pending when the test's own coroutine reaches its assertions, which is why step 5 matters more for async code than for synchronous code. A fixture that collects tasks the application creates — or an application hook that exposes its background task set — lets the test drain them before asserting.

The second is ordering under concurrency. Spans from concurrent coroutines end in an order that depends on scheduling, which varies between runs, particularly with real I/O mocked by asynchronous fakes that complete instantly. Assertions on sets of children — "the fan-out span has exactly these three children" — are stable; assertions on sequence are not.

import asyncio
import pytest

@pytest.mark.asyncio
async def test_quote_fan_out_structure(spans):
    await build_quote(order_fixture())
    fan = by_name(spans, "build quote")
    children = {s.name for s in spans.get_finished_spans()
                if s.parent and s.parent.span_id == fan.context.span_id}
    assert children == {"inventory", "pricing", "shipping"}

Keeping instrumentation tests fast and focused

A tracing test suite can grow expensive if every test spins up a full application. Two habits keep it cheap.

Test instrumentation at the lowest level that exercises it. A test of a span processor calls the processor directly with a fake span. A test of a helper that starts a business span calls the helper. Only tests of propagation across boundaries and of framework integration need a running application or a test client, and those are few.

Share expensive fixtures across tests. An application instance or a test client is usually safe to share for a module, as long as the exporter is cleared per test. The provider is already shared for the session. The cost of the suite then scales with the number of assertions rather than the number of application startups, which keeps it fast enough to run on every commit — which is where it catches the regressions it exists for, as argued in testing and validating instrumentation.

What to assert, and what not to A table of assertions on spans captured by the in-memory exporter, each marked worth asserting or brittle. The span name: worth asserting, it is what dashboards group by. Required attributes such as the route and status: worth asserting. The parent and child relationship: worth asserting, it is what makes the trace readable. The status on failure: worth asserting. Exact timestamps or durations: brittle, they vary per run. The full attribute dictionary: brittle, every library upgrade adds attributes. Span IDs: brittle, they are random. The note says to assert on the contract that consumers depend on, not on everything the span happens to contain. property assert? reason span name yes what dashboards group by route, status attributes yes queries filter on them parent and child yes makes the trace readable status on failure yes error rates depend on it exact durations no vary every run the whole attribute dict no upgrades add attributes assert on the contract consumers depend on not on everything the span happens to contain
Tests that pin the contract survive library upgrades. Tests that pin everything break on every upgrade.

Configuration options

Element Choice Why
Provider one per session, set in conftest global provider can be set once
Sampler ALWAYS_ON explicitly environment cannot make tests drop spans
Processor SimpleSpanProcessor spans available when the code returns
Exporter InMemorySpanExporter readable list of finished spans
Isolation clear before and after no cross-test leakage
Lookups by name, kind, attribute order-independent
Background work awaited in the test spans finished before assertions
Limits same as production truncation visible in tests

Verification

Confirm the setup itself by checking that a trivial span is captured and cleared.

def test_harness_captures_and_isolates(spans):
    trace.get_tracer("t").start_span("probe").end()
    assert [s.name for s in spans.get_finished_spans()] == ["probe"]

Expected Output:

.
1 passed

A failure listing extra spans means another test's work is leaking in; an empty list means the provider was set after the application's tracer bound to a different one.

Common mistakes

Setting the provider per test. Error signature: a warning about overriding the provider and spans going nowhere. Root cause: the global provider can be set only once. Remediation: set it once in conftest.

Indexing into the span list. Error signature: assertions on spans[0] that fail when code changes order. Root cause: the list is in end order. Remediation: look spans up by name and relationship.

No clearing fixture. Error signature: tests passing alone and failing in the suite. Root cause: shared exporter state. Remediation: clear before and after each test.

Not waiting for background work. Error signature: a span sometimes missing, sometimes present. Root cause: the span ends after the assertion. Remediation: await or drain the work.

Inheriting a production sampler. Error signature: tests finding no spans in some environments. Root cause: a ratio sampler from environment variables. Remediation: an explicit always-on sampler in the test provider.

Frequently Asked Questions

Why does the exporter sometimes contain spans from another test?

Because the exporter is shared and nothing cleared it. Background work from a previous test may also finish during the next one. A fixture that clears before and after each test, plus waiting for background work within the test that started it, isolates them.

Why are spans in the exporter in a different order than they started?

The simple processor exports each span when it ends, and children usually end before their parents. The exporter's order is end order, which is why tests should look spans up by name and relationship rather than by index.

Can I set the global tracer provider in each test?

No — the global provider can be set once per process, and later attempts are ignored with a warning. Set it once for the session and clear the exporter between tests.

How do I test code that uses a tracer created at import time?

Tracers obtained from the global API delegate to whatever provider is set, even if the tracer was created before the provider. Setting the test provider early in conftest, before application modules are imported, covers both cases.

Related Guides