Python Observability

Recording Exceptions and Span Events

A span with a duration tells you an operation was slow. A span with events tells you what happened during it, and in what order. This page covers record_exception and add_event, the status decision that is separate from both, and the limits that keep one span from becoming unmanageable. It builds on span lifecycle and attributes, part of the distributed tracing and OpenTelemetry in Python section.

Events mark instants; child spans occupy time One span drawn as a horizontal bar with two kinds of annotation. Three events are marked as points along the bar: a cache miss near the start, a retry decision in the middle, and an exception recorded near the end — each has a timestamp and attributes but no duration, and each costs one entry on the span that already exists. One child span is drawn as its own shorter bar beneath, covering the outbound call that took a hundred and eighty milliseconds; it has a start and an end, appears as its own row in a waterfall, and costs a full span in the export pipeline and in storage. The guidance drawn is that the question to ask is whether the thing has a duration worth seeing: a retry decision does not, so it is an event, while the retried call does, so it is a span. Using a child span for an instant produces a waterfall full of zero-width bars, and using an event for a duration loses the timing entirely. one span, three events, one child SERVER · GET /orders/{id} · 412 ms cache.miss retry.scheduled exception the span CLIENT · inventory /stock · 186 ms a child span an event, when there is no duration a cache miss · a retry decision · a config reload a message acknowledged · an exception caught costs one entry on a span that already exists a child span, when the time matters an outbound call · a query · a batch of work anything you would want as a bar in a waterfall costs a whole span in the pipeline and in storage
The question is whether the thing has a duration worth seeing. A waterfall full of zero-width bars is what using a span for an instant looks like.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

export OTEL_SPAN_EVENT_COUNT_LIMIT=128
export OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=128
export OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT=4096

Implementation

Step 1 — Record the exception and set the status separately. They are independent operations and both are usually wanted, but not always.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def charge(order_id: int) -> None:
    with tracer.start_as_current_span("payment.charge") as span:
        span.set_attribute("order.id", order_id)
        try:
            gateway.charge(order_id)
        except PaymentDeclined as exc:
            span.record_exception(exc)                     # context: this happened
            span.set_attribute("payment.declined_reason", exc.reason)
            # status stays UNSET — the operation completed and returned an answer
            raise
        except GatewayUnavailable as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "gateway unavailable"))
            raise

The two arms show the distinction. A declined payment is an outcome — the operation worked and the answer was no. An unreachable gateway is a failure of the operation itself. Recording both as errors makes the error rate a measure of customer behaviour rather than of system health.

Step 2 — Let the context manager handle the uncaught case. start_as_current_span records the exception and sets error status automatically when one escapes, which is the correct default.

with tracer.start_as_current_span("orders.reconcile") as span:
    reconcile()          # if this raises, the span is recorded and marked ERROR automatically

To opt out — for an exception you expect and handle upstream — pass record_exception=False and set_status_on_exception=False to the context manager.

Step 3 — Add events for the moments in between.

span.add_event("cache.miss", attributes={"cache.key_prefix": "order", "cache.backend": "redis"})

span.add_event("retry.scheduled", attributes={
    "retry.attempt": attempt,
    "retry.delay_ms": delay_ms,
    "retry.reason": "upstream_503",
})

Events carry a timestamp automatically. Pass one explicitly only when recording something that happened earlier than the call — a message's enqueue time, for instance.

Step 4 — Bound the event count. A retry loop, a chatty library or a long-lived span can attach thousands of events, and the result is a span the backend may reject outright.

from opentelemetry.sdk.trace import SpanLimits, TracerProvider

provider = TracerProvider(
    span_limits=SpanLimits(
        max_events=128,
        max_attributes=128,
        max_attribute_length=4096,
        max_event_attributes=32,
    ),
)

When the limit is hit, the oldest events are dropped and a dropped count is recorded on the span. That is better than an unbounded list, and it is still worth avoiding: if a loop is producing more than a hundred events, the loop probably wants its own child spans, or a counter.

record_exception and set_status are two decisions A two-by-two grid of the two independent choices. Recording the exception and setting error status is the ordinary failure: the operation did not succeed and the traceback explains why — an unreachable gateway, a database that refused the connection. Recording the exception without setting error status is the handled case: something went wrong, the code dealt with it, and the operation returned a valid answer — a declined payment, a cache backend that failed over, a parse error that fell back to a default. Setting error status without recording an exception is the failure with no exception object: a downstream returned a 500, a validation check failed, a deadline passed — there is nothing to record but the span did fail. Recording neither is the success path. The note underneath is that most instrumentation gets the first and last right and collapses the middle two into the first, which is what makes an error rate stop meaning anything. two independent decisions, four outcomes status ERROR status UNSET record exception the ordinary failure gateway unreachable connection refused handled — the one most people skip payment declined · cache failed over parse error with a working fallback do not record failure, no exception object downstream returned 500 a deadline passed success nothing to say the overwhelming majority of spans collapsing the top-right cell into the top-left is what makes an error rate track customer behaviour instead of system health
The top-right cell is the one that gets collapsed into the top-left. Do that consistently and your error rate measures how often customers' cards are declined.

Step 5 — Sanitise before recording. record_exception serialises the traceback verbatim into exception.stacktrace, which ships to the backend as an attribute.

import re

SECRET = re.compile(r"(password|token|api[_-]?key)\s*[=:]\s*[^\s,;)'\"]+", re.I)

def record_safely(span, exc: BaseException) -> None:
    if exc.args and isinstance(exc.args[0], str):
        exc.args = (SECRET.sub(r"\1=***", exc.args[0]),) + exc.args[1:]
    span.record_exception(exc)

A Collector attribute processor can hash or delete exception.stacktrace as a second layer, and should — but by then the data has already left the process, so the in-process pass is the one that matters. The full pattern set is in redacting sensitive data in log records.

What happens past the event limit A span whose code attaches an event for every retry attempt in a loop that runs four hundred times. The first hundred and twenty-eight events are kept; every event after that displaces the oldest, so the span ends up holding only the most recent hundred and twenty-eight and a dropped-event count of two hundred and seventy-two. The consequence is that the beginning of the loop — the attempts that show what started the failure — is exactly what was discarded, while the tail, which is the least informative part, survives. The alternative shown records a single attribute holding the attempt count plus one event for the first failure and one for the last, which fits comfortably inside the limit, keeps both ends of the story, and produces a span a backend will accept without complaint. a retry loop with an event per attempt, 400 attempts one event each attempts 1–272: dropped 273–400: kept dropped_events_count = 272 the attempts that show what started it are the ones discarded — the limit drops oldest first counter + two events retry.attempts = 400 first failure last failure both ends of the story, three entries, comfortably inside every limit the general rule if a loop can produce more than a handful of events, it wants a counter attribute — or child spans, if the iterations have durations
The limit drops oldest first, which discards exactly the attempts that explain how the failure started.

Configuration options

Option Env var Default Recommended
Event count limit OTEL_SPAN_EVENT_COUNT_LIMIT 128 128
Attribute count limit OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT 128 128
Attribute value length OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT unlimited 4096
Event attribute limit OTEL_SPAN_EVENT_ATTRIBUTE_COUNT_LIMIT 128 32
record_exception context manager True True
set_status_on_exception context manager True True
Escaped flag escaped= False True when the exception left the span

Verification

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("probe")
with tracer.start_as_current_span("probe.op") as span:
    span.add_event("cache.miss", {"cache.backend": "redis"})
    try:
        raise ValueError("payment declined; token=sk-live-9f3c")
    except ValueError as exc:
        record_safely(span, exc)
        # deliberately no set_status — this is a handled outcome

Expected Output (Collector debug exporter):

Span #0
    Name           : probe.op
    Status code    : Unset
    Events:
    SpanEvent #0
         -> Name: cache.miss
         -> Attributes:
              -> cache.backend: Str(redis)
    SpanEvent #1
         -> Name: exception
         -> Attributes:
              -> exception.type: Str(ValueError)
              -> exception.message: Str(payment declined; token=***)
              -> exception.stacktrace: Str(Traceback (most recent call last)…)

Three things confirm the setup: the status is Unset despite the recorded exception, the token is masked, and the exception arrived as an event named exception with the semantic-convention attributes — the same names a log record gets through the OTLP bridge, which is what lets one query cover both.

Common mistakes

The error rate tracks customer behaviour

Error signature: span error rate sits at a constant 20% and correlates with declined payments rather than with incidents. Root cause: every caught exception sets error status. Remediation: record the exception for context; set error status only when the operation itself failed.

One span is megabytes

Error signature: the backend rejects spans, or a single trace view takes seconds to render. Root cause: a retry loop attached thousands of events, or an attribute value contains a full response body. Remediation: set span limits, and convert a high-frequency event into a counter attribute or into child spans.

The exception is recorded and the span is not marked

Error signature: a genuinely failed operation shows a green span with an exception event on it. Root cause: record_exception was called and set_status was not — the two are independent. Remediation: call both when the operation failed; let the context manager do it for uncaught exceptions.

Events, attributes, or a child span

Three ways to attach information to a trace, and choosing between them is mostly mechanical once the question is framed correctly.

An attribute describes a property of the whole span: the route, the tenant, the entity ID, the outcome. It exists for the span's entire duration, it has no timestamp, and it can be filtered on directly in most backends — which is what makes attributes the right place for anything you will search by.

An event describes something that happened at a point during the span: a cache miss, a retry decision, a lock acquired, an exception caught. It has a timestamp and its own attributes, and it is the right shape whenever when it happened is part of the information.

A child span describes a sub-operation with a duration: a query, a call, a computation. It appears as its own row in the waterfall and costs a full span in the pipeline.

The question that decides between them is: does it have a duration worth seeing, and does it have a moment worth knowing? Duration means a span. A moment without a duration means an event. Neither means an attribute.

Information Shape Example
A property of the operation attribute order.id, http.route, tenant
A moment during it event cache.miss, retry.scheduled, exception
A sub-operation with a duration child span a query, an outbound call
A count of moments attribute retry.attempts = 4
An outcome status + attribute Status(ERROR) plus a reason

The fourth row is the escape hatch for the event-limit problem: a loop that would produce hundreds of events almost always has a summary that fits in one attribute, and the summary is usually what somebody would have queried anyway.

Searchability differs by backend

One practical caveat that affects how much to invest in events: backends vary considerably in how well event attributes can be searched. Span attributes are almost universally indexed and filterable; event attributes are sometimes only visible when the span is opened, which makes them excellent for reading and useless for finding.

The consequence is that anything you will search for belongs on the span as an attribute, even when it conceptually happened at a moment. A pragmatic pattern is both: the event carries the detail and the timestamp, and a span attribute carries the fact that it happened at all.

span.add_event("cache.miss", {"cache.key_prefix": "order"})
span.set_attribute("cache.missed", True)          # the searchable version

That costs one boolean and makes "show me requests with a cache miss" a query rather than an inspection.

Exception recording across service boundaries

An exception recorded on a span stays on that span, which means a failure that propagates through three services is recorded three times — once per service, each with its own local traceback. That is the correct behaviour and it has a reading consequence worth knowing: the innermost recorded exception is the origin, and the outer ones are translations. A trace view that shows three error spans is not three failures.

Two habits make that legible. Record the exception where it is caught rather than re-recording it at every layer as it propagates, so a span carries a failure it actually handled. And when a service translates a downstream failure into its own error type, record the translated one and let the downstream span carry the original — the parent-child relationship already expresses the connection, and duplicating the original traceback upward adds bytes without adding information.

Frequently Asked Questions

Does record_exception mark the span as failed?

No, and that surprises people. record_exception adds a timestamped event carrying the exception type, message and stacktrace; span status stays UNSET unless you call set_status separately. That separation is deliberate: a caught and handled exception is worth recording as context without declaring the operation a failure, and only the code that caught it knows which case it is.

When should I use an event instead of a child span?

Use an event when the thing happened at an instant and has no meaningful duration — a cache miss, a retry decision, a config reload, a message being acknowledged. Use a child span when something took time you would want to see as a bar in a waterfall. An event costs one entry on an existing span; a child span costs a whole span in the pipeline and in storage.

How many events can a span hold?

By default 128, controlled by OTEL_SPAN_EVENT_COUNT_LIMIT, and each event's attributes are limited too. Exceeding the limit drops the oldest events silently and increments a dropped count on the span. That default is deliberately low: an unbounded event list on a long-running span is how a single span reaches megabytes and gets rejected by the backend.

Does the exception's stacktrace get redacted?

Only if you redact it. record_exception serialises the traceback verbatim into the exception.stacktrace attribute, and a Collector attribute processor can hash or delete that attribute, but by then the data has left the process. If the traceback can contain secrets, sanitise the exception before recording it.

Related Guides