Python Observability

Tracing AWS Lambda Python Functions

A Lambda function breaks the assumption every telemetry SDK is built on: that the process keeps running after the work finishes. It does not — the execution environment is frozen the instant the handler returns, and a background exporter scheduled for five seconds later simply never runs. This page covers the initialisation, the flush, the cold start signal and the context extraction that make a function's traces arrive. It is a task article under telemetry from serverless and batch Python, part of the Python telemetry pipelines and delivery section.

Where the freeze lands One execution environment is drawn across three invocations separated by idle periods. Before the first invocation the module is imported and the tracer provider is created, which is the cold start cost and happens exactly once for this environment. The first invocation runs the handler and returns, at which point the environment is frozen: no thread runs, no timer fires, and anything queued for export stays queued. Minutes later the second invocation thaws the environment, and the background thread finally gets a chance to run — exporting the first invocation's spans alongside the second's, with timestamps minutes apart from their arrival. The third invocation behaves the same way. Beneath the timeline the remedy is drawn: a forced flush at the end of each handler, which moves the export inside the billed window and makes delivery immediate and reliable. one execution environment, three invocations import + init invoke 1 frozen — no thread runs invoke 2 frozen invoke 3 invocation 1's spans wait here until something thaws the environment cold with a forced flush at the end of the handler import + init invoke 1 flush frozen invoke 2 flush the flush is inside the billed duration — which is what a telemetry extension moves back out of it
Nothing runs while the environment is frozen. Everything about instrumenting a function follows from that one fact.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-http>=1.27.0,<2.0.0" \
            "opentelemetry-propagator-aws-xray>=1.0.1,<2.0.0"

# the extension listens locally; the function never talks to a remote endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_SERVICE_NAME=order-webhook

Implementation

Step 1 — Initialise at module scope, not in the handler. Module-level code runs once per execution environment; handler code runs once per invocation. Building a tracer provider, resolving a resource and opening a connection are initialisation costs that belong in the first category, and putting them in the handler pays them on every invocation and leaks a provider each time.

# handler.py — everything above `def handler` runs once per environment.
import os
import time

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

_COLD_START = True

_provider = TracerProvider(resource=Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
    "faas.name": os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "unknown"),
    "faas.version": os.environ.get("AWS_LAMBDA_FUNCTION_VERSION", "$LATEST"),
    "cloud.region": os.environ.get("AWS_REGION", "unknown"),
}))
_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(timeout=3)))
trace.set_tracer_provider(_provider)
tracer = trace.get_tracer("order-webhook")

Step 2 — Extract the incoming context from the event. Where the context lives depends entirely on what invoked the function, and getting this wrong produces a function whose traces are all roots — technically present, and useless for following a request across services.

from opentelemetry.propagate import extract

def _incoming_context(event: dict):
    """Find the propagation headers wherever this trigger puts them."""
    if "headers" in event:                              # HTTP / API gateway
        return extract({k.lower(): v for k, v in event["headers"].items()})
    records = event.get("Records") or []
    if records and "messageAttributes" in records[0]:   # queue message
        attrs = records[0]["messageAttributes"]
        return extract({k.lower(): v.get("stringValue", "")
                        for k, v in attrs.items()})
    return extract(event.get("_trace_context", {}))     # direct invoke

Step 3 — Wrap the handler body in one span. One span for the invocation, with the platform's request identifier so a trace can be matched to a platform log entry, and the cold start flag so cold and warm latencies can be separated. Cold starts are usually several times slower, and mixing them into one latency distribution makes both numbers meaningless.

def handler(event, context):
    global _COLD_START
    cold = _COLD_START
    _COLD_START = False

    with tracer.start_as_current_span(
        context.function_name,
        context=_incoming_context(event),
    ) as span:
        span.set_attribute("faas.invocation_id", context.aws_request_id)
        span.set_attribute("faas.coldstart", cold)
        span.set_attribute("faas.max_memory", context.memory_limit_in_mb)
        try:
            return do_work(event)
        finally:
            # 1. Nothing runs after this returns. Flush now or lose it.
            _provider.force_flush(timeout_millis=2000)

Expected Output: a span per invocation, joined to its caller, with the cold start visible.

{
  "name": "order-webhook",
  "traceId": "9f2a71c4f0b84c2e9d5f1a7b3c8e6d02",
  "parentSpanId": "4b1e77a2c9de5013",
  "durationMs": 412,
  "attributes": {
    "faas.invocation_id": "5c1f8a2e-7b4d-11f0-9b1a-0242ac120002",
    "faas.coldstart": true,
    "faas.max_memory": 512,
    "cloud.region": "eu-west-1"
  }
}

Step 4 — Bound the flush, and accept that it can fail. force_flush takes a timeout, and it should be short: an invocation held for five seconds because a collector is unreachable is a worse outcome than losing one span. Two seconds is a reasonable ceiling against a local extension, and the flush returning false — meaning it did not complete — is worth counting rather than ignoring.

Step 5 — Move the export outside the billed window with an extension. A telemetry extension runs as a separate process alongside the function, accepts OTLP on a local port, and forwards after the response has been returned to the caller. The function still flushes, but it flushes to something on localhost that responds in single-digit milliseconds, so the cost inside the billed duration is negligible. This is the arrangement to use wherever the platform offers it.

Why the cold start flag matters Three latency distributions for the same function. The warm distribution is narrow and centred around forty milliseconds, representing the overwhelming majority of invocations. The cold distribution is much wider and centred around eight hundred milliseconds, representing a small percentage of invocations that paid the initialisation cost. The combined distribution, which is what a dashboard shows when the two are not separated, is bimodal: its mean sits in a region where almost no invocation actually lands, and its p99 is dominated entirely by cold starts, so a latency alert on it fires when the function's concurrency pattern changes rather than when the function gets slower. Separating them with the cold start attribute gives two distributions that each describe something real. the same function, measured three ways latency warm · 97% of invocations cold · 3% combined — bimodal, and its mean describes nothing a p99 alert on the combined series fires on concurrency changes, not on the function getting slower
One attribute separates two populations that have nothing in common. Without it every latency number is an average of two unrelated things.

What else changes in this environment

Beyond the flush, four properties of the environment affect instrumentation decisions.

Concurrency is horizontal and invisible. Each concurrent invocation gets its own execution environment, so there is no shared process state, no connection pool to instrument and no thread pool to watch. Metrics that describe a process — memory, thread count, pool saturation — either do not apply or describe one environment among hundreds. What is worth measuring instead is per-invocation: duration, cold start, memory used against memory allocated, and the outcome.

Initialisation cost is a latency feature, not a startup detail. In a long-lived service, import time is paid once and forgotten. Here it is paid on every cold start and shows up directly in user-visible latency, so a heavy import graph is a performance problem rather than an aesthetic one. Instrumenting the module-scope initialisation itself — a span around the imports, emitted on the first invocation — is unusual and occasionally very informative.

The platform already produces some telemetry. Invocation counts, durations, errors and throttles exist as platform metrics without any instrumentation. Reproducing them in application code adds cost and a second source of truth that will disagree at the edges. The application's telemetry should add what the platform cannot see: what the function did, which downstream calls it made, and why it failed.

Log output goes somewhere specific. Anything written to standard output is captured by the platform's logging, which is convenient and means the structured logging practices from elsewhere in this site apply directly: one JSON object per line, with the trace identifier included so the logs join the trace. The one adjustment is that the logging configuration must be at module scope and guarded, or a reused environment accumulates a handler per invocation and duplicates every record.

Deciding how much to instrument

There is a real tension here that does not exist in a long-lived service: every millisecond of instrumentation appears on an invoice, and a function invoked ten million times a month is paying for its telemetry ten million times.

The arithmetic is worth doing rather than guessing at. A flush against a local extension costs perhaps five milliseconds; at ten million invocations that is fourteen hours of billed duration a month, which at typical pricing is a small number but not a negligible one for a 128 MB function. Against a remote endpoint the same flush might cost eighty milliseconds, which is twenty times the bill and the reason the extension exists.

Two conclusions follow. First, the extension is not an optimisation to consider later; for any function invoked at scale it is the difference between telemetry being affordable and not. Second, span granularity matters more here than anywhere else — each additional span is serialisation work inside the billed window, so the discipline of one span per invocation plus spans for genuine downstream calls, rather than a span per internal function, is enforced by the billing model rather than merely recommended.

The exception is a function invoked rarely, where none of this matters and the cost of an unflushed span — total invisibility of a function nobody watches — is far higher than a few milliseconds. Sparse functions should flush generously; hot functions should be spare with spans and use an extension.

Where each part of the setup lives A table of the parts of a traced Python Lambda function and where each is configured. The OpenTelemetry SDK and instrumentation: in a layer or the deployment package, initialised at import so warm invocations reuse it. The collector: as a Lambda extension layer that receives OTLP on localhost. Context from the caller: extracted from the event, such as API Gateway headers or SQS message attributes. The flush: force_flush on the tracer provider before the handler returns. The cold-start flag: a span attribute set on the first invocation of each execution environment. The note says initialising at import and flushing per invocation is the pattern that makes Lambda tracing both cheap and complete. part where SDK + instrumentation layer or package; init at import collector extension layer, OTLP on localhost caller's context extracted from the event flush force_flush() before the handler returns cold-start flag span attribute on first invocation initialise at import, flush every invocation
Initialising once and flushing per invocation keeps warm calls cheap and makes sure no call's spans are frozen with the sandbox.

Configuration options

Setting Value Why
Provider creation module scope once per environment, not per invocation
Span processor batch, with forced flush or simple, for very low volume
force_flush timeout 2000 ms bounded; a held invocation is worse than a lost span
Exporter endpoint localhost extension keeps the export out of the billed window
faas.coldstart attribute on every span separates two latency populations
faas.invocation_id attribute joins the trace to the platform's own log
Context extraction per trigger type headers, message attributes, or payload

Verification

Check the two properties that fail silently: that spans arrive from sparse invocations, and that they have a parent.

# invoke once, then wait longer than any export interval before checking
aws lambda invoke --function-name order-webhook --payload '{"test":true}' /dev/null
sleep 30
# the span must already be in the backend, not waiting for the next invocation

Expected Output: the span present within seconds of the invocation, with a parent identifier when invoked from a traced caller.

trace 9f2a71c4…  span order-webhook  parent 4b1e77a2…  coldstart=true  412ms

A span that appears only after the next invocation means the flush is not running; a span with no parent, when the caller was traced, means the extraction in step 2 did not match this trigger's shape.

Common mistakes

Provider created inside the handler. Error signature: latency growing with the environment's age, and export connections accumulating. Root cause: initialisation per invocation. Remediation: move it to module scope, as in step 1.

No flush before returning. Error signature: telemetry from sparse functions never arriving, and from busy ones arriving attributed to the wrong time. Root cause: the environment frozen before the background export ran. Remediation: force_flush in a finally block.

Cold starts mixed into one latency series. Error signature: a p99 that moves when traffic patterns change and not when the code does. Root cause: two populations in one distribution. Remediation: record the cold start flag and split every latency query by it.

Context extraction written for one trigger. Error signature: traces that connect for HTTP invocations and break for queue-driven ones. Root cause: only the headers case implemented. Remediation: dispatch on event shape, as in step 2.

Logging handlers added per invocation. Error signature: each record appearing two, then three, then four times. Root cause: logging configured in the handler in a reused environment. Remediation: configure at module scope and guard against reconfiguration.

Frequently Asked Questions

Why does telemetry from my Lambda function arrive late or not at all?

The execution environment is frozen as soon as the handler returns, so a background export thread scheduled for later does not run until the environment is thawed for another invocation. For a busy function that looks like delay; for a sparse one it looks like total loss.

Does flushing on every invocation cost much?

It adds the export round trip to the billed duration — typically tens of milliseconds against a local extension, more against a remote endpoint. Using the platform's telemetry extension moves that cost outside the billed window, which is the main reason to use one.

How do I mark a cold start?

Set a module-level flag when the module is imported and clear it at the end of the first invocation. The module is imported once per execution environment, so a true value on entry means this invocation paid the initialisation cost.

Where does the incoming trace context come from?

It depends on the trigger. An HTTP invocation carries it in the request headers; a queue message carries it in message attributes; a direct invocation carries whatever the caller put in the payload. Each needs its own extraction, which is why a small dispatch on event shape is worth writing once.

Related Guides