Python Observability

Correlating Logs, Traces and Metrics

A dashboard says the 99th percentile went from 200 ms to 4 seconds. A trace shows one request that took 4 seconds. A log line shows a timeout. On most systems, connecting those three is manual work performed under pressure. This guide is about making the connections structural, so each signal names the others. It is part of the distributed tracing and OpenTelemetry in Python section, and the concrete walkthroughs are in exporting Python logs through OpenTelemetry and linking metrics to traces with exemplars.

Three joins do the work, and they are not symmetrical: two of them can be reconstructed later, and one cannot.

What actually joins each pair of signals The three telemetry signals drawn as a triangle with the join that connects each pair. Between logs and traces the join is the trace and span identifier pair, written onto every log record at the moment it is created; this direction works both ways, since a trace ID found in a log leads to a trace and a trace ID leads to every log line that carried it. Between metrics and traces the join is an exemplar, a trace ID attached to a recorded measurement, which is the only way to get from an aggregate bucket to an individual request that landed in it. Between logs and metrics there is no direct join and none is needed: the path runs through the trace. Underneath all three sits the resource — service name, version and deployment environment — which must be byte-identical across the three providers, because a mismatch makes the same process look like two or three different systems in the backend. A note marks the asymmetry: log-to-trace correlation can be added retroactively by reprocessing, while exemplars cannot, because the trace context existed only at the moment of measurement. three signals, three joins, one shared identity traces one request, in full logs what happened, in words metrics how much, over time trace_id · span_id on every record exemplars bucket → a real trace no direct join — the path runs through the trace one Resource: service.name · service.version · deployment.environment identical across all three providers, or the same process looks like three different systems the asymmetry: log-to-trace links can be added later by reprocessing — an exemplar cannot, because the trace context existed only at the moment of measurement
Two of these joins can be retrofitted. The exemplar cannot: if the trace ID was not captured when the value was recorded, that link never existed and never will.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0" \
            "opentelemetry-instrumentation-logging>=0.48b0,<1.0.0"

export OTEL_SERVICE_NAME=checkout-api
export SERVICE_VERSION=2026.8.1
export DEPLOY_ENV=production
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_METRICS_EXEMPLAR_FILTER=trace_based

Concept and architecture

Correlation is identity, applied at three levels.

Resource identity answers "which service produced this". It is a set of attributes — service.name, service.version, deployment.environment, plus whatever the environment detects — attached to every span, every metric point and every log record. Build one Resource object at startup and pass it to all three providers. Constructing it separately per provider is how a service ends up as checkout-api in traces and checkout_api in metrics, and no query joins the two.

Request identity answers "which request produced this". The trace ID is that identifier, and its advantage over a home-grown request ID is that it is already propagated across service boundaries by the same machinery that builds the trace, described in context propagation and baggage.

Measurement identity answers "which request produced this number". That is the exemplar, and it only exists if it was captured at record time.

The asymmetry is worth stating plainly. Log-to-trace correlation can be added retroactively, by reprocessing stored logs and extracting an ID that was already in the message. An exemplar cannot: the trace context existed for the microsecond the value was recorded, and if nothing captured it then, the link is gone forever.

Step-by-step implementation

Step 1 — Build one resource and share it.

# observability/otel.py
import os
from opentelemetry.sdk.resources import Resource

RESOURCE = Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
    "service.version": os.environ.get("SERVICE_VERSION", "0"),
    "deployment.environment": os.environ.get("DEPLOY_ENV", "dev"),
})

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk._logs import LoggerProvider

tracer_provider = TracerProvider(resource=RESOURCE)
meter_provider  = MeterProvider(resource=RESOURCE, metric_readers=[reader])
logger_provider = LoggerProvider(resource=RESOURCE)

One object, three providers. Every attribute that identifies the service is now defined in exactly one place.

Step 2 — Attach trace context to every log record. A logging filter reads the active span and writes the IDs onto the record. Omit the fields when there is no span rather than writing zeros.

import logging
from opentelemetry import trace

class TraceContextFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        span = trace.get_current_span()
        ctx = span.get_span_context()
        if ctx.is_valid:                                   # zeros are worse than absence
            record.trace_id = format(ctx.trace_id, "032x")
            record.span_id = format(ctx.span_id, "016x")
            record.trace_flags = "01" if ctx.trace_flags.sampled else "00"
        return True

Attach it at the logger, not the handler, so it runs on the producing thread while the span is still current. The full treatment — including what happens across a queue boundary — is in adding trace IDs to log records.

Step 3 — Turn on exemplars. The metrics SDK attaches the active trace context to recorded values when the exemplar filter allows it.

export OTEL_METRICS_EXEMPLAR_FILTER=trace_based    # only sampled spans become exemplars

histogram = meter.create_histogram(
    "http.server.request.duration",
    unit="s",
    description="Request latency",
)

with tracer.start_as_current_span("GET /orders/{id}"):
    start = time.perf_counter()
    ...
    histogram.record(                                # the active span becomes the exemplar
        time.perf_counter() - start,
        attributes={"http.route": "/orders/{id}", "http.response.status_code": 200},
    )

trace_based is the right filter: it records exemplars only for sampled spans, so every exemplar points at a trace that actually exists in the backend. always_on produces exemplars pointing at traces that were never exported, which is a link to a 404.

Step 4 — Use the same attribute names. Semantic conventions matter here for a practical reason: a filter written once should work on all three signals.

Concept Attribute Traces Metrics Logs
Route http.route span attribute metric attribute log field
Method http.request.method span attribute metric attribute log field
Status http.response.status_code span attribute metric attribute log field
Service service.name resource resource resource
Request trace_id span context exemplar log field

Metrics are the constrained one: every attribute multiplies series count, so a metric carries the route but never the request ID, for the reasons in controlling label cardinality in Prometheus.

The same three signals, walked in both directions An investigation traversing all three signals twice. Starting from an alert, a latency histogram shows a spike in the tail bucket; an exemplar on that bucket carries a trace ID, which opens a single request that took four seconds; that trace's span IDs select the log records emitted during it, one of which is a timeout from a downstream call. Total: three clicks, no manual time-window guessing. Starting from the other end, an engineer finds a suspicious log line while searching for an error string; its trace_id field opens the full request including every other service that took part; the span's route attribute identifies which metric series that request contributed to, showing whether the failure is isolated or systemic. The point made underneath is that both directions have to work: an exemplar without log correlation leaves you at a trace with no detail, and log correlation without exemplars leaves you unable to get from an alert to any specific request at all. the alert-down path p99 alert fires the tail bucket is full exemplar → a trace one request that took 4 s span_id → its log records "upstream timeout after 3.9 s" three clicks, no guessing at time windows the log-up path a suspicious log line found by searching an error trace_id → the request every service that took part http.route → the metric isolated, or systemic? both directions have to work exemplars without log correlation land you on a trace with no detail · log correlation without exemplars leaves an alert with no way in
The second row is the one teams usually have. The first row — getting from an alert to a specific request — is what exemplars add, and it is the one that cannot be retrofitted.
One process, three resources, three services A single Python process whose tracer, meter and logger providers each construct their own Resource. The tracer's resource names the service checkout-api, the meter's names it checkout_api with an underscore because someone derived it from a Python module name, and the logger's omits the version because that provider was added later by a different change. In the backend these appear as three unrelated services: a trace view that cannot find the metrics for the service it is showing, a dashboard whose service selector lists near-duplicates, and log records that join to neither. Nothing errors, every signal arrives, and the correlation simply does not exist. The fix shown is one Resource object constructed once and passed to all three providers, which makes the identity impossible to get wrong because there is only one of it. the same process, as the backend sees it tracer provider service.name = "checkout-api" version 2026.8.1 meter provider service.name = "checkout_api" derived from the module name logger provider service.name = "checkout-api" no version — added later three services in the backend, from one process a trace that cannot find its metrics · a service selector full of near-duplicates · log records that join to neither the fix is structural, not vigilance one Resource object, constructed once, passed to all three providers — impossible to get wrong because there is only one
Every signal arrives and nothing errors. The correlation just does not exist, and the cause is three constructor calls that should have been one.

Configuration reference

Setting Env var Default Production value
Service name OTEL_SERVICE_NAME unknown_service set it; it is the join key
Exemplar filter OTEL_METRICS_EXEMPLAR_FILTER trace_based trace_based
Log correlation OTEL_PYTHON_LOG_CORRELATION false true, or a filter of your own
Log format OTEL_PYTHON_LOG_FORMAT stdlib default a JSON formatter with the ID fields
Trace flags on records — absent present — shows whether the trace was kept
Resource — per provider one object, all three providers
Metric attributes — — conventions only; never a request ID

Async and concurrency considerations

Everything here depends on "the currently active span", which lives in a contextvars context. That gives the same semantics described in async tracing patterns: a task created with create_task inherits the context as it was at creation, and a value set afterwards does not reach it.

The consequence for correlation is specific. A log record emitted from a background task created before the span was started carries no trace ID; one created inside the span carries it correctly. And a record that crosses a queue to a listener thread must have been enriched on the producing side, because the listener has no span context at all — which is why the filter belongs on the logger and not on the handler.

For metrics, the same rule applies to exemplars: histogram.record() must be called while the span is current. Recording latency after the span has ended — in a finally block that runs after span.end(), for instance — produces a measurement with no exemplar and no error.

Production code examples

A complete three-signal setup sharing one resource, with correlation on all three joins.

# observability/otel.py
import logging
import os
import time
from opentelemetry import trace, metrics, _logs
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

RESOURCE = Resource.create({
    "service.name": os.environ["OTEL_SERVICE_NAME"],
    "service.version": os.environ.get("SERVICE_VERSION", "0"),
    "deployment.environment": os.environ.get("DEPLOY_ENV", "dev"),
})

def configure() -> None:
    tp = TracerProvider(resource=RESOURCE)
    tp.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(insecure=True)))
    trace.set_tracer_provider(tp)

    mp = MeterProvider(
        resource=RESOURCE,
        metric_readers=[PeriodicExportingMetricReader(
            OTLPMetricExporter(insecure=True), export_interval_millis=15000,
        )],
    )
    metrics.set_meter_provider(mp)

    lp = LoggerProvider(resource=RESOURCE)
    lp.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(insecure=True)))
    _logs.set_logger_provider(lp)

    root = logging.getLogger()
    root.addFilter(TraceContextFilter())            # on the logger, not the handler
    root.addHandler(LoggingHandler(level=logging.INFO, logger_provider=lp))
    root.setLevel(logging.INFO)

Using it in a handler, so all three signals fire for one request:

tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
latency = meter.create_histogram("http.server.request.duration", unit="s")
log = logging.getLogger("orders")

def get_order(order_id: int):
    with tracer.start_as_current_span("GET /orders/{id}") as span:
        span.set_attribute("http.route", "/orders/{id}")
        started = time.perf_counter()
        try:
            order = repository.fetch(order_id)
            log.info("order fetched", extra={"order_id": order_id})
            return order
        finally:
            # inside the span, so the exemplar carries this trace
            latency.record(
                time.perf_counter() - started,
                attributes={"http.route": "/orders/{id}"},
            )

Expected Output (the log record):

{"ts": "2026-08-02T14:19:03Z", "level": "INFO", "logger": "orders",
 "message": "order fetched", "order_id": 8812,
 "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7",
 "trace_flags": "01",
 "service.name": "checkout-api", "service.version": "2026.8.1"}

Expected Output (the metric, with its exemplar):

http_server_request_duration_seconds_bucket{http_route="/orders/{id}",le="5.0"} 1
  # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736",span_id="00f067aa0ba902b7"} 4.021 1754146743

Same trace ID in both. The comment line on the bucket is the exemplar — that is the link from the aggregate to the request.

Common mistakes

Three resources instead of one. Each provider gets its own Resource.create(...) call, they drift, and the backend shows the same process as two or three services. Build one object and pass it around.

Writing zero IDs. get_current_span() returns an invalid span outside a trace, and formatting its IDs produces thirty-two zeros. A search for that value matches every unrelated record in the system. Check is_valid and omit the fields.

Enriching in the handler. With a queue between the logger and the sink, the handler runs on a listener thread where no span is active, so every record gets no trace ID or a wrong one. The filter belongs on the logger.

Recording metrics after the span ends. The exemplar needs the span to be current. A finally that runs after the context manager exits records the value with no trace context and no complaint.

Exemplars with always_on. Exemplars are then produced for unsampled traces, so half the links point at traces the backend never received. trace_based is the correct filter.

A request ID that duplicates the trace ID. Carrying both is common during a migration and confusing afterwards: two identifiers for the same thing, and queries that use the wrong one silently return less. Pick the trace ID and treat the request ID as a compatibility field with an end date.

What correlation is worth

Correlation has a cost — fields on every record, exemplars on every histogram, a shared resource to maintain — and it is worth being specific about what it buys, because the argument for it is usually made in generalities.

It removes the time-window guess. Without a join key, finding the log records for a slow request means guessing a window, filtering by service, and reading. That is minutes of work performed under pressure, repeated for every hop, and it gets harder as traffic rises because the window contains more unrelated records. With a trace ID it is one query, and it returns exactly the records for that request across every service that took part.

It makes the cross-service question answerable at all. "Which downstream call made this request slow" is not answerable from logs alone unless every service logs durations for every call and someone correlates them by hand. It is what a trace is for, and the log records only become useful evidence once they are attached to it.

It changes what an alert can say. An alert that fires with a trace ID attached — from an exemplar, or from the log record that triggered it — starts the investigation at a specific request rather than at a dashboard. That is the difference between an alert that says "latency is up" and one that says "latency is up, here is a request that was slow".

Without correlation With it
Guess a time window, filter, read one query on the trace ID
"Which service was slow?" — check each one the waterfall shows it
"Why was that span slow?" — no answer the log records from inside it
An alert names a metric an alert names a request
Cross-signal questions need a human they need a query

The cost, honestly

Two fields per log record is roughly 80 bytes — a trace ID, a span ID, and their keys — which at 40 000 records per second is about 11 GB a day of pure join key. That is not free, and it is the price of every question above being answerable. Exemplars are cheaper: one per bucket per interval, attached to series that already exist.

The resource attributes are cheaper still, because most backends store them once per resource rather than per data point, and they are the part with the highest ratio of value to cost — service name and version on every signal is what makes any cross-signal query possible in the first place.

If the budget is tight, the order to add them in is: resource attributes first (nearly free, and everything else depends on them), then trace and span IDs on log records (the join people use most), then exemplars (the one that cannot be retrofitted, so worth adding before you need it rather than after).

Sampling and the joins

One interaction is worth anticipating: sampling discards traces, and two of the three joins depend on a trace existing. A log record carrying a trace ID for a trace that was never exported links to nothing, and an exemplar recorded under the trace_based filter is simply not produced for an unsampled span.

The log case is the one to think about, because the field is written regardless of the sampling decision. That is the right behaviour — the ID still groups the records of one request across services, which is useful even with no trace behind it — and it means a link that resolves to nothing is expected rather than broken. Recording the sampled flag alongside the IDs makes the distinction visible: a record with trace_flags of 01 has a trace to open, and one with 00 does not.

Where the fields come from at each layer

One implementation detail decides whether correlation survives contact with a real codebase: the fields must be added by infrastructure rather than by call sites. A service where every log call is expected to pass a trace ID will have correlation on the calls somebody remembered and nowhere else, which is worse than none, because the gaps are invisible and a query that returns nothing looks like an absence of events rather than an absence of instrumentation.

The three mechanisms that make it automatic are all described elsewhere on this site and are worth naming together here. A logging filter attached to the logger reads the active span and writes the IDs onto every record, including records from dependencies. A record factory or a shared Resource object attaches the deployment identity once. And the metrics SDK captures the exemplar itself, provided the measurement is recorded while the span is current.

None of the three requires anything at a call site, which is the property that makes correlation hold across a codebase with many authors and a long history. The only discipline the call sites retain is where to record a measurement — inside the span rather than after it — and that is a rule with a visible failure mode rather than an invisible one.

Correlation in a partially instrumented system

Few systems are instrumented all at once, and the intermediate state has a specific failure worth planning for: a trace that passes through an uninstrumented service comes out the other side as a new trace, so the two halves of the request appear unrelated. The uninstrumented service is invisible, which is expected, but it also breaks the link between the services on either side of it, which is not.

Two mitigations help. Instrument the edges first — the entry point and the services with the most downstream calls — so that the trace covers the widest possible span of the request before the gaps appear. And where a service genuinely cannot be instrumented, have the one before it record the downstream call's identifiers as span attributes, so an investigation can at least be continued by hand.

The same logic applies to log correlation during a rollout: a service that does not yet attach trace IDs still benefits from the ones that do, because the trace it participates in is discoverable from its neighbours. That makes correlation worth adopting incrementally rather than waiting for a coordinated change across every service.

Frequently Asked Questions

What is the minimum needed to correlate logs with traces?

Two fields on every log record: trace_id and span_id, taken from the active span at the moment the record is created. That is enough to search for a trace ID and get every log line from every service that participated. Everything else — exemplars, shared resource attributes, semantic conventions — makes correlation easier and broader, but this one pair is what makes it possible at all.

Why do my log records have a trace ID of all zeros?

Because there was no active span when the record was created. get_current_span returns an invalid span whose IDs are zero rather than raising, so the field is present and useless. Check whether the record came from outside a request — startup, a background thread, or a task created before the span existed — and omit the field entirely rather than writing zeros, which are indistinguishable from a real ID in a search.

What is an exemplar?

A sample value recorded on a metric together with the trace context that was active when it was recorded. On a latency histogram it means a bucket is no longer just a count: you can click the tail bucket and land on a trace that actually took that long. It is the join from aggregate to individual, and it is the one direction that is impossible to reconstruct after the fact.

Do all three signals need the same attributes?

The resource attributes yes — service name, version and environment must be identical or the signals appear to come from different systems. Span and metric attributes should follow the same semantic conventions where they overlap, so http.route means the same thing in both, but they do not need to be the same set; a metric has to stay low-cardinality while a span can carry a request ID.

Can I correlate without OpenTelemetry?

Yes, with a request ID generated at the edge and propagated in headers, which is what most services did before tracing. It gives you log-to-log correlation across services and nothing else: no timing, no parent-child structure, no metric join. It is worth doing as a first step and worth replacing, because the trace ID does everything the request ID does and more.

Where does baggage fit?

Baggage carries application-defined key-value pairs alongside the trace context, so a tenant ID set at the edge is readable in every downstream service. It is the mechanism for propagating business dimensions that all three signals may want to record. Use it sparingly — it travels in headers on every request, and it is not a place for anything sensitive.

Related Guides