Linking Metrics to Traces with Exemplars

An alert tells you the tail got slower. A trace tells you why one request was slow. An exemplar is the only thing that connects the two — a trace ID recorded alongside a measurement, so the tail bucket in a histogram becomes clickable. This page covers emitting them from Python, the two things that silently prevent them, and how to verify one arrived. It builds on correlating logs, traces and metrics, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0" \
            "prometheus-client>=0.20.0,<1.0.0"

export OTEL_METRICS_EXEMPLAR_FILTER=trace_based
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

Implementation

Step 1 — Set the exemplar filter. trace_based records an exemplar only when the active span is sampled, which keeps every link pointing at a trace the backend actually received.

export OTEL_METRICS_EXEMPLAR_FILTER=trace_based    # the default, and the right choice

The alternatives are worth knowing so you recognise them in someone else's configuration: always_on records exemplars regardless of sampling, producing links to traces that were never exported; always_off disables them entirely.

Step 2 — Record the measurement inside the span. This is where exemplars are actually lost. The context manager must still be open when record() runs.

import time
from opentelemetry import trace, metrics

tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
latency = meter.create_histogram("http.server.request.duration", unit="s")

def handle(request):
    with tracer.start_as_current_span("GET /orders/{id}") as span:
        span.set_attribute("http.route", "/orders/{id}")
        started = time.perf_counter()
        try:
            return do_work(request)
        finally:
            latency.record(                          # still inside the with-block
                time.perf_counter() - started,
                attributes={"http.route": "/orders/{id}"},
            )

Move that latency.record() one line down — outside the with — and everything still works, no error appears, and every exemplar silently disappears. That is the single most common cause of "we enabled exemplars and got none".

Step 3 — Keep the identity in the exemplar, not in the labels. This is what exemplars are for: a histogram must stay low-cardinality, and the exemplar carries the specific request without adding a series.

# wrong: one series per user, forever
latency.record(elapsed, attributes={"http.route": route, "user_id": user_id})

# right: the route is a label; the request is an exemplar, attached automatically
latency.record(elapsed, attributes={"http.route": route})

The cardinality argument in full is in controlling label cardinality in Prometheus; exemplars are the escape hatch that makes the strict rule livable.

Step 4 — Expose them. On the OTLP path, exemplars travel inside the metric point and need no extra configuration. On the Prometheus path, the scrape must negotiate OpenMetrics and the server must have exemplar storage enabled.

# prometheus_client, on a histogram observation
from prometheus_client import Histogram
from opentelemetry import trace

LATENCY = Histogram("http_request_duration_seconds", "Latency", ["route"])

span_ctx = trace.get_current_span().get_span_context()
LATENCY.labels(route="/orders/{id}").observe(
    elapsed,
    exemplar={"trace_id": format(span_ctx.trace_id, "032x")} if span_ctx.is_valid else None,
)

# prometheus.yml — the storage side
storage:
  exemplars:
    max_exemplars: 100000

Step 5 — Reconcile sampling with drillability. At 1% sampling, 99 of every 100 measurements produce no exemplar. For the routes you most need to investigate, raise the sampling rate rather than the exemplar filter — the filter only decides whether to record a link to a trace that may not exist.

Configuration options

Verification

On a Prometheus scrape, the exemplar is a trailing comment on the bucket line:

curl -s -H 'Accept: application/openmetrics-text' localhost:8000/metrics | grep -A1 'le="5.0"'

Expected Output:

http_request_duration_seconds_bucket{route="/orders/{id}",le="5.0"} 41.0 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 4.021 1754146743.0

The part after the # is the exemplar: the trace ID, the measured value, and the timestamp. Without the Accept header the server returns the older text format and the comment is absent — which looks exactly like exemplars not working.

On the OTLP path, read the Collector's debug exporter:

Expected Output:

Histogram #0
  -> route: Str(/orders/{id})
  Count: 41
  Sum: 63.882
  ExemplarValue: 4.021
  ExemplarTraceID: 4bf92f3577b34da6a3ce929d0e0e4736
  ExemplarSpanID: 00f067aa0ba902b7

Then close the loop: take that trace ID to the tracing backend and confirm it opens a request whose duration matches the exemplar value. A mismatch means the measurement is being recorded against a different span than the one you think.

Common mistakes

Exemplars appear in staging and not in production

Error signature: the same code produces exemplars locally and none in production. Root cause: production samples at a low rate and the trace_based filter correctly skips unsampled spans. Remediation: compare the exemplar rate with the sampling rate. If drilling into a specific route matters, raise sampling for that route rather than changing the filter.

The bucket line has no trailing comment

Error signature: the Prometheus scrape looks normal and contains no exemplars. Root cause: the scrape negotiated the classic text format, which cannot carry them, or exemplar storage is off. Remediation: confirm the Accept header negotiates OpenMetrics and that storage.exemplars.max_exemplars is set.

Every exemplar points at a trace that does not exist

Error signature: clicking an exemplar opens an empty trace view. Root cause: the filter is always_on, so exemplars were recorded for spans that sampling discarded. Remediation: set trace_based, which is also the default.

Which metrics deserve exemplars

Not every instrument benefits, and adding them indiscriminately produces storage cost without investigative value. Three properties identify the ones worth it.

The metric aggregates something that varies per request. Latency histograms are the canonical case: the aggregate hides individual requests, and finding one is exactly what an exemplar provides. A gauge reporting queue depth does not have this property — there is no individual request behind the value — and correspondingly most implementations do not support exemplars on gauges at all.

Someone will want to drill in from an alert. If the metric appears in an alert or on an incident dashboard, the first question after "it moved" is "show me one". If it appears only in a capacity report, nobody will ever click it.

The tail is where the interest is. Exemplars are most valuable on the buckets that contain the unusual cases, which is another way of saying they are most valuable on distributions with a meaningful tail. A metric whose values are tightly clustered has an uninteresting exemplar in every bucket.

Reading an exemplar responsibly

An exemplar is one example, chosen by whichever measurement happened to be recorded when the storage had room for it. It is not a representative sample, and treating it as one leads to a specific mistake: concluding that because the exemplar for the tail bucket shows a slow database call, all the requests in that bucket were slow for the same reason.

The disciplined reading is that an exemplar proves at least one request in that bucket looked like this. Confirming the pattern means opening several — most backends keep more than one over a window — or, better, using the exemplar to form a hypothesis and then testing it against the aggregate: if the theory is a slow database, the database call duration metric should show the same shape at the same time.

That is a small discipline and it prevents the failure mode where an incident is diagnosed from a single trace that turns out to be unrepresentative.

The Prometheus specifics

Two details decide whether exemplars work end to end on the Prometheus path, and both live outside the application.

The scrape must negotiate the OpenMetrics format, because the classic text format has no syntax for an exemplar. Prometheus does this automatically when the target advertises support, but a scrape configured with an explicit Accept header, or a proxy that rewrites it, silently downgrades — and the symptom is missing exemplars with everything else working normally.

And the server needs exemplar storage enabled, with a size limit that bounds how many it retains. That storage is separate from the series storage and is small, but it is off by default in some configurations, which is the second place to look when the exposition clearly contains exemplars and the queries return none.

storage:
  exemplars:
    max_exemplars: 100000

On the OTLP path neither applies: exemplars travel inside the metric point and the backend either supports them or does not, with no negotiation to get wrong.

Related

• Correlating logs, traces and metrics — the parent guide: the three joins and the shared resource.
• Exporting Python logs through OpenTelemetry — the logs half of the correlation.
• Controlling label cardinality in Prometheus — why request identity belongs in an exemplar and not a label.
• Recording counters and histograms with OpenTelemetry — the instruments these exemplars attach to.
• Sampling strategies for distributed tracing — the rate that decides how many exemplars you get.

Frequently Asked Questions