Migrating from prometheus_client to OpenTelemetry

Many Python services started with prometheus_client and a /metrics endpoint, and many teams now want their metrics to travel with traces and logs through OpenTelemetry. The migration itself is small — a few dozen instruments in most services — and the risk is entirely in what depends on them: dashboards, alerts and SLO rules keyed on exact metric names, label names and bucket boundaries. This article describes a migration in stages that keeps every query working at each step. It belongs to OpenTelemetry vs Prometheus for Python metrics in the Python metrics and instrumentation section; for keeping prometheus_client metrics and forwarding them through OpenTelemetry instead, see bridging Prometheus metrics into OpenTelemetry.

Prerequisites

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-prometheus>=0.48b0,<1.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0"

Implementation steps

Step 1 — Inventory metrics and consumers. Every metric the service exposes, with its type, labels and buckets; and every query that refers to it. The second list is the one that matters, and Prometheus can help: searching the rule files and dashboard JSON for each metric name finds most consumers. Metrics with no consumers are candidates for deletion rather than migration.

curl -s localhost:8000/metrics | awk '/^# TYPE/ {print $3, $4}' | sort

Step 2 — Map each metric. The mapping is mostly direct.

Buckets need explicit attention. OpenTelemetry's default histogram boundaries differ from prometheus_client's, and any SLO query on le="0.3" breaks if 0.3 is not a boundary. A view pins them:

from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation

latency_view = View(
    instrument_name="http.server.request.duration",
    aggregation=ExplicitBucketHistogramAggregation(
        boundaries=(0.01, 0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5, 5.0)),
)

Step 3 — Choose names that translate to the existing ones. The Prometheus exporter turns dots into underscores, appends the unit, and adds _total to counters. The old http_request_duration_seconds is produced by an instrument named http.request.duration with unit s; the old orders_placed_total by a counter named orders.placed with an annotation unit such as {order}, which adds no suffix. Label names carry over as attribute keys. Getting this right means no query changes at all.

Step 4 — Run the migrated code as a canary. Running both instrument sets in one process does not work cleanly: the OpenTelemetry Prometheus reader registers on prometheus_client's global registry, so identical names would appear twice in one exposition. A canary avoids the problem. One replica runs the migrated code, serving the OpenTelemetry instruments on the usual port; its scrape target carries a distinct label, such as track="canary", set by a pod label and a relabel rule.

from prometheus_client import start_http_server
from opentelemetry import metrics
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.metrics import MeterProvider

reader = PrometheusMetricReader()                   # registers on the default registry
provider = MeterProvider(metric_readers=[reader], views=[latency_view])
metrics.set_meter_provider(provider)
start_http_server(9464)                             # the same port as before

The canary handles a fraction of the traffic, so raw rates differ from the stable replicas. Compare what does not depend on traffic share: the availability and latency ratios, the per-instance request rate, and the set of series and label values present. Ratios within a few hundredths of a percent and identical series sets are the signal to proceed.

Step 5 — Roll out. Deploy the migrated code to every replica and remove the canary label. With names preserved, queries see nothing change. Keep the previous release ready to redeploy for a few days, since rolling back is the fastest fix for anything the canary missed.

Step 6 — Optionally move to OTLP push. Replace the Prometheus reader with a periodic exporting reader and an OTLP exporter, and have the collector write to the same backend through its Prometheus remote-write exporter. The collector's exporter performs the same name translation, so series keep their names. Target labels that Prometheus used to attach at scrape time — job, instance, Kubernetes pod labels — now come from resource attributes, and need mapping so that queries filtering on them still match.

exporters:
  prometheusremotewrite:
    endpoint: https://prometheus.internal/api/v1/write
    resource_to_telemetry_conversion: {enabled: true}

Multi-process servers change the picture

A service using prometheus_client multiprocess mode under Gunicorn has no direct equivalent in OpenTelemetry. The SDK's Prometheus reader serves only its own process's values, so the canary stage under prefork shows exactly the unaggregated behaviour described in metrics in multi-process Python servers.

The practical path for such services is to skip the scrape-based cutover and go straight to OTLP push: each worker, initialised after fork, exports its own metrics with a worker identity, and the collector or backend aggregates. Queries that previously summed over one aggregated series now sum over one series per worker, which sum by handles without change, as long as the queries already aggregated rather than selecting a single series. Checking that every dashboard query uses sum or another aggregation, before the cutover, avoids panels that suddenly show one line per worker.

What changes for the people using the metrics

A migration done this way is invisible to dashboards, and still changes a few things for the engineers who work with the service's metrics day to day.

The most visible change for authors is where configuration lives. In prometheus_client, a histogram's buckets are declared with the histogram; in OpenTelemetry, they are declared in a view registered with the meter provider, often in a different module from the instrument. A new histogram added without a matching view gets the SDK's default boundaries, which silently breaks any SLO threshold that is not among them. A short convention — every histogram gets a view, in one module — prevents that, and a test that lists instruments without views catches it in CI.

The most useful new capability is exemplars: histogram points that carry the trace ID of a request that fell in a bucket, so a latency spike on a dashboard links straight to a trace of a slow request. Linking metrics to traces with exemplars covers enabling them, and they are often the feature that justifies the migration to the people who did not ask for it.

Configuration options

Verification

At each stage, run the service's SLO and alert queries against the new data and compare with the old: the availability ratio, the latency ratio at each threshold, and the burn-rate expressions from burn-rate alerts and error budgets. Agreement on those is the test that matters; agreement on raw counts alone can hide a bucket or label difference.

promtool query instant http://prometheus:9090 \
  'sum(rate(http_request_duration_seconds_bucket{le="0.3",track="canary"}[1h])) / sum(rate(http_request_duration_seconds_count{track="canary"}[1h]))'

Expected Output: a value within a few hundredths of a percent of the same query with track!="canary".

Common mistakes

Default OpenTelemetry buckets. Error signature: latency SLI shifting at the cutover. Root cause: the threshold no longer a boundary. Remediation: a view with the old boundaries.

Unit in the name. Error signature: http_request_duration_seconds_seconds in the backend. Root cause: the unit given twice. Remediation: the unit field only.

Renamed labels. Error signature: panels empty after cutover for one breakdown. Root cause: an attribute key that differs from the old label. Remediation: identical keys, or a relabel rule during transition.

Cutting over under Gunicorn with the Prometheus reader. Error signature: counters jumping between scrapes. Root cause: no multiprocess aggregation in the SDK. Remediation: OTLP push with per-worker export.

Queries that select a single series. Error signature: panels showing one line per worker after moving to OTLP. Root cause: queries that relied on pre-aggregated series. Remediation: aggregate with sum by in every panel before the cutover.

Both instrument sets in one process. Error signature: duplicate metric families in one exposition, or a registration error at startup. Root cause: the OpenTelemetry reader shares prometheus_client's global registry. Remediation: compare through a canary rather than inside one process.

A new histogram without a view. Error signature: an SLO query returning nothing for a recently added route group. Root cause: default boundaries that omit the threshold. Remediation: a view for every histogram, checked by a test.

No rollback plan. Error signature: a long evening restoring dashboards after a subtle mismatch. Root cause: the previous release no longer deployable. Remediation: keep it ready for a few days after rollout.

Frequently Asked Questions