Python Observability

Bridging Prometheus Metrics into OpenTelemetry

Most Python services with metrics have prometheus_client instrumentation and a /metrics endpoint, and adopting OpenTelemetry does not require throwing that away. This page covers the two bridging paths, what happens to metric names in transit, and how to migrate one family at a time. It builds on OpenTelemetry vs Prometheus for Python metrics, part of the Python metrics and instrumentation section.

Two ways across, and what each one costs Two paths from an existing Prometheus-instrumented Python service into an OpenTelemetry pipeline. In the first, the service is not modified at all: a Prometheus receiver in the Collector scrapes the existing metrics endpoint on the usual interval, converts the exposition into OTLP metric points, and forwards them through the same pipeline as every other signal. The cost is that the metric names are translated, so dashboards must be updated, and the scrape interval rather than an export interval now governs resolution. In the second, the OpenTelemetry SDK is added to the service and runs alongside the Prometheus client: new instruments are created with the OpenTelemetry API and exported over OTLP directly, while the existing instruments keep serving the old endpoint until their dashboards have moved. The cost is duplicated series for the duration of the migration and two libraries in the process. The recommendation drawn is to start with the first, because it separates the transport decision from the instrumentation decision, and to use the second for metrics being added or rewritten anyway. path 1 · no code change your service prometheus_client · /metrics Collector · prometheus receiver scrapes, translates to OTLP the same pipeline as traces and logs names are translated — dashboards move path 2 · both libraries, one family at a time your service prometheus_client · old otel sdk · new /metrics, while dashboards move OTLP, directly the same pipeline duplicated series while both run and two libraries in the process start with path 1 — it separates the transport decision from the instrumentation one, which is the coupling that makes this hard
Path one is a Collector configuration change and no code. That is the right first move, because it lets you decide about transport before deciding about instrumentation.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0" \
            "opentelemetry-exporter-prometheus>=0.48b0,<1.0.0"

export OTEL_SERVICE_NAME=orders-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative

Implementation

Step 1 — Scrape the existing endpoint from the Collector. No application change; a receiver configuration.

# otel-collector.yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: orders-api
          scrape_interval: 15s
          static_configs:
            - targets: ["orders-api.default.svc:8000"]
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "go_.*|python_gc_.*"
              action: drop                 # keep the runtime noise out of the pipeline

processors:
  batch: { timeout: 5s }

exporters:
  otlphttp/backend:
    endpoint: ${env:BACKEND_ENDPOINT}

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp/backend]

The service now participates in an OpenTelemetry pipeline with zero code change, and everything the Collector does for traces — redaction, routing, fan-out — applies to its metrics too. The Collector deployment shapes are in running the OpenTelemetry Collector for Python services.

Step 2 — Know what happens to the names. The translation follows conventions, and a metric that never followed them may not survive a round trip unchanged.

Prometheus OTLP Back to Prometheus
http_requests_total http.requests (sum, monotonic) http_requests_total
http_request_duration_seconds http.request.duration unit s http_request_duration_seconds
queue_depth queue.depth (gauge) queue_depth
db_pool_size_bytes_total ambiguous — two suffixes may not round-trip

The last row is the one to check before dashboards depend on it: a name carrying both a unit suffix and _total is ambiguous to the translator, and the result differs by version.

What the translator does to a name, in both directions One metric name traced through a round trip. Starting as http_request_duration_seconds in Prometheus exposition, the translator recognises the trailing seconds as a unit suffix, strips it, records the unit separately as the letter s, and converts the remaining underscores to dots, producing http dot request dot duration with unit s in OTLP. Exported back to a Prometheus-compatible endpoint, the dots become underscores again and the unit is re-appended as a suffix, restoring the original name exactly. A second example shows a counter: http_requests_total loses its total suffix on the way in, because OTLP represents monotonicity as a property of the data point rather than as part of the name, and regains it on the way out. The warning attached is that a name that carries both a unit suffix and a total suffix, or that uses a unit the translator does not recognise, does not necessarily return unchanged — and the failure is a dashboard that queries a series that no longer exists under that name. one name, there and back http_request_duration_seconds Prometheus exposition http.request.duration · unit "s" suffix moved into the unit field http_request_duration_seconds restored exactly http_requests_total a counter http.requests · monotonic sum "total" is a data-point property http_requests_total suffix re-added where it stops being reversible a name with both a unit suffix and _total, or a unit the translator does not know — and the symptom is a dashboard querying a name that no longer exists
Conventional names round-trip exactly. The ones that do not are the ones that were never conventional, and the failure appears as an empty dashboard panel.

Step 3 — Run both libraries while migrating. They are independent, so nothing conflicts.

# observability/metrics.py — during the migration
from prometheus_client import Counter                     # existing
from opentelemetry import metrics                         # new

# old: still serving /metrics, still on the dashboards
LEGACY_REQUESTS = Counter("http_requests_total", "Requests", ["route", "status"])

# new: exported over OTLP
meter = metrics.get_meter("orders-api")
requests = meter.create_counter("http.requests", description="Requests")

def record_request(route: str, status: int) -> None:
    LEGACY_REQUESTS.labels(route, str(status)).inc()      # remove once dashboards move
    requests.add(1, {"http.route": route, "http.response.status_code": status})

Two lines per event for the duration, and one metric family at a time — the alternative is a single change that moves every dashboard and every alert at once.

Step 4 — Set the temporality explicitly. The default differs by exporter and destination.

export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative   # Prometheus-shaped
# or
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta        # some vendors

Cumulative counters reset to zero on restart and the query layer compensates. Delta streams lose only the interval in flight but require the destination to accumulate. Pick for the destination, and set it in the environment so it is visible in the deployment rather than buried in code.

Step 5 — Expose an OpenTelemetry Prometheus endpoint if you still need one. Sometimes the target is the OpenTelemetry API with a Prometheus destination.

from prometheus_client import start_http_server
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.metrics import MeterProvider

reader = PrometheusMetricReader()
provider = MeterProvider(resource=RESOURCE, metric_readers=[reader])
metrics.set_meter_provider(provider)
start_http_server(9000)          # the client's own server, serving OTel-sourced metrics

Instrumentation is OpenTelemetry; exposition is Prometheus. This is the arrangement that lets a new service defer the transport decision entirely, which is the position the migration is trying to reach.

The order the migration has to happen in A four-phase migration timeline. In the first phase the Collector begins scraping the existing endpoint, so the metrics reach the new pipeline while nothing in the application changes and every existing dashboard keeps working from the original scrape. In the second, the OpenTelemetry SDK is added and one metric family at a time is recorded through both libraries, producing duplicated series that are expected and temporary. In the third, dashboards and alerts are repointed at the new series names — this is the phase that takes the longest and involves the most people, and it is the reason the duplication has to exist. Only in the fourth are the old instruments and the exposition endpoint removed, at which point the duplication ends. The failure mode marked is doing the fourth phase before the third, which turns a migration into an outage of the dashboards everyone uses to tell whether the migration worked. four phases, and the order is the whole point 1 · scrape it Collector receiver no code change at all 2 · both libraries one family at a time duplication is expected 3 · move the queries dashboards and alerts the longest phase 4 · retire old instruments and the endpoint doing 4 before 3 the names changed in translation, so every dashboard that still queries the old name goes empty at once including the dashboards everyone is using to judge whether the migration worked the duplication in phase 2 exists precisely so phase 3 can take as long as it takes
Phase three is the slow one, and it is the reason phase two's duplication exists. Skipping ahead empties the dashboards you would use to check the result.

Configuration options

Option Where Default Recommended
Prometheus receiver Collector absent the no-code first step
scrape_interval Collector 60 s 15–30 s
metric_relabel_configs Collector none drop runtime noise at the edge
Temporality env exporter-dependent set explicitly for the destination
Both libraries app — during migration only
PrometheusMetricReader app — OTel API, Prometheus exposition
Endpoint removal app — last, after dashboards move

Verification

Confirm the bridged series arrive with the translated names.

curl -s localhost:8000/metrics | grep http_requests_total

Expected Output:

http_requests_total{route="/orders/{id}",status="200"} 412.0

Expected Output (Collector debug exporter, after translation):

Metric #0
     -> Name: http.requests
     -> Description: Requests
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
     -> route: Str(/orders/{id})
     -> status: Str(200)
Value: 412

Same value, translated name, monotonicity as a property rather than a suffix. Before removing the old endpoint, check the last thing that matters:

# both series should agree while both are running
sum(rate(http_requests_total[5m])) - sum(rate(http_requests[5m]))

A non-zero difference means one path is missing events — usually because the new instrument is recorded on a code path the old one is not, or vice versa.

Common mistakes

Dashboards break on the day of the switch

Error signature: panels go empty although the data is arriving. Root cause: the names changed in translation and nothing updated the queries. Remediation: run both paths, migrate the queries while both series exist, and remove the old endpoint last.

Counters reset unexpectedly

Error signature: rate queries show negative values or large spikes after a deploy. Root cause: a temporality mismatch between the exporter and what the backend expects. Remediation: set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE explicitly for the destination.

Runtime noise floods the new pipeline

Error signature: the OTLP pipeline carries thousands of series nobody queries, from default collectors. Root cause: the Prometheus receiver scrapes everything the endpoint exposes. Remediation: add metric_relabel_configs to drop them at the receiver, which is the cheapest point available on this path.

What you gain, and what you give up

The migration is worth doing for specific reasons, and it is worth being clear about them because "we should be on OpenTelemetry" is not one.

Gained: one pipeline for three signals. Metrics that travel the same path as traces and logs inherit the same Collector, the same redaction policy, the same routing and the same credentials. In a fleet with more than one language that consolidation is the largest practical benefit, because the policy stops being reimplemented per ecosystem.

Gained: resource attributes for free. Service name, version and environment attached at the SDK rather than assembled from scrape labels, which means the metric agrees with the trace and the log record about which service produced it — the prerequisite for any cross-signal query.

Gained: exemplars end to end. The link from a latency bucket to a real trace works natively over OTLP, without the OpenMetrics negotiation and storage flags the Prometheus path requires.

Given up: pull-model simplicity. A scrape endpoint is trivially debuggable — curl it and read the state of the process. A push pipeline requires the Collector to be up to see anything, and a service that is not sending is indistinguishable from a service that is not running.

Given up: some ecosystem maturity. The Prometheus client library, its exposition format and its query language are older and have more surrounding tooling. The gap has narrowed considerably and it is not zero.

Property Prometheus scrape OTLP push
Debuggability curl the endpoint needs the Collector
Service identity scrape labels resource attributes
Exemplars OpenMetrics + a flag native
One pipeline for all signals no yes
Failure when the collector is down scrape gaps, data intact locally dropped after the queue fills

A hybrid that is often the right answer

The two are not mutually exclusive, and the arrangement that suits many services is to instrument with the OpenTelemetry API and expose a Prometheus endpoint through PrometheusMetricReader. The instrumentation becomes portable, the resource attributes are set once, Views are available for bucket and cardinality control — and the transport stays pull-based, keeping the debuggability and the existing scrape infrastructure.

Moving to OTLP later is then a reader swap rather than a re-instrumentation, which is the property that makes this a good default for a new service rather than a compromise.

# today: OTel API, Prometheus transport
provider = MeterProvider(resource=RESOURCE, metric_readers=[PrometheusMetricReader()])

# later: the same instruments, a different transport
provider = MeterProvider(resource=RESOURCE, metric_readers=[
    PeriodicExportingMetricReader(OTLPMetricExporter(insecure=True)),
])

Timing the migration

Two pieces of advice on sequencing, both learned expensively.

Do not migrate metrics and change the backend at the same time. Each one alone produces a set of unfamiliar numbers to validate; together they produce a set nobody can attribute to either change. Move the transport first with the destination unchanged where possible, confirm the numbers match, and only then change where they go.

And do the migration when nothing else is in flight. The window where both paths run is the window where every dashboard has two possible sources, and it is a bad time to also be investigating a performance regression — because the first question about any anomaly becomes "is that real, or is that the migration".

Frequently Asked Questions

Do I have to rewrite my prometheus_client instrumentation to adopt OpenTelemetry?

No. The Collector's Prometheus receiver scrapes your existing /metrics endpoint and converts what it finds into OTLP, so a service can join an OpenTelemetry pipeline with no code change at all. That is the right first step: it decouples the transport decision from the instrumentation decision, which are otherwise entangled in a single large migration.

What happens to metric names when they cross the bridge?

They are translated, and the translation is not always symmetric. Going Prometheus to OTLP, a trailing _total on a counter is stripped and unit suffixes such as _seconds are recognised and moved into the unit field. Going the other way, OpenTelemetry names using dots are converted to underscores and the suffixes are re-added. A round trip usually returns the original name, but a name that did not follow the conventions in the first place may not survive it — which is worth checking before dashboards depend on it.

Cumulative or delta temporality?

Cumulative when the destination is Prometheus or anything Prometheus-shaped: a counter that only ever increases, with resets detected by the query layer. Delta when a vendor expects it. The practical difference is restart behaviour — a cumulative counter resets to zero and the query layer compensates, while a delta stream loses only the interval in flight. Set it explicitly rather than relying on a default that may differ per exporter.

Can I run both the Prometheus client and the OpenTelemetry SDK at once?

Yes, and during a migration it is the safest arrangement. They are independent libraries with independent registries, so nothing conflicts, and the cost is duplicated series while both are active. Migrate one metric family at a time and remove the old instrument once its dashboards read the new series.

Which direction should a new service go?

Instrument with the OpenTelemetry API and choose the exporter later — either OTLP to a Collector, or a Prometheus endpoint through the OpenTelemetry Prometheus exporter. That keeps the instrumentation decision separate from the transport decision, which is exactly the coupling that makes this migration awkward for services that started the other way round.