Python Metrics and Instrumentation: Architecture and Implementation Guide

Metrics are the cheapest, highest-leverage observability signal for Python backends: a handful of counters and histograms tell you request rate, error rate, and latency distribution at a fraction of the storage cost of traces or logs. This guide is for backend engineers and SREs who need to instrument production Python services correctly, and it links the focused guides that go deeper: instrumenting with the prometheus_client library, recording metrics with the OpenTelemetry metrics SDK, choosing metric types and controlling cardinality, and deciding between OpenTelemetry and Prometheus for Python metrics. Metrics also close the loop with the other two signals: they pair with distributed tracing in Python through exemplars, and with structured logging fundamentals through shared trace_id correlation.

Key architectural decisions to make before you write the first instrument:

• Transport model: choose pull (Prometheus scrape) or push (OTLP export) per service, deliberately, and make the choice consistent across a deployment tier so dashboards and alerting rules do not have to special-case it.
• Aggregation boundary: treat metrics as aggregates, not events — a metric is a pre-aggregated number per time series, never one data point per request — and decide explicitly where per-process values are combined into a service total.
• Cardinality budget: cap the label value sets per metric before shipping it; cardinality, not request volume, is what breaks a metrics backend.
• Histogram bucket layout: pick boundaries from your SLO thresholds rather than accepting library defaults, because quantiles are interpolated inside a bucket and buckets multiply your series count.
• Process topology: decide up front how a prefork worker pool reports — shared memory-mapped files under PROMETHEUS_MULTIPROC_DIR, or one independent exporter per worker tagged with an instance identifier.
• Resource identity: define service.name, service.version, and deployment.environment once and share that definition with traces and logs, because cross-signal correlation is only possible when all three agree on who produced the data.

Foundational Architecture and Metric Standards

A metric is a numeric measurement identified by a name and a set of key/value labels, sampled over time. The combination of metric name plus a unique set of label values defines a single time series. This is the unit your backend stores, indexes, and queries, and it is the unit that determines cost. Understanding that one metric expands into many time series — one per label combination — is the single most important mental model for instrumentation.

There are two dominant collection models in the Python ecosystem. The Prometheus pull model has each process expose a plain-text /metrics endpoint; a Prometheus server scrapes that endpoint on a fixed interval (commonly every 15 seconds) and stores the result. The application is passive — it only maintains current values in an in-process registry. The OpenTelemetry push model runs a metrics SDK inside the process that periodically reads instrument values and exports them over OTLP to a collector or backend. The application is active — it owns the export cadence.

The pull model gives the monitoring system control over scrape timing, makes liveness obvious (a failed scrape is a signal), and needs no per-app egress configuration. The push model fits short-lived jobs, serverless functions, and environments where the app cannot be reached by a scraper, and it unifies metrics with the same OTLP pipeline used for traces and logs. The detailed trade-offs are covered in OpenTelemetry vs Prometheus for Python metrics.

Both models share the same instrument families, and picking the wrong one is expensive to undo once dashboards and alerts depend on it:

Choosing correctly between these is consequential enough that it has its own guide on picking counter, gauge, histogram, and summary.

The Prometheus exposition format is the wire contract for the pull model. It is line-oriented UTF-8 text with # HELP and # TYPE comments followed by metric_name{label="value"} number samples. Because it is just text over HTTP, anything that can serve a response can expose metrics, which is why the format became a de facto standard well beyond Prometheus itself. OpenMetrics is its standardized successor and adds exemplars and explicit unit metadata; prometheus_client can serve either, negotiated by the scraper's Accept header.

Naming is part of the contract, not cosmetics. Prometheus convention is snake_case with a unit suffix and a _total suffix on counters — http_request_duration_seconds, http_requests_total, process_resident_memory_bytes — always in base units (seconds, not milliseconds; bytes, not megabytes), because dashboards and recording rules assume it. OpenTelemetry semantic conventions use dotted names and a separate unit field — http.server.request.duration with unit="s" — and the Prometheus exporter mechanically translates dots to underscores and appends the unit. Adopting the semantic conventions rather than inventing names means shared dashboards, alerts, and backend features work on your service without translation.

What you instrument matters as much as how. For request-driven services, the RED method (rate, errors, duration) gives complete coverage with three instruments: a request counter labeled by outcome, an error counter or an error-status label on the same counter, and a latency histogram. For resources — pools, queues, caches, workers — the USE method (utilization, saturation, errors) maps naturally onto gauges plus a counter. Start with those, then add domain metrics that describe business outcomes (orders placed, payments declined) because those are the ones on-call engineers actually alert on.

Instrumentation Strategy and SDK Configuration

The prometheus_client library is the canonical Python implementation of the pull model. You declare instruments once at module scope, mutate them inside request handlers, and expose the default registry over HTTP. Because instruments live in a global REGISTRY by default, declaring the same metric name twice raises a duplicate-timeseries error — a deliberate guard against accidental double registration. That guard also bites in test suites and in modules imported twice under different names, which is why long-lived codebases usually centralize declarations in one metrics.py module and import the instrument objects everywhere else. The companion guide on instrumenting with prometheus_client walks through registry management and the framework integrations, including adding Prometheus metrics to Flask and exposing custom application metrics.

The OpenTelemetry path centers on a MeterProvider configured with one or more metric readers. The PeriodicExportingMetricReader collects instrument values on an interval and hands them to an exporter such as OTLPMetricExporter. You acquire a Meter from the provider and create instruments — create_counter, create_histogram, create_up_down_counter, create_observable_gauge — from that meter. Observable (asynchronous) instruments take a callback that the reader invokes at collection time, which is the right pattern for values you sample rather than increment, like resident memory or pool size. The mechanics are detailed in the OpenTelemetry metrics SDK guide, with focused walkthroughs on exporting OTLP metrics to the collector and recording counters and histograms.

Provider lifecycle follows the same rules as the tracing SDK: build it once, as early in process startup as possible but after any fork, and shut it down deliberately. MeterProvider.shutdown() performs a final collection and export, which is the difference between having and not having the last ten seconds of data before a deploy or a crash — exactly the window an incident review needs. Register it in an atexit hook, a SIGTERM handler, or an ASGI lifespan shutdown. For batch jobs the final flush is not a nicety but the entire point: a job that finishes in eight seconds with a ten-second export interval reports nothing at all without it.

Configuration should be environment-driven so the same image runs in every tier without a rebuild. The OpenTelemetry SDK reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, and OTEL_METRIC_EXPORT_INTERVAL directly, so most deployments need no exporter arguments in code at all. On the Prometheus side the equivalent knobs are the scrape port, the bind address, and PROMETHEUS_MULTIPROC_DIR. Keep endpoints and credentials in environment variables, keep bucket boundaries and instrument names in code — the former change per environment, the latter must stay identical across environments or your dashboards will not compare.

Views are the OpenTelemetry SDK's configuration layer for instruments you do not own. A view matches instruments by name, kind, or meter and then rewrites them: rename the output metric, drop specific attribute keys, replace the default histogram bucket boundaries, or discard the instrument entirely. This is how you tame a noisy third-party library without forking it, and it is the push-model equivalent of Prometheus metric_relabel_configs.

Whichever SDK you choose, resource attributes such as service.name and deployment.environment must be attached so that series from different deployments stay distinguishable. In Prometheus these typically arrive as scrape-target labels or relabel rules; in OpenTelemetry they are set on the Resource passed to the MeterProvider, identically to how the tracing guide sets resources on the TracerProvider. Sharing one resource definition across signals is what makes cross-signal correlation work.

Label and Cardinality Discipline

Labels are the most powerful and most dangerous feature of dimensional metrics. A label like method or status_code has a small, fixed set of values and produces a manageable number of series. A label like user_id, session_id, a raw URL path with embedded IDs, or a full exception message is unbounded: it grows without limit as traffic flows, and each new value permanently allocates another time series. This is the number-one cause of Prometheus out-of-memory incidents.

The discipline is simple to state and easy to violate: every label must draw from a small, enumerable set known at design time. Normalize before labeling. Replace /orders/84321 with a route template /orders/{id}. Bucket a continuous quantity into ranges rather than labeling the raw value. Strip user-supplied strings entirely. The dedicated guide on controlling label cardinality in Prometheus covers route normalization, allow-lists, and how to find offending series before they cost you an outage.

A practical ceiling: estimate the cartesian product of all label values per metric before shipping it. A metric with method (5) × status (6) × endpoint (40) is 1,200 series — fine. The same metric with endpoint replaced by raw path is unbounded — a latent incident. Multiply that product by the number of replicas, because each instance contributes its own copy of every series, and a 30-pod deployment turns 1,200 into 36,000. When in doubt, drop the label; you can always add a dimension later, but you cannot cheaply reclaim the memory a bad one has already cost.

Cardinality also decays badly over time. A label whose values are bounded but churning — a deployment ID, a pod name, a version string — stays small at any instant while accumulating dead series in the backend's index, which is why long retention windows amplify a label that looked harmless in staging. Prefer attaching such identity to the scrape target or resource, where the backend can treat it as metadata, over baking it into every application-level series.

Two questions catch almost every bad label before it ships. First: can I write down the complete list of values this label will ever take? If not, it is unbounded. Second: will anyone ever group by or alert on this dimension? If not, it is dead weight even when bounded — put it in a structured log line or a span attribute instead, where high-cardinality context belongs and costs far less.

Histogram Bucket Design for Latency SLOs

A histogram is only as useful as its bucket boundaries. prometheus_client ships default buckets tuned for sub-second web latency (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10 seconds), but defaults rarely match a specific SLO. If your SLO is "99% of requests under 300 ms," you need a bucket boundary at or very near 0.3, because Prometheus computes quantiles by linear interpolation within a bucket — a boundary far from your target produces a quantile estimate that is wrong by exactly the bucket width.

Design buckets around the thresholds you actually report on. Place explicit boundaries at your SLO targets (for example 0.1, 0.3, 0.5, 1, 2, 5) and add resolution where your traffic concentrates. Logarithmically spaced buckets give good coverage across orders of magnitude when latency varies widely. Remember that every bucket is a separate time series multiplied by every label combination, so a 12-bucket histogram with 1,200 label combinations is over 14,000 series for one metric — bucket count is a cardinality lever too.

There is a second, subtler reason to place a boundary exactly at the SLO threshold: it makes the error budget query exact rather than estimated. Dividing the bucket counter at le="0.3" by the total count gives the true fraction of requests under 300 ms with no interpolation at all, which is the correct way to compute an SLO compliance ratio. Quantile queries are for exploration; bucket ratios are for alerting.

The reason histograms beat summaries for SLOs is aggregation. Histogram buckets are counters, and counters from many instances can be summed by the backend before computing a quantile, giving you a correct fleet-wide p99. Summary quantiles are computed per process and cannot be averaged into a meaningful global quantile. For any latency metric you intend to alert on across a fleet, use a histogram.

One caution when changing boundaries: a histogram's bucket layout is part of its identity. Editing the boundary list on a live metric makes the new series incompatible with the old ones, so quantile queries spanning the deploy return nonsense. Either accept a gap, or ship the new layout under a new metric name and retire the old one after the retention window rolls over.

Async and Concurrency Patterns

Recording a metric is one of the few observability operations cheap enough to do anywhere. Both prometheus_client and the OpenTelemetry SDK guard instrument state with short-lived locks, so counter.inc() and histogram.observe() are thread-safe and cost microseconds. There is no network call, no serialization, and nothing to await — which means you should never wrap a metric call in a thread executor or an async helper. Call it inline, including from coroutines.

The event loop hazards live around the instruments, not in them. Three patterns cause real incidents. First, observable-instrument callbacks: the OpenTelemetry reader invokes them synchronously on its own collection thread, and they must be plain functions that return immediately. A callback that queries Redis, opens a database cursor, or blocks on a lock stalls the entire export cycle and, if it contends with the loop's own resources, drags request latency with it. Sample the value on a background asyncio task or a timer and let the callback return the last cached number. Second, /metrics handlers that compute rather than serialize (covered below). Third, gauges updated without finally: an in-flight-requests gauge incremented at the start of a coroutine and decremented at the end leaks permanently on cancellation, and asyncio cancels tasks routinely on client disconnects and timeouts. Use a context manager or try/finally so CancelledError still decrements.

The event loop itself deserves instrumentation. Loop lag — the delay between when a callback was scheduled and when it ran — is the single best indicator that some coroutine is blocking, and it is trivial to measure: schedule a task that sleeps a fixed interval, compare the actual elapsed time to the intended interval, and record the difference in a gauge or histogram. Complement it with the number of pending tasks and the executor queue depth. When latency rises and loop lag rises with it, the problem is blocking code in the loop, not a slow dependency — a distinction that would otherwise take a long profiling session to reach.

Thread and process boundaries behave differently for metrics than for traces. Instruments are global objects, so a ThreadPoolExecutor worker incrementing a counter needs no context handoff at all — this is where metrics are strictly easier than tracing, which requires copying context across the boundary as described in async tracing patterns. What does need context is exemplars: attaching the active trace_id to an observation reads the current OpenTelemetry context via contextvars, so an exemplar recorded on a bare executor thread will be empty unless the context was propagated. Record the observation on the coroutine that owns the span, not on the worker thread.

The GIL rarely limits metric recording, because the critical section is a few bytecodes long. It does matter for the collection path: serializing tens of thousands of series into exposition text is CPU work that holds the GIL, so a very large registry scraped every five seconds can measurably steal time from request handling. If a scrape takes hundreds of milliseconds, the fix is fewer series, not a faster serializer.

Multiprocess Collection Under Gunicorn and Uvicorn

The pull model has a sharp edge with WSGI/ASGI worker pools. Under gunicorn (or uvicorn with multiple workers), each worker is a separate OS process with its own in-memory registry. A single scrape of one shared port reaches exactly one randomly chosen worker, so counters appear to bounce around and reset — you are sampling one worker's view, not the service total.

prometheus_client solves this with multiprocess mode. You set the PROMETHEUS_MULTIPROC_DIR environment variable to a writable directory; each worker writes its metric state to memory-mapped files there. The scrape endpoint then builds a fresh CollectorRegistry, attaches a MultiProcessCollector pointed at that directory, and aggregates across all workers at scrape time. Counters and histograms sum correctly; gauges support modes like livesum, max, and liveall to control how per-process values combine.

Multiprocess mode has operational requirements that are easy to miss. The directory must be empty at startup — stale files from a previous run are read as if those processes still existed, inflating counters after every restart — so clear it in the entrypoint. It should live on a tmpfs or emptyDir volume, not on durable storage, because every increment touches a memory-mapped page. Worker death must be reported via multiprocess.mark_process_dead(worker.pid) from gunicorn's child_exit hook, or a recycled worker's series linger. And a few features are unavailable in this mode: custom collectors that compute at scrape time do not participate, and the default process and platform collectors report per-process values that need care when aggregated.

The OpenTelemetry push model sidesteps this differently: each worker process runs its own MeterProvider and exports independently over OTLP, tagging exports with a process or instance identifier. The backend aggregates across instances. There is no shared-file dance, but you must ensure the SDK is initialized after the worker forks, because exporter background threads and HTTP connections do not survive fork(). Initialize in a gunicorn post_fork hook or an ASGI lifespan startup, never at import time before the master forks. The same rule applies to Celery workers with the prefork pool, where the worker_process_init signal is the correct hook.

Container schedulers add one more consideration: with the push model every replica is a distinct series producer, so an autoscaling deployment that churns pods produces a stream of short-lived instance identifiers. Aggregate away the instance dimension in recording rules or collector processors when you only ever query the service total.

Network and Protocol Integration

For the pull model, the contract is a single HTTP GET /metrics returning exposition text with a Content-Type of text/plain; version=0.0.4, or the OpenMetrics content type when the scraper negotiates it via Accept — which is required for exemplars to travel at all. prometheus_client provides start_http_server(port) for a standalone thread, make_wsgi_app() to mount inside an existing WSGI app, make_asgi_app() for ASGI frameworks, and generate_latest(registry) to render the body yourself for custom routes. Mounting inside your app reuses its port and TLS; a separate port isolates metrics from request traffic, keeps them off your public listener, and lets you firewall it independently. In Kubernetes the separate port is usually the better default, because a ServiceMonitor or scrape annotation can target a named port that no ingress ever exposes.

Keep the endpoint cheap. It should read current registry state and serialize — never trigger database queries or recompute expensive values inline. For values that are expensive to sample (queue depth from an external system, for instance), refresh them on a background timer or via an observable instrument and let the endpoint serve the cached number. A /metrics handler that blocks on I/O will time out scrapes under load and create gaps exactly when you most need data. Set the scrape timeout below the scrape interval so a slow endpoint degrades into missing samples rather than overlapping scrapes.

For the push model the network surface is OTLP. The SDK speaks gRPC on port 4317 or HTTP/protobuf on port 4318, selected by OTEL_EXPORTER_OTLP_PROTOCOL or by importing the matching exporter class. gRPC keeps one multiplexed connection open and has the lowest per-export cost; HTTP/protobuf traverses proxies, service meshes, and egress rules that only understand HTTP. Authentication travels as headers — OTEL_EXPORTER_OTLP_HEADERS carries API keys for managed backends — and TLS is negotiated by the endpoint scheme. Compression (gzip) is worth enabling on any link leaving the node, since exposition-shaped payloads compress extremely well.

Where the two worlds meet, three bridges matter. The Prometheus exporter in the OpenTelemetry SDK turns a MeterProvider into a scrapeable endpoint, letting you instrument with the vendor-neutral API while the platform team keeps pulling. The collector's Prometheus receiver does the reverse: it scrapes your existing exposition endpoints and forwards them as OTLP, which is how a fleet migrates without touching application code. And remote-write ships Prometheus data onward to long-term storage. Choosing among these is a topology decision, covered end to end in OpenTelemetry vs Prometheus for Python metrics.

Short-lived processes are the one case pull genuinely cannot serve. A batch job or cron task may finish before any scrape fires, so either push a final OTLP export on shutdown or write its result to a Pushgateway that Prometheus scrapes on the job's behalf. Do not reach for a Pushgateway for long-lived services — it becomes a stale-metric graveyard, since it keeps serving the last value forever after the pusher disappears.

Data Volume Control and Cost Management

Metrics cost is dominated by active series count, not request rate, because storage and memory scale with the number of distinct time series a backend must hold in its head block. The levers are therefore all about series count: fewer labels, bounded label values, fewer histogram buckets, and dropping series you never query. Audit your metrics periodically and delete instruments and labels nothing alerts or dashboards on — unused series are pure cost.

On the pull side, control cost with scrape interval and metric_relabel_configs that drop noisy series at ingestion. Raising an interval from 15s to 30s halves sample volume without changing series count, which helps storage but not memory — series count is the memory lever, sample rate is the disk lever, and knowing which one you are constrained by decides which knob to turn. Recording rules pre-compute expensive aggregations so dashboards query small derived series instead of scanning raw ones.

On the push side, the OpenTelemetry SDK supports views that rename instruments, drop attributes, or change histogram bucket boundaries before export, plus delta vs cumulative temporality choices that affect backend storage. Delta temporality sends only the change since the last export, which keeps payloads small and suits backends that sum on ingest; cumulative sends running totals, which is what Prometheus expects and what survives a dropped export without losing data. The collector adds a second filtering tier — processors can drop metrics, trim attributes, and batch — and it is the right place to enforce policy, because changing collector config does not require redeploying every service.

Exemplars add a small, bounded cost (a trace_id attached to a sampled bucket) and are worth it for the trace correlation they unlock. The same trace_id injected into your structured logs lets a latency spike jump straight to the offending logs and traces — one dashboard click from "p99 doubled at 14:03" to the exact request that caused it, without a text search.

Production Code Examples

Prometheus Instrumentation with a Latency Histogram

Declares instruments once at module scope, records request count and latency with bounded labels, and exposes them on a dedicated metrics port.

# pip install "prometheus-client>=0.20.0,<1.0.0"
import time
from prometheus_client import Counter, Histogram, start_http_server

# Bounded labels only: method and a normalized route template, plus status class.
REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests.",
    ["method", "route", "status"],
)
LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency in seconds.",
    ["method", "route"],
    # Buckets placed around a 300ms p99 SLO target.
    buckets=(0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5),
)


def handle(method: str, route: str) -> int:
    start = time.perf_counter()
    status = "200"
    try:
        time.sleep(0.12)  # simulated work
        return 200
    finally:
        LATENCY.labels(method, route).observe(time.perf_counter() - start)
        REQUESTS.labels(method, route, status).inc()


if __name__ == "__main__":
    start_http_server(9100)  # serves GET /metrics on :9100
    handle("GET", "/orders/{id}")
    handle("GET", "/orders/{id}")

Expected Output: A scrape of http://localhost:9100/metrics returns exposition text:

# HELP http_requests_total Total HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/orders/{id}",status="200"} 2.0
# HELP http_request_duration_seconds Request latency in seconds.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.2",method="GET",route="/orders/{id}"} 0.0
http_request_duration_seconds_bucket{le="0.3",method="GET",route="/orders/{id}"} 2.0
http_request_duration_seconds_bucket{le="+Inf",method="GET",route="/orders/{id}"} 2.0
http_request_duration_seconds_count{method="GET",route="/orders/{id}"} 2.0
http_request_duration_seconds_sum{method="GET",route="/orders/{id}"} 0.24...

OpenTelemetry Metrics SDK with OTLP Export

Builds a MeterProvider with a periodic reader and OTLP exporter, then records a counter and a histogram from a meter.

# pip install "opentelemetry-sdk>=1.30.0,<2.0.0" \
#   "opentelemetry-exporter-otlp-proto-grpc>=1.30.0,<2.0.0"
import os
import time
from opentelemetry import metrics
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

resource = Resource.create({
    "service.name": os.getenv("OTEL_SERVICE_NAME", "order-service"),
    "deployment.environment": os.getenv("DEPLOY_ENV", "production"),
})

# Reader collects every 10s and pushes over OTLP. Initialize AFTER any fork.
reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "localhost:4317")),
    export_interval_millis=10_000,
)
provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

meter = metrics.get_meter("order-service")
requests = meter.create_counter("http.server.requests", unit="1")
latency = meter.create_histogram("http.server.duration", unit="s")


def handle(route: str) -> None:
    start = time.perf_counter()
    time.sleep(0.12)
    attrs = {"http.route": route, "http.status_code": 200}
    latency.record(time.perf_counter() - start, attrs)
    requests.add(1, attrs)


handle("/orders/{id}")
provider.shutdown()  # final collection + export; never exit without it

Expected Output: No console output; every 10 seconds — and once more on shutdown — the reader exports an OTLP payload. A debug collector logs a representative metric:

{
  "name": "http.server.duration",
  "unit": "s",
  "histogram": {
    "dataPoints": [
      {
        "attributes": {"http.route": "/orders/{id}", "http.status_code": 200},
        "count": 1,
        "sum": 0.121,
        "bucketCounts": [0, 0, 1, 0]
      }
    ],
    "aggregationTemporality": "CUMULATIVE"
  }
}

Multiprocess Aggregation Under Gunicorn

Aggregates per-worker metric files into one registry at scrape time so a multi-worker deployment reports correct service-wide totals.

# pip install "prometheus-client>=0.20.0,<1.0.0" "gunicorn>=21.2.0,<23.0.0"
# Run with: PROMETHEUS_MULTIPROC_DIR=/tmp/prom gunicorn -w 4 app:app
import os
from prometheus_client import CollectorRegistry, generate_latest, CONTENT_TYPE_LATEST
from prometheus_client import multiprocess


def metrics_app(environ, start_response):
    # Build a fresh registry per scrape and aggregate all worker files.
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)  # reads PROMETHEUS_MULTIPROC_DIR
    data = generate_latest(registry)
    start_response("200 OK", [("Content-Type", CONTENT_TYPE_LATEST)])
    return [data]


def child_exit(server, worker):
    # Required so a dead worker's series are cleaned up.
    multiprocess.mark_process_dead(worker.pid)


def on_starting(server):
    # Stale files from a previous run would inflate counters after a restart.
    path = os.environ["PROMETHEUS_MULTIPROC_DIR"]
    os.makedirs(path, exist_ok=True)
    for name in os.listdir(path):
        os.remove(os.path.join(path, name))

Expected Output: A scrape returns counters summed across all four workers rather than one worker's partial count, e.g. http_requests_total{...} 812.0 instead of a number that resets each scrape.

Common Mistakes

• Putting unbounded values in labels: Labeling with user_id, raw URLs, request IDs, or exception messages creates an unbounded number of time series and is the most common cause of Prometheus running out of memory. Normalize to bounded templates before labeling.
• Running multiple gunicorn/uvicorn workers without multiprocess mode: Without PROMETHEUS_MULTIPROC_DIR and MultiProcessCollector, each scrape hits one random worker, so counters look like they reset and totals are silently wrong.
• Using a summary when you need fleet-wide quantiles: Summary quantiles are computed per process and cannot be aggregated, so your "p99" is really one instance's p99. Use a histogram so the backend can compute a correct global quantile.
• Default histogram buckets that miss the SLO threshold: Quantiles are interpolated within a bucket, so if no boundary sits near your SLO target the p99 estimate is off by the bucket width. Place explicit boundaries at the thresholds you alert on.
• Initializing the OTel MeterProvider before the worker forks: Exporter threads and connections do not survive fork(), so a provider built at import time silently stops exporting in forked workers. Initialize in a post_fork hook or lifespan startup.
• Doing expensive work in the /metrics handler or an observable callback: Querying a database or recomputing values inline blocks scrapes and stalls the export cycle, creating data gaps under load. Sample expensive values on a background task and serve the cached number.
• Exiting without a final flush: A batch job or a pod receiving SIGTERM drops whatever the reader has not yet exported, losing exactly the window before a crash or deploy. Call MeterProvider.shutdown() from a termination hook.
• Leaving a gauge incremented on cancellation: An in-flight counter raised at the start of a coroutine and lowered at the end leaks on CancelledError, so the gauge drifts upward forever. Wrap it in try/finally or a context manager.

Taken together, these decisions compose into one pipeline: bounded instruments declared once, buckets aligned to the thresholds you alert on, a process model that aggregates worker state correctly, a transport chosen to match how your workloads live and die, and a filtering tier that keeps series count — the thing you actually pay for — under control. Each guide in this section drills into one stage of that pipeline, and they are designed to fit together into a low-overhead metrics layer that correlates cleanly with your traces and logs.

Related Reading

• Instrumenting with the prometheus_client Library — registries, exposition endpoints, framework integration, and multiprocess mode.
• The OpenTelemetry Metrics SDK in Python — meter providers, readers, views, and OTLP export.
• Metric Types and Cardinality — choosing instruments and keeping label sets bounded.
• OpenTelemetry vs Prometheus for Python Metrics — the push-versus-pull decision and the bridges between them.
• Runtime and Service Metrics — event loop lag, GC pauses, memory and pool saturation: the signals that say whether the cause is inside the process.
• Distributed Tracing and OpenTelemetry in Python and Python Logging Fundamentals — the two signals metrics correlate with, both reachable from the Python observability guides.

Frequently Asked Questions