Python Runtime and Service Metrics

Request metrics tell you the service is slow. Runtime metrics tell you whether the cause is inside the process. This guide covers the measurements that make that distinction — event loop lag, garbage collection, memory, pool saturation — for backend engineers and SREs who have request instrumentation and still find themselves guessing during incidents. It is part of the Python metrics and instrumentation section, and the walkthroughs are in monitoring Python GC and memory usage and measuring asyncio event loop lag.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "psutil>=5.9.0,<7.0.0"

export PROMETHEUS_MULTIPROC_DIR=/tmp/app_prom   # under gunicorn; see the note below
export RUNTIME_METRICS_INTERVAL=1.0             # seconds between loop-lag samples

Concept and architecture

Runtime metrics divide into four families, and each answers a question request metrics cannot.

Process and platform — resident memory, CPU seconds, open file descriptors, thread count. The prometheus_client default collectors provide these for free on Linux by reading /proc, and resident memory is the one number that corresponds to what the container limit enforces.

Interpreter — garbage collection counts and durations by generation, and object counts. A generation-2 collection walking a large object graph produces a pause during which nothing runs, which appears in latency and nowhere else.

Concurrency — event loop lag for async services, thread pool queue depth for threaded ones. This is the family with the best ratio of diagnostic value to instrumentation cost, and the one most often missing.

Resource pools — database connections checked out against the pool ceiling, HTTP connection pool saturation, queue depths. A pool at its limit produces waiting that is indistinguishable, from the outside, from a slow dependency.

All four share a property that makes them cheap: they describe the process, so they carry no request-derived labels and their series count does not grow with traffic. A service with two hundred request-metric series might have twenty runtime series, and they will still be twenty next year.

Step-by-step implementation

Step 1 — Turn on the default collectors. With prometheus_client, these are registered automatically unless you cleared the registry.

from prometheus_client import REGISTRY, start_http_server
# ProcessCollector, PlatformCollector and GCCollector are registered by default.
start_http_server(9000)

Expected Output:

process_resident_memory_bytes 2.68435456e+08
process_cpu_seconds_total 412.35
process_open_fds 87
python_gc_collections_total{generation="0"} 18422.0
python_gc_collections_total{generation="2"} 41.0

Under a prefork server these are not exported in multiprocess mode, because a single figure across four workers would be meaningless — scrape them from the container runtime instead. That mechanism is covered in Prometheus client instrumentation.

Step 2 — Measure event loop lag. Schedule a callback, compare due time with run time, record the difference.

import asyncio
import time
from prometheus_client import Histogram

LOOP_LAG = Histogram(
    "python_asyncio_loop_lag_seconds",
    "Delay between a callback's due time and its execution",
    buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)

async def monitor_loop_lag(interval: float = 1.0) -> None:
    loop = asyncio.get_running_loop()
    while True:
        due = loop.time() + interval
        await asyncio.sleep(interval)
        LOOP_LAG.observe(max(0.0, loop.time() - due))   # how late did we actually wake?

One coroutine, one series, and it converts a whole class of incidents from mysteries into measurements. The details — including why the histogram buckets matter more here than almost anywhere else — are in measuring asyncio event loop lag.

Step 3 — Record GC pauses, not just counts. The default collector gives counts; durations need a callback.

import gc
import time
from prometheus_client import Counter, Histogram

GC_PAUSE = Histogram(
    "python_gc_pause_seconds", "Garbage collection pause duration",
    ["generation"], buckets=(0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5),
)
_start: dict[int, float] = {}

def _gc_callback(phase: str, info: dict) -> None:
    gen = info["generation"]
    if phase == "start":
        _start[gen] = time.perf_counter()
    elif (started := _start.pop(gen, None)) is not None:
        GC_PAUSE.labels(str(gen)).observe(time.perf_counter() - started)

gc.callbacks.append(_gc_callback)

The generation label has exactly three values, which is the kind of label a runtime metric is allowed to have.

Step 4 — Instrument pool saturation. The useful number is not the pool size, which is a constant, but how much of it is in use.

from prometheus_client import Gauge

POOL_IN_USE = Gauge("db_pool_connections_in_use", "Checked-out connections", ["pool"])
POOL_SIZE = Gauge("db_pool_connections_max", "Configured pool ceiling", ["pool"])

def observe_pool(engine) -> None:
    pool = engine.pool
    POOL_IN_USE.labels("primary").set(pool.checkedout())
    POOL_SIZE.labels("primary").set(pool.size() + pool.overflow())

A ratio of in-use to maximum that sits at 1.0 is a queue forming, and it produces latency that looks exactly like a slow database — the queries are fast, and the waiting happens before they start.

Configuration reference

Async and concurrency considerations

Runtime metrics interact with the runtime they measure, which creates two specific traps.

A metrics endpoint served by the application renders the exposition synchronously. Under asyncio that work happens on the event loop, so a very large registry adds loop lag to the very metric that reports loop lag. At normal cardinality this is irrelevant; at hundreds of thousands of series it is self-referential, and the fix is fewer series rather than a different endpoint.

Under a prefork server, the default process collectors are disabled in multiprocess mode because a single resident-memory figure across four workers means nothing. Runtime metrics are per-process by nature, which is an argument for scraping them from the container runtime — where they are per-container and unambiguous — and keeping the application's registry for application-level signals.

The GC callback runs during collection, on whichever thread triggered it. Keep it to a timestamp and a histogram observation: anything that allocates inside a GC callback is asking for trouble.

Production code examples

A single module that installs everything and works in both async and threaded services.

# observability/runtime.py
import asyncio
import gc
import time
from prometheus_client import Counter, Gauge, Histogram

LOOP_LAG = Histogram(
    "python_asyncio_loop_lag_seconds", "Event loop scheduling delay",
    buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
GC_PAUSE = Histogram(
    "python_gc_pause_seconds", "GC pause duration", ["generation"],
    buckets=(0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5),
)
GC_UNCOLLECTABLE = Counter(
    "python_gc_uncollectable_total", "Objects GC could not free", ["generation"],
)
POOL_IN_USE = Gauge("db_pool_connections_in_use", "Checked-out connections", ["pool"])

_gc_started: dict[int, float] = {}

def _on_gc(phase: str, info: dict) -> None:
    gen = info["generation"]
    if phase == "start":
        _gc_started[gen] = time.perf_counter()
        return
    started = _gc_started.pop(gen, None)
    if started is not None:
        GC_PAUSE.labels(str(gen)).observe(time.perf_counter() - started)
    if info.get("uncollectable"):
        GC_UNCOLLECTABLE.labels(str(gen)).inc(info["uncollectable"])

def install_runtime_metrics() -> None:
    gc.callbacks.append(_on_gc)

async def run_loop_monitor(interval: float = 1.0) -> None:
    loop = asyncio.get_running_loop()
    while True:
        due = loop.time() + interval
        await asyncio.sleep(interval)
        LOOP_LAG.observe(max(0.0, loop.time() - due))

Wire it into a FastAPI lifespan:

from contextlib import asynccontextmanager
from fastapi import FastAPI
from observability.runtime import install_runtime_metrics, run_loop_monitor

@asynccontextmanager
async def lifespan(app: FastAPI):
    install_runtime_metrics()
    monitor = asyncio.create_task(run_loop_monitor(1.0))
    yield
    monitor.cancel()

app = FastAPI(lifespan=lifespan)

Expected Output:

python_asyncio_loop_lag_seconds_bucket{le="0.005"} 3591.0
python_asyncio_loop_lag_seconds_bucket{le="0.05"} 3597.0
python_asyncio_loop_lag_seconds_bucket{le="+Inf"} 3600.0
python_gc_pause_seconds_bucket{generation="2",le="0.05"} 39.0
python_gc_pause_seconds_bucket{generation="2",le="+Inf"} 41.0
db_pool_connections_in_use{pool="primary"} 18.0

Two of those readings are already findings: three loop-lag samples in an hour landed above 50 ms, and two generation-2 collections took longer than 50 ms. Neither would be visible in any request metric.

Common mistakes

Instrumenting requests and stopping there. The resulting dashboard can say the service is slow and cannot say whether the cause is inside the process. That gap is where incident time goes.

Adding request labels to runtime metrics. A route label on resident memory is meaningless — the process has one memory figure — and it multiplies a metric whose value is that it never grows.

Reading Python-level memory as a health signal. gc.get_objects() counts and tracemalloc totals are diagnostic tools for finding a leak's cause. The number that decides whether the container is killed is resident set size.

Sampling loop lag too rarely. A one-second interval catches sustained lag and misses short stalls. If the question is "did anything block the loop for 200 ms", sample every 100 ms and accept the extra wakeups.

Ignoring pool saturation. A pool pinned at its ceiling produces queueing before any query runs, so the database's own metrics look healthy while the service is not. Instrument the ratio, not the size.

Serving a huge registry from the event loop. The exposition is rendered synchronously; at very high cardinality that adds loop lag to the metric reporting loop lag. Cut the series count rather than moving the endpoint.

Building the dashboard

Runtime metrics are most useful arranged as a single panel that answers one question: is the cause inside this process? Five rows do that, and the arrangement matters as much as the metrics themselves, because the value is in reading them together.

Row one: the outcome. Request latency percentiles and error rate — not a runtime metric at all, but the thing everything else is explaining. Without it on the same screen, the other four rows are numbers with no reference point.

Row two: the loop. Event loop lag percentiles for an async service, or executor queue depth for a threaded one. This is the row that most often moves with row one, and when it does the investigation stops here and turns into a search for a blocking call.

Row three: memory and collection. Resident set size against the container limit, and generation-2 pause duration. These move slowly and together; a rise in both, with latency spiking at the collection cadence, is the signature described earlier.

Row four: pools. In-use against maximum for every connection pool. A ratio pinned at one is queueing, and queueing before work starts is invisible in every downstream system's own metrics.

Row five: the pipeline itself. Dropped spans, dropped log records, exporter queue depth. This row exists to answer "is the data I am looking at complete", which is a question worth being able to answer before drawing conclusions from a gap.

Alerting on runtime metrics

Most of these are diagnostic rather than alertable — they explain an incident rather than announce one — and treating them otherwise produces a page every time garbage collection has an unusual afternoon. Three exceptions are worth alerting on directly.

Resident memory above roughly 85% of the container limit, sustained, because the alternative to alerting is an out-of-memory kill with no warning. A pool pinned at its ceiling for more than a few minutes, because that is a saturation condition that will not resolve on its own. And a non-zero dropped-record counter, sustained, because it means the data underpinning every other alert is incomplete.

Everything else belongs on the dashboard rather than in the alert policy. Event loop lag is the clearest case: it is enormously useful when read next to a latency graph and almost useless as an alert, because a service can have brief lag spikes for perfectly ordinary reasons and the number that matters is whether it correlates with the outcome.

Thread and file-descriptor growth

Two process-level signals deserve a mention of their own, because both are leak indicators with a long lead time and neither is watched by default.

Thread count should be flat in a steady-state service: a fixed pool, a few background workers, the exporter threads. A count that climbs is nearly always a thread created per request, per connection or per task and never joined — a pattern that survives review because each individual creation looks reasonable. The failure is eventual and abrupt, when the process hits the system's thread limit.

Open file descriptors behave the same way and leak for a wider variety of reasons: a socket not closed on an error path, a file opened in a retry loop, a connection pool that creates and discards rather than reusing. Comparing the count against the soft limit is the useful form, because the absolute number is meaningless without it.

Both are already exported by the default process collector on Linux, which makes them free to watch and correspondingly easy to forget about entirely. A single panel with thread count and the descriptor ratio, looked at when anything else is odd, has a good track record of explaining slow-burning problems that no request-level metric describes — and both series are flat in a healthy service, which makes any movement in them worth a second look rather than an interpretation.

Runtime metrics under multiple processes

Every metric on this page describes one interpreter, which makes the multi-process case worth thinking through rather than inheriting.

Under a prefork server, each worker has its own memory, its own loop and its own collection cycle, so a single aggregated number across four workers is meaningless for most of them: resident memory summed across workers is not what the container limit compares against, and an averaged event loop lag hides the one worker that is stalling. That is why the Prometheus client disables the default process collectors in multiprocess mode rather than trying to combine them.

The practical arrangement is to take the process-level signals from the container runtime, where they are per-container and unambiguous, and to keep the interpreter-level and concurrency signals in the application registry with a per-worker label — accepting that the label multiplies the series by the worker count, which for a handful of runtime metrics is affordable.

import os
from prometheus_client import Gauge

WORKER = str(os.getpid())
LOOP_LAG = Histogram("python_asyncio_loop_lag_seconds", "Scheduling delay", ["worker"])
LOOP_LAG.labels(WORKER).observe(lag)

A process ID is not a stable label across restarts, which is normally a cardinality sin. It is tolerable here precisely because the series count is tiny and the alternative — an average that hides the outlier — defeats the purpose of the metric. Where the deployment provides a stable worker index, use that instead.

Overhead of the instrumentation itself

Worth stating because it comes up in review: the whole set costs very little. The default process collector reads /proc once per scrape rather than continuously. The GC callback is two clock reads per collection, which happens hundreds of times a second at generation 0 and is still a few microseconds a second in total. The loop-lag sampler is one coroutine waking ten times a second. The pool gauges are attribute reads on an object you already have.

The exception is anything that walks the object graph — gc.get_objects(), tracemalloc — which is genuinely expensive and belongs behind a diagnostic endpoint rather than in the steady-state metrics. That distinction between cheap continuous signals and expensive on-demand ones is worth keeping firmly, because the on-demand tools are the ones that look most attractive when you are trying to understand a leak.

Related

• Python metrics and instrumentation — the parent section: instrument types, cardinality and exposition.
• Monitoring Python GC and memory usage — the interpreter family in detail.
• Measuring asyncio event loop lag — the single highest-value runtime series.
• Prometheus client instrumentation in Python — registries, default collectors and multiprocess behaviour.
• Controlling label cardinality in Prometheus — why runtime metrics stay cheap.
• Logging from asyncio tasks without blocking — one of the most common causes of the lag you will measure.

Frequently Asked Questions