Python Observability

Monitoring Python GC and Memory Usage

Memory metrics in Python are easy to collect and easy to misread: the number the container kills you for and the number your objects account for are not the same, and neither one shows the pauses that turn a memory problem into a latency problem. This page covers the three signals worth exporting and how to read them together. It builds on runtime and service metrics, part of the Python metrics and instrumentation section.

Three numbers, all called "memory" Python memory drawn as three nested layers. The outermost is resident set size, the memory the operating system has actually given the process, and the figure a container limit compares against; it includes the interpreter, loaded extension modules, thread stacks and everything below. Inside it are the allocator's arenas: blocks CPython has requested from the operating system and manages itself, which are not returned promptly when objects are freed and can stay resident because of fragmentation even when almost empty. Innermost are the live Python objects themselves, which is what garbage collection and tracemalloc measure. The gap between the innermost and outermost layers is why freeing a large number of objects often produces no change in resident memory at all, and why alerting on object counts tells you nothing about whether the container is about to be killed while alerting on resident memory tells you nothing about which objects are responsible. three nested numbers, only one of which the container cares about resident set size — process_resident_memory_bytes interpreter · extension modules · thread stacks · everything below the allocator's arenas requested from the OS, released reluctantly, fragmented live Python objects what gc and tracemalloc measure the layer your code creates and frees what follows freeing many objects often changes RSS by nothing at all alert on RSS — it is what kills you diagnose with objects — RSS cannot name a culprit two tools, two jobs
Alert on the outer number and diagnose with the inner one. Swapping them produces either an alert that never fires or an investigation with no suspects.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0"

export GC_METRICS_ENABLED=1
export TRACEMALLOC_FRAMES=10      # only when diagnosing

Implementation

Step 1 — Export the process metrics. prometheus_client registers the process collector automatically on Linux; the figure to alert on is resident memory.

process_resident_memory_bytes 4.02653184e+08
process_virtual_memory_bytes  1.2884902e+09
process_open_fds 143

Resident set size is what the cgroup limit compares against, so it is the alerting signal. Virtual memory is nearly meaningless for this purpose — a process can map far more address space than it will ever touch.

Step 2 — Time the collections. The default collector counts them; counts do not show pauses.

import gc
import time
from prometheus_client import Counter, Histogram

GC_PAUSE = Histogram(
    "python_gc_pause_seconds", "Garbage collection pause duration", ["generation"],
    buckets=(0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5),
)
GC_COLLECTED = Counter("python_gc_collected_objects_total", "Objects freed", ["generation"])
GC_UNCOLLECTABLE = Counter("python_gc_uncollectable_total", "Objects not freed", ["generation"])

_started: dict[int, float] = {}

def _on_gc(phase: str, info: dict) -> None:
    gen = str(info["generation"])
    if phase == "start":
        _started[info["generation"]] = time.perf_counter()
        return
    began = _started.pop(info["generation"], None)
    if began is not None:
        GC_PAUSE.labels(gen).observe(time.perf_counter() - began)
    if info.get("collected"):
        GC_COLLECTED.labels(gen).inc(info["collected"])
    if info.get("uncollectable"):
        GC_UNCOLLECTABLE.labels(gen).inc(info["uncollectable"])

gc.callbacks.append(_on_gc)

Keep the callback to timestamps and counters. It runs during collection, on whichever thread triggered it, and anything that allocates there is asking for a bad time.

Step 3 — Watch generation 2 separately. Generations 0 and 1 are frequent and short; generation 2 walks everything that has survived, and its pause scales with the size of the live object graph — which is exactly what grows when there is a leak.

# the pause that shows up as latency
histogram_quantile(0.99, sum(rate(python_gc_pause_seconds_bucket{generation="2"}[5m])) by (le))
Three generations, three very different costs The three garbage collection generations compared. Generation zero runs constantly — hundreds or thousands of times a minute in a busy service — and scans only the objects allocated since the last pass, so its pause is measured in microseconds and is effectively invisible. Generation one runs roughly once per ten generation-zero passes and scans the objects that survived those, with a pause typically under a millisecond. Generation two runs roughly once per ten generation-one passes and scans every tracked object that has survived this far, which in a large service can be millions; its pause scales with the size of the live object graph and is measured in tens or hundreds of milliseconds, which is squarely inside the latency budget of a web request. The conclusion drawn is that only generation two is worth alerting on, and that its pause duration is a proxy for the size of the surviving object graph — which makes it a leak indicator as well as a latency one. how often, over what, for how long gen 0 constantly objects allocated since the last pass microseconds — invisible gen 1 ~1 per 10 gen-0 what survived generation 0 under a millisecond gen 2 ~1 per 10 gen-1 every tracked object that has survived tens to hundreds of ms why generation 2 earns its own alert its pause scales with the size of the surviving graph — so it is a latency signal and a leak indicator at the same time
Generation 2 is the only one that reaches a web request's latency budget, and its duration doubles as a measure of how much has survived.

Step 4 — Freeze the startup graph before forking. gc.freeze() moves everything currently tracked into a permanent generation the collector never rescans.

# gunicorn.conf.py
import gc

def when_ready(server):
    gc.collect()          # tidy up first
    gc.freeze()           # imports and startup objects are never rescanned again

def post_fork(server, worker):
    gc.enable()

Two benefits: generation-2 pauses stop re-walking the interpreter's startup graph, and the pages holding those objects are not touched by the collector, so they stay copy-on-write shared between prefork workers instead of being duplicated.

Step 5 — Use tracemalloc only to diagnose. It roughly doubles allocation cost, so bound it in time and scope.

import tracemalloc
from fastapi import APIRouter

router = APIRouter(prefix="/admin/memory")
_baseline = None

@router.post("/start")
async def start():
    global _baseline
    tracemalloc.start(10)
    _baseline = tracemalloc.take_snapshot()
    return {"tracing": True}

@router.get("/top")
async def top():
    snapshot = tracemalloc.take_snapshot()
    stats = snapshot.compare_to(_baseline, "lineno")[:10]
    return {"top": [str(s) for s in stats]}

@router.post("/stop")
async def stop():
    tracemalloc.stop()
    return {"tracing": False}
Growth plateaus; a leak keeps its peak Two week-long traces of resident memory, one for a healthy service and one with a leak. The healthy trace rises during each working day as caches, connection pools and buffers fill, then falls back overnight when traffic drops, and the daily minimum is the same on Friday as it was on Monday: the growth is real, bounded, and returns. The leaking trace has the same daily shape superimposed on a rising floor — each night it gives back less than it took, so the daily minimum climbs steadily through the week and eventually meets the container limit, at which point the process is killed and the trace restarts from the bottom. The diagnostic drawn out is that the daily peak looks similar in both traces and the daily minimum does not, so the query worth alerting on is the minimum over twenty-four hours rather than the current value or the maximum. resident memory over a week healthy the daily minimum is flat — it gives back what it took a leak container limit the minimum climbs all week — then the process is killed and it starts again
The daily peaks look alike. The daily minimum is where the two traces differ, which makes min_over_time the query worth alerting on.

Configuration options

Option Type Default Recommended
process_resident_memory_bytes gauge auto the alerting signal
python_gc_pause_seconds histogram absent add via gc.callbacks
Buckets tuple — 0.0005 s to 0.5 s
generation label str — three values only
gc.freeze() — not called after imports, before fork
gc.set_threshold tuple (700, 10, 10) raise gen-2 only with evidence
tracemalloc — off on, briefly, one replica

Verification

curl -s localhost:9000/metrics | grep -E 'process_resident|python_gc'

Expected Output:

process_resident_memory_bytes 4.02653184e+08
python_gc_collections_total{generation="0"} 41822.0
python_gc_collections_total{generation="2"} 61.0
python_gc_pause_seconds_bucket{generation="2",le="0.05"} 58.0
python_gc_pause_seconds_bucket{generation="2",le="0.25"} 61.0
python_gc_pause_seconds_sum{generation="2"} 3.94
python_gc_uncollectable_total{generation="2"} 0.0

Three readings to check. Generation-2 collections are rare — 61 against 41 822 generation-0 passes is normal. Three of those exceeded 50 ms, which is worth knowing and is invisible in any request metric. And uncollectable is zero; a rising value there means a reference cycle involving objects the collector will not free, which is a genuine leak rather than ordinary growth.

Then confirm the leak-versus-growth shape over a longer window:

# does memory fall during the quiet period?
min_over_time(process_resident_memory_bytes[24h])

A service without a leak gives some memory back overnight. One with a leak keeps its peak, and the minimum rises day over day.

Common mistakes

Alerting on Python object counts

Error signature: the alert never fires and the container is killed anyway. Root cause: object counts do not track resident memory, because the allocator does not return freed arenas promptly. Remediation: alert on process_resident_memory_bytes against the container limit, and use object counts for diagnosis only.

Disabling the collector to remove pauses

Error signature: pauses disappear and memory rises without bound. Root cause: cycle collection is what frees reference cycles, and ORM and framework object graphs are full of them. Remediation: raise the generation-2 threshold if pauses are genuinely the problem, and call gc.freeze() after startup — both keep collection working.

tracemalloc left enabled

Error signature: allocation-heavy endpoints slow by tens of percent, and memory rises by the tracing overhead itself. Root cause: a diagnostic session that was started and never stopped. Remediation: expose start and stop as separate operations, and add a timer that stops tracing automatically.

Finding what is actually leaking

Metrics tell you memory is climbing; they do not tell you what is holding it. Three techniques, in the order they are worth trying, because each is cheaper than the next.

Object counts by type. gc.get_objects() walked once and counted by type name gives a distribution, and comparing two snapshots taken twenty minutes apart shows which type grew. It is coarse — a leak of dictionaries tells you very little — and it is often enough, because the growing type is frequently domain-specific and names the subsystem immediately.

import gc
from collections import Counter

def type_histogram(top: int = 15) -> list[tuple[str, int]]:
    counts = Counter(type(obj).__name__ for obj in gc.get_objects())
    return counts.most_common(top)

Run it behind an admin endpoint, not on a timer: walking every object is expensive and pauses the process for the duration.

Allocation traces. tracemalloc attributes memory to the source line that allocated it, which is the answer when the type histogram says "dict" and you need to know which dict. Take a baseline, wait, take a second snapshot, and compare — the top entries by growth are the allocation sites responsible.

Referrer chains. When the allocation site is known and the reason the objects survive is not, gc.get_referrers() on a sample of the leaking objects shows what is holding them. This is the most laborious of the three and the one that finds the genuinely puzzling cases: a cache without an eviction policy, a list that accumulates for a report nobody generates, a closure capturing a request.

Technique Cost Answers
Type histogram a pause per run which type is growing
tracemalloc comparison ~2× allocation cost while on which line allocated it
Referrer chains manual, slow why it is still alive
Heap dump and offline analysis heavy, needs tooling everything, eventually

The leaks that are not bugs

Two very common patterns look exactly like leaks in the metrics and are working as designed, and recognising them saves a day of investigation.

Caches without bounds. An lru_cache with maxsize=None, a module-level dictionary used as a memo, a connection pool that grows to its high-water mark and stays there. Each of these grows to the size of its key space and then plateaus — which is fine when the key space is small, and indistinguishable from a leak when the key is something like a customer ID. The distinction is whether the plateau exists at all, which is why the daily-minimum reading matters more than the slope.

Fragmentation. A workload that allocates a large number of objects, frees them, and repeats can leave the allocator holding arenas it cannot return because a few live objects are scattered across them. Resident memory stays high while Python-level object counts are flat, which looks like a leak in the outer metric and is invisible in the inner one. There is no fix in application code beyond allocating differently; the practical response is to size the container for the high-water mark.

Choosing what to alert on

Given all of the above, the alerting policy that works is narrower than it first appears. Alert on resident memory approaching the container limit, because that is the failure. Alert on the twenty-four-hour minimum rising over several days, because that is the leak signature and it fires early enough to act on. Do not alert on generation-2 pause duration, object counts, or the slope of resident memory within a day — all three move for ordinary reasons and produce pages nobody can act on.

The GC pause metric earns its place on the dashboard rather than in the alert policy: it is what you look at when latency is spiking and you need to know whether collection explains it, which is a question asked during an incident rather than one that should start one.

Frequently Asked Questions

Why does resident memory not fall after objects are freed?

Because CPython's allocator holds freed arenas for reuse rather than returning them to the operating system immediately, and fragmentation can prevent a whole arena from ever becoming free. The consequence is that resident set size is a high-water mark more than a current-usage figure, which is fine for capacity purposes and misleading if you expect it to track object counts.

Should I disable the garbage collector?

Almost never in a long-running service. Disabling gc removes cycle collection, so any reference cycle — which includes many ordinary framework and ORM object graphs — leaks permanently. Tuning the thresholds so generation-2 runs less often is the useful version of that idea, and freezing long-lived startup objects with gc.freeze before forking is the version with the clearest payoff.

What does gc.freeze actually do?

It moves everything currently tracked into a permanent generation that collections never examine. Called after imports and before forking workers, it means the collector stops re-walking the interpreter's startup object graph on every generation-2 pass, which both shortens those pauses and avoids touching pages that would otherwise be copy-on-write shared between prefork workers.

How do I tell a leak from ordinary growth?

By the shape over a longer window than one deploy cycle. Ordinary growth rises and plateaus as caches and pools fill. A leak rises without plateauing, and the ratio of resident memory to request rate rises with it. The clearest confirmation is that memory does not fall during a quiet period — a service with a leak keeps its overnight peak, a service without one gives some back.

Is tracemalloc safe to run in production?

It works, and it roughly doubles allocation cost and adds memory of its own, so it is a diagnostic rather than a monitoring tool. The usual approach is an admin endpoint that starts it, takes a snapshot after a few minutes, compares against a baseline snapshot and stops it — bounded in time, on one replica.

Related Guides