Memory Profiling and Leak Detection in Python

"The service is using too much memory" describes at least three unrelated failures, and the tool that diagnoses one is useless against the others. This guide covers telling them apart first — which takes minutes — and then the technique appropriate to each. It is part of the Python profiling and performance observability section and pairs with monitoring Python garbage collection and memory usage, which covers the metrics that make these distinctions visible continuously. The focused articles in this topic are Diagnosing RSS Growth in Python Containers, Finding Memory Leaks with tracemalloc and Profiling Memory with memray.

Prerequisites

pip install "memray>=1.13.0,<2.0.0" \
            "objgraph>=3.6.0,<4.0.0"

tracemalloc and gc are in the standard library and cover most of what is needed.

Concept and architecture

Python's memory behaviour has three layers, and the diagnosis depends on knowing which one is growing.

The object layer. Live Python objects, reachable from somewhere. This is what a leak grows. The reference counter frees objects as soon as their count reaches zero, and the cycle collector handles the rest; an object that stays alive does so because something still points at it. Finding a leak is therefore finding the reference, not finding the object.

The allocator layer. Python's own allocator requests memory from the operating system in arenas and sub-allocates small objects from pools inside them. A freed object's memory returns to the pool, not to the system. An arena is returned only when it is completely empty, and a single surviving object keeps a whole arena resident. This is why a process that briefly used two gigabytes tends to stay near two gigabytes, and it is the source of most "leak" reports that are not leaks.

The system layer. Resident set size, which is what a container limit is measured against. It includes the allocator's arenas, the interpreter itself, loaded extension modules, thread stacks and any memory a C extension allocated with its own allocator — which Python's tools cannot see at all.

The practical consequence is a diagnostic order. Compare live object counts with resident memory first: if objects are flat and memory has risen, it is the allocator layer and no amount of object graph analysis will help. If objects are rising, it is a leak and the allocation snapshot will name the line. If neither is rising and there is still a problem, it is churn, and the measurement is collection frequency.

Step-by-step implementation

Step 1 — Plot the daily minimum over a fortnight. This is the cheapest and most decisive measurement available, and it requires no profiler. Ordinary variation produces peaks that differ and troughs that agree; a leak raises the trough. Two weeks of a minimum-over-a-day series answers "is this a leak" definitively, and it can be computed from metrics that already exist.

# the floor, which is what a leak moves
min_over_time(process_resident_memory_bytes{service="checkout"}[1d])

Step 2 — Compare object counts against resident memory. The interpreter can report how many objects it is tracking. Rising memory with a flat object count is retention; both rising together is a leak.

import gc, os, resource

def snapshot() -> dict:
    return {
        "tracked_objects": len(gc.get_objects()),
        "rss_mb": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
        "gc_counts": gc.get_count(),
    }

Expected Output: the two numbers, whose relationship is the diagnosis.

start   tracked_objects=412_083  rss_mb=381
+6h     tracked_objects=414_201  rss_mb=902     <- retention: objects flat, memory up
+6h     tracked_objects=981_446  rss_mb=902     <- leak: objects up with memory

Step 3 — Diff two allocation snapshots. Once a leak is established, tracemalloc names the line that allocated the growth. Grouping by line rather than by object type is what makes the result actionable, because the type is usually dict and the line is usually obvious once you see it.

import gc
import tracemalloc

tracemalloc.start(25)                      # 25 frames, so the caller is identifiable
baseline = tracemalloc.take_snapshot()

serve_requests(count=5_000)                # the workload under suspicion

gc.collect()                               # drop anything merely unreferenced
current = tracemalloc.take_snapshot()

for stat in current.compare_to(baseline, "lineno")[:5]:
    print(f"{stat.size_diff/1024:9.1f} KiB  {stat.count_diff:+7d} blocks  {stat}")

Expected Output: one line accounting for nearly all of it, which is the ordinary shape of a real leak.

  84210.4 KiB   +50021 blocks  app/cache.py:31: size=84.4 MiB (+84.2 MiB), count=50021
    412.9 KiB     +311 blocks  app/serializers.py:88: size=1.2 MiB (+412 KiB), count=902
     31.0 KiB      +12 blocks  logging/__init__.py:1620: size=88 KiB (+31 KiB), count=44

Step 4 — Find what holds the reference. The allocating line tells you where the object was created; the leak is whatever kept it. Walking the referrers of a sample object usually identifies it in one step, and the answer is nearly always a container that is appended to and never pruned.

import gc, objgraph

objgraph.show_growth(limit=5)              # which types grew since last call

leaked = objgraph.by_type("OrderSnapshot")[0]
objgraph.show_backrefs([leaked], max_depth=4, filename="/tmp/refs.png")

Step 5 — Distinguish churn from growth. If neither memory nor objects are rising but something is wrong, measure allocation rate. A collection count climbing with traffic, with flat memory, means the process is allocating and freeing at a rate that costs latency through pauses.

import gc
gc.callbacks.append(lambda phase, info: COLLECTIONS.labels(gen=info["generation"]).inc()
                    if phase == "stop" else None)

Step 6 — Decide whether the fix is in the code or in the deployment. A genuine leak is a code fix. Allocator retention is not fixable in Python code, and the honest remedies are a memory limit that accommodates the high-water mark, a periodic worker recycle, or reducing the peak allocation that established the mark in the first place.

Configuration reference

Async and concurrency considerations

Memory analysis in a concurrent service has two complications worth anticipating.

The first is that per-request memory is not per-request in any isolated sense. Under threads or asyncio, many requests are in flight simultaneously, so a snapshot taken at an arbitrary moment contains the working sets of all of them. Attributing retained memory to a request therefore requires the snapshot to bracket a period of known workload — which is why the snapshot diff in step 3 is framed around a specific number of requests rather than around a duration.

The second is that a leak in an asyncio service frequently involves a task that was never awaited or a callback registered on a long-lived object. A task holds its entire frame chain alive, so one leaked task can retain a surprising amount — the request's parsed body, its database rows, its partial response. Checking the number of live tasks alongside memory is a cheap addition that identifies this class immediately.

Under a prefork server, memory analysis must be per worker. The master's memory is not interesting, workers differ, and a leak in one worker is diluted to invisibility in a fleet-wide average. Recording resident memory per worker process, with the worker's age, makes both the leak and the recycling behaviour visible.

There is also a pleasant property worth exploiting: forked workers share the parent's memory copy-on-write, so anything loaded before the fork costs once rather than once per worker. Loading a large model, a lookup table or a compiled schema in the master rather than in each worker can reduce a fleet's memory substantially, and getting it wrong — loading after the fork — multiplies it.

Production code examples

A periodic self-check that distinguishes the three cases without a profiler, cheap enough to leave running:

# memwatch.py — a background check that names which problem you have.
import gc
import logging
import os
import threading
import time

log = logging.getLogger("memwatch")


def _rss_bytes() -> int:
    with open("/proc/self/statm") as fh:
        return int(fh.read().split()[1]) * os.sysconf("SC_PAGE_SIZE")


def watch(interval_s: int = 300) -> None:
    prev_rss, prev_objects = _rss_bytes(), len(gc.get_objects())
    prev_collections = sum(s["collections"] for s in gc.get_stats())

    while True:
        time.sleep(interval_s)
        rss, objects = _rss_bytes(), len(gc.get_objects())
        collections = sum(s["collections"] for s in gc.get_stats())

        rss_delta = (rss - prev_rss) / 1_048_576
        obj_delta = objects - prev_objects
        gc_rate = (collections - prev_collections) / interval_s

        # 1. The classification, stated rather than left to interpretation.
        if rss_delta > 20 and obj_delta > 10_000:
            verdict = "leak: objects and memory both growing"
        elif rss_delta > 20:
            verdict = "retention: memory grew, object count did not"
        elif gc_rate > 5:
            verdict = "churn: allocation rate high, memory stable"
        else:
            verdict = "stable"

        log.info("memory check", extra={
            "rss_mb": round(rss / 1_048_576, 1),
            "rss_delta_mb": round(rss_delta, 1),
            "objects": objects,
            "object_delta": obj_delta,
            "gc_per_s": round(gc_rate, 2),
            "verdict": verdict,
        })
        prev_rss, prev_objects, prev_collections = rss, objects, collections


threading.Thread(target=watch, daemon=True, name="memwatch").start()

Expected Output: a classification rather than a number, logged every five minutes.

{"message": "memory check", "rss_mb": 902.4, "rss_delta_mb": 41.2, "objects": 981446, "object_delta": 54021, "gc_per_s": 1.8, "verdict": "leak: objects and memory both growing"}

A reproduction harness that isolates the leaking workload, which is the step that turns a production symptom into something fixable:

# isolate.py — narrow the workload until the growth stops
import gc, tracemalloc

def measure(work, iterations: int = 2000) -> float:
    gc.collect()
    tracemalloc.start(15)
    before = tracemalloc.take_snapshot()
    for _ in range(iterations):
        work()
    gc.collect()
    after = tracemalloc.take_snapshot()
    grew = sum(s.size_diff for s in after.compare_to(before, "lineno"))
    tracemalloc.stop()
    return grew / 1_048_576

for name, work in CANDIDATES.items():
    print(f"{name:28s} {measure(work):8.2f} MiB retained per 2000 calls")

Expected Output: one candidate retaining, the rest flat, which localises the leak to a code path.

serialize_order                 0.01 MiB retained per 2000 calls
resolve_pricing                18.44 MiB retained per 2000 calls
render_receipt                  0.00 MiB retained per 2000 calls

Living with a memory ceiling

Not every memory problem gets fixed, and some should not be. A service whose working set is genuinely large, or whose peak is set by a legitimate workload, needs an operational answer rather than a code change, and there are three of them worth knowing.

Size the limit for the high-water mark, not the average. A container limit set from average usage will kill the process during the first legitimate peak, and the kill happens without warning and without a traceback. Setting the limit above the observed maximum over a representative period — and alerting well below it — converts an abrupt failure into a signal. This is the single most common cause of mysterious restarts in Python services, and the evidence is a terminated container with no application-level error at all.

Recycle workers on a bounded schedule. A prefork server can restart each worker after a number of requests, which returns its memory to the operating system wholesale. This is a blunt instrument and an effective one: it caps the effect of both a slow leak and allocator retention, at the cost of the occasional cold worker. Setting it by request count with jitter, rather than by time, spreads the restarts and avoids every worker recycling simultaneously.

Reduce the peak rather than the steady state. Because retention follows the high-water mark, the leverage is in whatever established that mark — a report that loads a million rows, a bulk import that builds a list before writing it, a serialisation that materialises an entire response. Streaming those operations rather than materialising them frequently halves a service's resident memory without changing anything about its ordinary behaviour, and it is more durable than any tuning.

The judgement to make is whether the memory is doing useful work. A service holding a large cache that materially reduces database load is using memory correctly, and the right response is a larger limit. A service holding a large cache nobody measured the benefit of is a different situation, and measuring it is cheaper than either fixing or accommodating it.

What to record continuously

The measurements above are diagnostic and mostly run on demand. Four that are cheap enough to leave on turn a future investigation from an hour into a minute.

Resident memory per process, with the process's age. The age is what makes a leak legible: plotting memory against uptime across many workers gives the growth curve directly, without waiting a fortnight, because the fleet contains workers of many ages at any moment.

Live object count, sampled infrequently. Counting tracked objects pauses the process briefly and in proportion to heap size, so once every few minutes is appropriate rather than every scrape. It is the single number that separates a leak from retention, and having it already recorded removes the first hour of every investigation.

Garbage collection counts and durations per generation. These are nearly free and answer the churn question. A generation-two collection count that climbs with traffic, or a collection duration that grows over a process's life, both indicate problems that no memory graph shows.

Peak memory per request, for the heaviest endpoints. Harder to obtain and worth it for the handful of endpoints that build large intermediate structures. It is the number that identifies which endpoint established the high-water mark, which is where the streaming change belongs.

Reference cycles, and why they are rarely the answer

Reference cycles have a reputation as the archetypal Python leak, and in modern code they very seldom are. The cycle collector handles them: objects in an unreachable cycle are collected, including most objects with finalisers. Assuming a cycle at the start of an investigation is usually a detour.

Where cycles do still matter is narrower than the reputation suggests. An object with a reference to a C extension's state that the collector cannot traverse can keep a cycle alive. A cycle involving an object whose finaliser resurrects it defers collection indefinitely. And generation-three collections are expensive, so a large number of long-lived cyclic structures costs pause time even when they are eventually collected.

The more productive framing is that Python leaks are almost always intentional references that outlived their purpose. The three that account for the large majority are worth checking before anything else: a module-level dictionary used as a cache with no eviction, a list appended to by a handler and never trimmed, and a callback or closure registered on a long-lived object that captures a request's data. Each is visible in a referrer graph in seconds, and each is a two-line fix once found.

Common mistakes

Treating retention as a leak. Error signature: days spent in an object graph for memory that plateaus on its own. Root cause: the allocator's arenas, not the application's references. Remediation: check the daily minimum and the object count before opening a profiler.

Grouping allocations by type. Error signature: a result saying most memory is in dictionaries. Root cause: every Python program's memory is in dictionaries. Remediation: group by line, which names the code rather than the shape.

Running tracemalloc continuously in production. Error signature: a service that slowed down when the investigation started. Root cause: a stack recorded for every allocation. Remediation: use it on one instance for a bounded window, or in a reproduction.

Ignoring native allocations. Error signature: resident memory far above what Python's tools account for. Root cause: a C extension allocating outside Python's allocator. Remediation: a native-aware profiler, or process-level accounting.

Averaging memory across workers. Error signature: a leak in one worker invisible in the fleet's graph. Root cause: per-process behaviour hidden by aggregation. Remediation: record per worker, with worker age, and look at the distribution.

Fixing churn as though it were growth. Error signature: a memory investigation that never finds anything while latency stays bad. Root cause: high allocation rate with stable memory. Remediation: measure collection frequency, and treat it as a throughput problem.

Frequently Asked Questions