Diagnosing RSS Growth in Python Containers

A container's memory graph climbs over a day, the process is killed at its limit, and a careful search for a leak finds nothing. This is one of the most common and most frustrating situations in running Python services, and the reason is that resident memory is made of several parts that grow for unrelated reasons — only one of which is a leak in the usual sense. This page covers taking the number apart. It is a task article under memory profiling and leak detection, part of the Python profiling and performance observability section.

Prerequisites

Most of the measurements here come from the kernel rather than from Python, so the tools are standard.

pip install "psutil>=5.9.0,<7.0.0" \
            "memray>=1.13.0,<2.0.0"

Implementation

Step 1 — Read the number the platform enforces. A container is killed against its cgroup memory accounting, which is not the same figure as the process's resident set size. It includes every process in the container and, importantly, page cache for files the container has touched. A service that writes a large log file to its own filesystem accumulates page cache that counts against the limit, and the kernel reclaims it only under pressure — sometimes not quickly enough to prevent a kill.

# inside the container: the figure the limit applies to, and its breakdown
cat /sys/fs/cgroup/memory.current
grep -E '^(anon|file|kernel|sock|shmem) ' /sys/fs/cgroup/memory.stat

Expected Output: anonymous memory is the process's own; file is page cache.

1873821696
anon 1204338688
file 612810752
kernel 41680896
sock 2101248
shmem 0

Six hundred megabytes of the container's figure is page cache here. It is reclaimable, and it is also counted.

Step 2 — Break the process's memory into its mappings. The process's own memory is a set of mappings: the heap, anonymous regions used by allocators, thread stacks, shared libraries. Summarising them by kind separates the Python heap from native allocator arenas from fixed per-thread costs.

import psutil

proc = psutil.Process()
by_kind = {"heap/anon": 0, "stacks": 0, "libs": 0, "other": 0}
for m in proc.memory_maps(grouped=False):
    path = m.path or ""
    if path.endswith(".so") or ".so." in path:
        by_kind["libs"] += m.rss
    elif "stack" in path:
        by_kind["stacks"] += m.rss
    elif path in ("", "[heap]", "[anon]"):
        by_kind["heap/anon"] += m.rss
    else:
        by_kind["other"] += m.rss
for kind, rss in by_kind.items():
    print(f"{kind:10s} {rss / 1_048_576:8.1f} MiB")

Step 3 — Compare the Python heap against the anonymous total. The Python objects a profiler can see are a subset of the anonymous memory. If the anonymous total is far above what Python-level accounting reports, the difference is allocator retention plus native allocations, and neither is a leak in the application's objects.

Step 4 — Check the shape over time. Resident memory that rises after each deploy and plateaus within hours is allocator retention reaching its high-water mark. Memory that rises steadily with a climbing daily floor is a leak. A sawtooth means worker recycling is already containing growth, and the amplitude of each tooth is the growth per worker lifetime.

# the daily floor, which only a leak moves
min_over_time(container_memory_working_set_bytes{container="app"}[1d])

Step 5 — Suspect the native allocator on many-threaded processes. The system allocator on glibc creates additional arenas to reduce contention between threads, and freed memory in those arenas is not always returned. A service with many threads and native-heavy work — database drivers, compression, image handling — can grow substantially from this alone. Limiting the arena count is a known and effective mitigation, and whether it helps is a quick experiment.

# fewer allocator arenas; test in staging and compare the growth curve
MALLOC_ARENA_MAX=2 gunicorn --workers 4 app:application

Step 6 — Contain while investigating. Recycling workers after a bounded number of requests caps growth from any cause and buys time. It is a containment rather than a fix, and it is worth keeping permanently if the investigation concludes the growth is retention rather than a leak.

# gunicorn.conf.py
max_requests = 5000
max_requests_jitter = 500      # spread recycles so workers do not all restart together

The four explanations, and how to confirm each

Working through the parts in order converts an open-ended investigation into four yes-or-no questions, and in practice one of them is almost always the answer.

Page cache. Confirmed when the container's file memory is large and grows with the service's file writes. Common with services that write logs or temporary files to the container filesystem. The remedy is to stop writing large files locally — send logs to standard output, as in standard output versus file logging in containers — or to accept it and size the limit to include it, since it is reclaimable.

Allocator retention. Confirmed when anonymous memory has risen, the Python object count has not, and the shape is a plateau after each deploy. The remedy is a limit sized for the high-water mark, or a reduction in the peak that set it, since no code change releases arenas that still contain one live object.

Native growth. Confirmed when anonymous memory greatly exceeds Python-level accounting and a native-aware profile attributes the difference to a C extension, or when limiting the allocator's arena count flattens the growth. The remedy is either the arena limit, a different allocator, or a change in how the extension is used — frequently, not holding large native result sets.

A genuine leak. Confirmed when Python object counts rise with memory and the daily floor climbs. This is the case the tools in finding memory leaks with tracemalloc are built for, and it is a code fix.

The order matters because the first three are cheap to confirm from outside the process and the fourth requires instrumenting it. Checking them first means the expensive investigation only happens when the cheap ones have been ruled out — and in a large share of cases, one of them is the answer and the leak hunt never needs to start.

Reporting the finding

Memory investigations produce a conclusion that is frequently counter-intuitive — "there is no leak, the limit is wrong" — and the conclusion needs to survive being explained to somebody who expected a bug fix. Three pieces of evidence make it stick.

The first is the shape over a fortnight, with the daily minimum overlaid. A floor that is flat across two weeks of deploys is the clearest possible statement that nothing is accumulating, and it is readable by anyone.

The second is the breakdown from step 1 and step 2, showing how much of the container's figure each part accounts for. A reader who sees that a third of the enforced memory is reclaimable page cache understands immediately why a Python-level leak search found nothing.

The third is the experiment: the remedy applied, and the curve changing shape as predicted. An explanation that has been tested is far more persuasive than one that has merely been reasoned out, and it is what separates a diagnosis from a plausible story.

A finding presented this way also answers the follow-up question before it is asked — what should the limit be — because the plateau level under the heaviest legitimate workload is visible on the same chart, and the limit belongs comfortably above it.

Configuration options

Verification

The check is that the explanation predicts the behaviour. Apply the remedy for the part you identified and confirm the curve changes shape.

# before and after, the same load test, the container figure at the end
kubectl exec deploy/checkout -- cat /sys/fs/cgroup/memory.current

Expected Output: for native fragmentation, limiting the arena count flattens the ramp into a plateau.

before  MALLOC_ARENA_MAX unset   after 6h: 2.41 GiB and rising
after   MALLOC_ARENA_MAX=2       after 6h: 0.97 GiB, flat since hour 2

A remedy that does not change the shape means the explanation was wrong, which is useful information in itself: it eliminates one of the four and narrows the search.

Common mistakes

Hunting for a leak first. Error signature: a day in an object graph for growth that turns out to be page cache or retention. Root cause: starting with the most expensive investigation. Remediation: check the cheap explanations from outside the process first.

Reading the process's memory rather than the container's. Error signature: a process that looks fine in a container that is killed. Root cause: page cache and other processes count against the limit. Remediation: read the cgroup figures.

Treating a sawtooth as healthy. Error signature: a leak discovered when recycling is turned off or the request rate drops. Root cause: recycling hiding the growth. Remediation: track tooth height as a metric even when nothing is failing.

A limit set from average usage. Error signature: kills during legitimate peaks. Root cause: retention settles at the high-water mark, not the average. Remediation: size the limit above the observed peak and alert well below it.

Recycling all workers at once. Error signature: periodic latency spikes as every worker restarts together. Root cause: no jitter on the request limit. Remediation: add jitter so recycles spread across time.

Frequently Asked Questions