Finding Memory Leaks with tracemalloc

Once a leak is established — memory rising and live object count rising with it — the next question is which line is responsible, and tracemalloc answers it directly by comparing two snapshots of allocation state. This page covers taking the snapshots correctly, reading the diff, and the step after the line is found, which is identifying what holds the reference. It is a task article under memory profiling and leak detection, part of the Python profiling and performance observability section.

Prerequisites

tracemalloc is in the standard library. A referrer inspector is useful for the step after the diff.

pip install "objgraph>=3.6.0,<4.0.0"

Implementation

Step 1 — Start tracing with a useful frame depth. The default depth of one records only the line that called the allocator, which in practice is inside a library and tells you nothing. A depth of twenty-five reaches application code from almost any library allocation, at a cost in both time and the memory tracemalloc itself uses to store the traces.

import tracemalloc
tracemalloc.start(25)

Step 2 — Take the baseline once the process is warm. A snapshot taken immediately after start is dominated by imports, module-level constants and framework initialisation, all of which grow once and never again. Letting the process serve a few hundred requests first means the difference describes steady-state behaviour rather than startup.

warm_up(requests=500)
baseline = tracemalloc.take_snapshot()

Step 3 — Run a bounded, repeatable workload. A known number of operations turns the result from "it grew by eighty megabytes" into "it retains seventeen kilobytes per request", which is the figure that identifies the problem and predicts when the process will hit its limit.

serve_requests(count=5_000)

Step 4 — Collect before the second snapshot. Without a collection, the difference includes objects that are unreferenced but not yet freed, which appear as growth and are not. Forcing a full collection first means everything remaining is genuinely retained by something.

import gc
gc.collect()
current = tracemalloc.take_snapshot()

Step 5 — Compare by line, then widen to traceback. Line grouping ranks allocating locations, which is the right first view. When the top entry is inside a library, switching to traceback grouping shows the chain that reached it, and the application frame in that chain is the call site to fix.

print("--- by line ---")
for stat in current.compare_to(baseline, "lineno")[:5]:
    print(f"{stat.size_diff/1024:9.1f} KiB  {stat.count_diff:+7d}  {stat}")

print("--- worst entry, full traceback ---")
top = current.compare_to(baseline, "traceback")[0]
for line in top.traceback.format():
    print("   ", line)

Expected Output: the line, and then the caller responsible for reaching it.

--- by line ---
  84210.4 KiB   +50021  app/cache.py:31: size=84.4 MiB (+84.2 MiB), count=50021
    412.9 KiB     +311  app/serializers.py:88: size=1.2 MiB (+412 KiB), count=902
     31.0 KiB      +12  logging/__init__.py:1620: size=88 KiB (+31 KiB), count=44
--- worst entry, full traceback ---
    File "app/views.py", line 142
      pricing = resolve_pricing(order, tenant)
    File "app/pricing.py", line 77
      return _memo.setdefault(key, compute(order, tenant))
    File "app/cache.py", line 31
      self._entries[key] = value

Ten entries retained per request, keyed on something request-specific, in a dictionary with no eviction. The diff named cache.py:31; the traceback named pricing.py:77, which is where the fix goes.

Step 6 — Express the result per operation. Dividing the growth by the workload size gives a number that both confirms the diagnosis and predicts behaviour: seventeen kilobytes per request against a five hundred megabyte headroom is roughly thirty thousand requests before the limit, which either matches the observed restart interval or means something else is also going on.

After the line: finding what holds the reference

The diff says where the memory was allocated. A leak is defined by what keeps it alive, and those are the same place only some of the time. When the traceback does not make the retainer obvious, one more step identifies it.

Take a sample of the leaked objects and walk backwards through whatever references them. In almost every real case the answer appears within two or three hops and is one of the same three structures: a dictionary at module scope, a list being appended to, or a closure held by something long-lived such as a signal handler, an event subscription or a cached partial function.

import gc, objgraph

objgraph.show_growth(limit=5)                 # which types grew
sample = objgraph.by_type("PricingResult")[-1]
objgraph.show_backrefs([sample], max_depth=4, filename="/tmp/refs.png")

What to look for in the resulting graph is any edge from a long-lived object into the sample. A request-scoped object referenced only by other request-scoped objects is not leaked; it is simply still in flight. The leak is the edge that crosses from something that lives for the process's lifetime into something that should have lived for one request.

Two patterns deserve specific mention because they are easy to miss. A closure captures its enclosing scope entirely, so a callback defined inside a request handler and registered on a long-lived object retains everything that handler had in scope — including the request body and any rows it fetched. And a default argument evaluated once at function definition time, if it is a mutable container, becomes a permanent accumulator that is invisible at the call site.

Doing this against production safely

The measurement above is easiest in a reproduction, and sometimes the leak only appears under production traffic. Three arrangements make a production measurement safe enough to do.

One instance, out of rotation or with reduced weight. Tracing every allocation costs real throughput, and the cost is paid by whichever requests that instance serves. Routing a small share of traffic to one instance with tracing enabled bounds the impact and still exercises the real code paths with real data shapes — which is often why the leak does not reproduce locally.

A trigger rather than a restart. Starting and stopping tracing through a protected internal endpoint, or a signal handler, means the instance does not need restarting to begin measuring, and the baseline can be taken after it is warm. A restart would reset exactly the state that makes the leak visible.

Write the snapshots to disk and analyse elsewhere. Snapshots can be dumped to a file and loaded in a separate process for comparison, so the analysis — which is itself memory-intensive — does not happen inside the service being measured. This also means the snapshots survive if the instance is recycled before anyone looks at them.

import signal, tracemalloc

def _toggle(signum, frame):
    if tracemalloc.is_tracing():
        tracemalloc.take_snapshot().dump(f"/tmp/snap-{int(time.time())}.pickle")
        tracemalloc.stop()
    else:
        tracemalloc.start(25)

signal.signal(signal.SIGUSR2, _toggle)

Configuration options

Verification

Confirm the fix by repeating the measurement, which is the only evidence that the change removed the leak rather than moving it.

import gc, tracemalloc

def retained_per_operation(work, iterations: int = 2000) -> float:
    gc.collect()
    tracemalloc.start(15)
    before = tracemalloc.take_snapshot()
    for _ in range(iterations):
        work()
    gc.collect()
    after = tracemalloc.take_snapshot()
    total = sum(s.size_diff for s in after.compare_to(before, "lineno"))
    tracemalloc.stop()
    return total / iterations

print(f"{retained_per_operation(handle_request):.0f} bytes retained per request")

Expected Output: a figure near zero after the fix, against a substantial one before it.

before   17_240 bytes retained per request
after        18 bytes retained per request

Eighteen bytes per request is ordinary noise from caches that legitimately grow and settle. A figure that is smaller but still clearly positive usually means a second, smaller leak that the first one was masking.

Common mistakes

A frame depth of one. Error signature: a report naming a line inside a standard library container. Root cause: the default depth records only the immediate allocator call. Remediation: start with enough frames to reach application code.

No collection before the second snapshot. Error signature: apparent growth that disappears on a second run. Root cause: unreferenced objects not yet freed counted as retained. Remediation: force a collection first.

Baseline taken at startup. Error signature: a diff dominated by imports and framework initialisation. Root cause: the snapshot captured one-time allocation. Remediation: warm the process before the baseline.

Grouping by type instead of by line. Error signature: a conclusion that the leak is dictionaries. Root cause: every Python program's memory is mostly dictionaries. Remediation: group by line, and widen to traceback when the line is generic.

Measuring a leak that only exists in production, locally. Error signature: a clean reproduction and a service that still grows. Root cause: the leak depends on data shapes, tenants or code paths that local traffic never exercises. Remediation: measure on one production instance with a trigger, as described above.

Leaving it enabled. Error signature: a service that stayed slow after the investigation ended. Root cause: tracing left on, recording a stack per allocation. Remediation: stop it explicitly, and prefer a bounded window from the start.

Frequently Asked Questions