Python Observability

Profiling Memory with memray

tracemalloc compares Python-level allocation state at two points. memray intercepts the allocator itself and records every allocation and every free with its full stack — including allocations made inside C extensions that Python's own tools cannot see. That makes it the tool for two questions tracemalloc handles poorly: what was holding memory at the peak, and where native libraries are spending it. This page covers running it, the reports worth reading, and where its overhead confines it. It is a task article under memory profiling and leak detection, part of the Python profiling and performance observability section.

The peak and the remainder are different questions A memory usage curve over the course of a workload rises to a sharp peak two thirds of the way through, when a report endpoint materialises a large result set, and then falls back to a level somewhat above where it started. Two regions are marked. The peak is the moment the container limit is measured against; the peak report explains it by showing which call paths held memory at that instant, which here is the report endpoint building a list of rows. The remainder is what is still allocated when the run ends; the leaks report explains it by showing only allocations that were never freed, which here is a small cache in an unrelated module. The note observes that the two reports point at different code, and that fixing the leak does nothing about the container being killed at the peak. one run, two different memory questions MB peak — what the container limit sees peak report: the report endpoint's row list remainder — leaks report: a cache elsewhere fixing the remainder does nothing about the peak — and the peak is what gets the container killed
Two reports, two answers, frequently in different modules. Knowing which question is being asked decides which one to open.

Prerequisites

pip install "memray>=1.13.0,<2.0.0"

memray runs on Linux and macOS. Attaching to a running process additionally needs the debugger permissions that any process attachment requires.

Implementation

Step 1 — Run the workload under the tracker. The simplest form wraps a script and writes a binary capture. For a service, running a reproduction script that exercises the suspect endpoints is usually better than tracking the whole server, because the capture is smaller and the reports are easier to read.

python -m memray run --output /tmp/report-endpoint.bin reproduce_report.py

Step 2 — Turn on native tracking when a library might be responsible. Without it, an allocation made by a C extension is attributed to whatever Python frame called into that extension, if it is seen at all. With it, the native frames appear and the allocation is attributed precisely. The cost is a slower run and a larger capture, which is worth paying whenever resident memory exceeds what Python-level tools account for.

python -m memray run --native --output /tmp/report-native.bin reproduce_report.py

Step 3 — Read the peak as a flame graph. The flame graph shows memory held at the moment of highest usage, laid out by call stack. Width here is bytes rather than time, but every other reading rule from reading flame graphs for Python services applies: scan the widest bars from the bottom, and find the widest one whose code you control.

python -m memray flamegraph /tmp/report-endpoint.bin --output /tmp/peak.html
python -m memray summary /tmp/report-endpoint.bin

Expected Output: the summary ranks locations by memory held at the peak.

┃ Location                                   ┃ Total Memory ┃ Total Memory % ┃ Allocations ┃
│ fetch_rows at app/reports.py:61            │    1.412GiB  │     78.31%     │     2104188 │
│ _parse at dateutil/parser.py:401           │   96.210MiB  │      5.21%     │      184000 │
│ to_representation at serializers.py:104    │   42.004MiB  │      2.28%     │      500000 │

A report endpoint holding one and a half gigabytes at the peak, in a list of rows, is the high-water mark — and it is also what established the level that the allocator will retain afterwards.

Step 4 — Use the leaks report for steady growth. Restricting the flame graph to allocations never freed by the end of the run isolates retained memory from temporary spikes. This is the equivalent of the tracemalloc diff, with the advantage of covering native allocations too.

python -m memray flamegraph --leaks /tmp/report-endpoint.bin --output /tmp/leaks.html

Step 5 — Read the allocation count as well as the size. A path making two million small allocations may not dominate the peak, but it dominates allocator time and drives collection frequency. The count column in the summary identifies churn, which appears as latency rather than memory and is covered in memory profiling and leak detection.

Step 6 — Attach to a live process for a bounded window when reproduction fails. Some problems only appear with production data. Attaching records from that point onward, and detaching ends the overhead; the window should be short and the process should be one that can tolerate slowing down.

python -m memray attach --duration 60 --output /tmp/live.bin <pid>
The memory Python's own tools cannot see A process's resident memory is shown as a single bar, divided into what Python-level tools can attribute and what they cannot. Python-level accounting covers Python objects allocated through Python's own allocator, which here is around four hundred megabytes. The remainder of the resident set, around six hundred megabytes, consists of allocations made directly with the system allocator by C extensions: a database driver's result buffers, a compression library's working memory and an image decoding library's pixel data. Python-level tools report the first figure and leave the rest unexplained. With native tracking enabled, memray attributes the second portion to the Python calls that triggered each allocation, so the full resident set is accounted for. The note records that this is the common explanation when resident memory is far above what a Python-level profiler reports. one process, 1 GB resident Python objects — ~400 MB allocated directly by C extensions — ~600 MB what each tool can attribute tracemalloc, gc — sees this invisible — the unexplained remainder memray with native tracking — sees all of it, attributed to the calling Python code driver buffers · compression windows · decoded images — the usual contents of the gap
When resident memory is far above what Python-level profiling reports, the gap is almost always native allocation, and only a native-aware tool can attribute it.

When to reach for it, and when not to

memray is the most informative memory tool in the Python ecosystem and also the most expensive to run, which makes the choice of when to use it worth thinking about rather than defaulting to.

Reach for it when the peak is the problem. A container killed for memory is killed at the high-water mark, and understanding what held memory at that instant requires a tool that records the peak rather than differences between two snapshots. Neither tracemalloc nor object counting answers this directly; the peak flame graph does.

Reach for it when Python's accounting does not add up. A process whose resident memory is several times what Python-level tools report is almost certainly allocating natively, and native tracking is the only way to attribute that memory without reading C source.

Reach for it when allocation rate matters. The allocation count per location identifies churn directly, which is otherwise inferred indirectly from collection metrics.

Do not reach for it first for a simple leak. A steady growth in Python objects is answered faster and more cheaply by a tracemalloc diff, as in finding memory leaks with tracemalloc, and the result is usually a single line.

Do not run it against a live service at normal load. The overhead of recording every allocation with its stack is substantial, and the capture file grows quickly. A reproduction that exercises the suspect path, or one drained instance for a minute, gives the same answer at a fraction of the risk.

The practical ordering for an unknown memory problem is therefore: metrics to classify it, tracemalloc for a Python-level leak, and memray when the problem is the peak, native memory, or churn — or when the cheaper tools have not explained it.

Reading a memory flame graph differently from a CPU one

The format is the same and one difference in meaning changes how it should be read. In a CPU flame graph, width is time accumulated across the whole window, so a wide frame was busy for a long time. In a peak memory flame graph, width is bytes held at a single instant, so a wide frame was holding a lot at that moment — regardless of whether it allocated slowly over the whole run or in one burst just before the peak.

That makes the question "what was alive at the worst moment" rather than "what was expensive overall", and it changes what counts as a fix. A frame that holds a large structure for a brief period is a peak problem even if it is cheap in every other sense, and the remedy is to not hold it all at once: stream the rows, process the file in chunks, build the response incrementally. A frame that allocates heavily but frees promptly may not appear in the peak view at all, even though it dominates allocator time.

A second difference is that temporal ordering matters more. Memory is a stock rather than a flow, so the sequence of allocation and release decides the peak. Two operations that each hold a gigabyte do not produce a two gigabyte peak unless they overlap. Reordering them — releasing one result before building the next — can halve a peak without changing any individual operation, and the temporal view that memray also provides is how such overlaps are found.

Which memray report for which question A table of memray's report types and the question each answers. The flame graph report shows where memory held at the peak was allocated, answering what made the peak so high. The flame graph with the leaks option shows allocations never freed by the end of the run, answering what leaked. The table report lists allocations sortable by size, answering which individual allocations were largest. The stats report summarises totals and allocation counts, answering how much and how often. The live mode shows allocations in a running process, answering what is allocating right now. The note says the peak and the leak are different questions with different reports. report answers flamegraph what made the peak so high? flamegraph --leaks what was never freed? table which single allocations were largest? stats how much, and how often? live what is allocating right now? the peak and the leak are different questions — pick the report for the one you have
A high peak and a leak look alike on a dashboard and need different reports to explain.

Configuration options

Option Use Cost
run wrap a script from the start full-run overhead
attach --duration a bounded window on a live process overhead only for the window
--native C extension memory slower, larger capture
flamegraph memory held at the peak none at analysis time
flamegraph --leaks memory never freed none at analysis time
summary ranked table with allocation counts none
--follow-fork prefork workers one capture per child

Verification

Confirm that the change reduced the peak rather than moving it, by comparing summaries from before and after.

python -m memray run -o /tmp/after.bin reproduce_report.py
python -m memray stats /tmp/before.bin | grep -E "Peak memory|Total allocations"
python -m memray stats /tmp/after.bin  | grep -E "Peak memory|Total allocations"

Expected Output: a peak that fell substantially, with allocation count falling alongside it.

Peak memory usage: 1.803GB
Total allocations: 6204417
Peak memory usage: 212.4MB
Total allocations: 1182904

Streaming the rows instead of materialising a list removed most of the peak. Because the allocator retains the high-water mark, that change also reduces the process's steady-state resident memory in production — which a leak fix would not have done.

Common mistakes

Running it against production traffic. Error signature: a slowed service and a capture file too large to analyse. Root cause: every allocation recorded at full rate. Remediation: reproduce the path, or attach to one drained instance for a bounded window.

Opening the leaks report for a peak problem. Error signature: a small leak fixed and the container still killed. Root cause: the peak and the remainder are different questions. Remediation: read the peak flame graph when the problem is hitting a limit.

Forgetting native tracking. Error signature: a report accounting for a fraction of resident memory. Root cause: C extension allocations unattributed. Remediation: rerun with native tracking whenever the numbers do not add up.

Ignoring allocation counts. Error signature: a memory investigation that finds nothing while latency stays poor. Root cause: churn, which is invisible in size-ranked views. Remediation: read the count column and treat high-count paths as a throughput problem.

Frequently Asked Questions

How is memray different from tracemalloc?

tracemalloc snapshots Python-level allocations and compares them. memray intercepts the allocator itself, records every allocation and deallocation with its stack, and can include native frames — so it sees memory allocated inside C extensions, reports the peak rather than only the difference, and produces flame graphs of where memory was held.

Can memray run in production?

Not on a live service under normal load. Recording every allocation with its stack has substantial overhead and produces large capture files. It suits a staging reproduction, a single drained instance, or a short attach window on one process.

What is the difference between the peak and the leaks report?

The peak report shows what was allocated at the moment of highest memory usage, which explains a container hitting its limit. The leaks report shows only what was never freed by the end of the run, which explains steady growth. They answer different questions and often point at different code.

Why does memray show memory that Python's own tools do not?

Because C extensions frequently allocate with the system allocator directly, bypassing Python's. Compression buffers, numeric arrays, database driver result sets and image data are common examples. With native tracking on, memray attributes those allocations to the Python call that triggered them.

Related Guides