Reading Flame Graphs for Python Services

A flame graph is a summary of thousands of stacks drawn so that identical prefixes merge. Almost everything people get wrong about it comes from reading the vertical axis as though it meant something. This page covers the rules, the shapes that recur specifically in Python services, and what each one implies about where a change belongs. It is a task article under CPU profiling Python services, part of the Python profiling and performance observability section.

Prerequisites

pip install "py-spy>=0.3.14,<0.5.0"

# a profile to read
py-spy record --pid 1 --duration 60 --format speedscope --output profile.json

Implementation

Step 1 — Fix the axes in your head before looking. The horizontal extent of a bar is the fraction of samples in which that frame was on the stack, which is proportional to time. The vertical position is how deep the frame sits in the call chain, which is proportional to nothing. Bars are also usually sorted alphabetically rather than chronologically, so horizontal position carries no time ordering — a flame graph is not a timeline, and reading it left to right as a sequence of events is a misreading that produces confident nonsense.

Step 2 — Find and collapse the floor. In a web service, every stack begins with the same twenty or thirty frames: the server's accept loop, the application dispatcher, the middleware chain, the router, the view dispatcher. They span the full width because they are in every sample, and they contain no cost of their own. Most viewers can hide or merge them, and doing so is the difference between a graph dominated by machinery and one showing the application's own shape.

Step 3 — Scan upward for the widest bar you own. From the top of the floor, look for the widest frame implemented in your codebase or in a library you can influence. Frames below it are its callers and frames above it are its callees; the one you have found is where the cost concentrates at a level you can act on.

Step 4 — Read the shape above that bar. This decides where the fix goes, and there are two cases. A flat top means the function is executing its own code, so the change is inside it: a better algorithm, an avoided allocation, a cached result. Many narrow towers above it means it is a dispatcher, and its cost is the volume of calls arriving from elsewhere — the change belongs in whatever is calling it so many times.

Step 5 — Check which kind of profile you are holding. An on-CPU graph shows execution; a wall-clock graph shows elapsed time including blocking. The same visual — a wide bar in a database driver — means "this driver is burning cycles" in the first and "this query is slow" in the second, and those lead to entirely different work. The profile's own metadata says which it is, and it is worth confirming rather than assuming.

Step 6 — Diff rather than read. An absolute percentage is hard to interpret; a change in a percentage is not. Most viewers support a differential view, and even without one, a folded-stack comparison is a few lines of script.

# fold two speedscope profiles down to frame totals and compare
import collections, json, sys

def totals(path):
    doc = json.load(open(path))
    frames = doc["shared"]["frames"]
    counts = collections.Counter()
    for prof in doc["profiles"]:
        for frame_index, weight in zip(prof["samples"], prof["weights"]):
            leaf = frame_index[-1] if frame_index else None
            if leaf is not None:
                counts[frames[leaf]["name"]] += weight
    total = sum(counts.values()) or 1
    return {name: n / total for name, n in counts.items()}

old, new = totals(sys.argv[1]), totals(sys.argv[2])
rows = [(new.get(k, 0) - old.get(k, 0), k) for k in set(old) | set(new)]
for delta, name in sorted(rows, reverse=True)[:6]:
    print(f"{delta:+7.2%}  {name}")

Expected Output: one frame accounting for a regression, which is the usual shape.

 +6.02%  parse
 +0.44%  to_representation
 -0.31%  execute
 -0.77%  loads

Shapes that are specific to Python

Several recurring patterns are artefacts of the interpreter or of common Python libraries rather than of the application, and recognising them saves time.

A wide <module> frame means you profiled a startup. Module-level code executes at import, so a profile taken in the first seconds of a process is dominated by it. This is not a finding about the service; it is a finding about when the profile was taken.

<listcomp>, <genexpr> and <lambda> appear as their own frames. They are separate code objects, so a comprehension inside a hot function shows up one level above it. This is useful — it localises the cost within a function — and occasionally confusing when the same name appears in several places.

A wide __getattr__ or descriptor frame means attribute access is the cost. Common in ORMs and serialisation libraries, where every field access goes through a descriptor. The fix is almost never in the descriptor; it is in the number of attribute accesses, which usually means fetching or serialising fields nobody needs.

Garbage collection appears as its own frames, attributed to whoever triggered it. A collection runs on the thread that allocated past the threshold, so its cost lands in an unrelated function that happened to allocate at the wrong moment. A meaningful gc share across many callers is a signal about allocation rate rather than about any of those callers, and it connects to monitoring Python garbage collection and memory usage.

Under asyncio, everything hangs off the loop's frames. run_once, _run, and the task's step function sit between the loop and every coroutine, so the application's structure is fragmented beneath them. Collapsing those frames, or using a profiler that groups by task, restores a readable shape.

What a flame graph cannot tell you

Three questions come up constantly in front of a flame graph and none of them is answerable from one, which is worth knowing before an hour is spent trying.

Whether the cost is one slow call or many fast ones. A frame occupying twenty percent of samples might be one call taking twelve seconds or twelve thousand calls taking a millisecond each. The graph is identical either way. Answering it requires call counts, which means a deterministic profile as described in using cProfile and pstats in production, or a span around the operation.

Which request the cost belongs to. Samples are attributed to frames, not to requests, so a profile of a process serving six endpoints mixes all six. If one endpoint is the problem and it is five percent of traffic, it contributes five percent of the samples no matter how slow it is. Recovering per-request attribution needs the span context recorded alongside each sample.

Whether it got worse. A single graph has no time dimension. Everything about regression detection requires two profiles, which is why storing them matters more than rendering them nicely. A profile that exists only as a picture somebody looked at once has answered a question and left nothing behind.

Configuration options

Verification

Confirm your reading by predicting something and checking it. If a frame is claimed to be a third of the profile, an independent measurement should agree.

# how much of the profile is one frame, computed directly
import collections, json
doc = json.load(open("profile.json"))
frames = doc["shared"]["frames"]
counts = collections.Counter()
for prof in doc["profiles"]:
    for stack, weight in zip(prof["samples"], prof["weights"]):
        for idx in set(stack):
            counts[frames[idx]["name"]] += weight
total = sum(prof["weights"][0] for prof in doc["profiles"]) or 1
print(f"parse appears in {counts['parse'] / sum(doc['profiles'][0]['weights']):.1%} of samples")

Expected Output: a figure matching what the graph appeared to show.

parse appears in 31.4% of samples

A number substantially different from the visual impression usually means the frame appears at several depths and the eye merged only one of them, which is a good reason to check rather than to eyeball.

Common mistakes

Reading height as cost. Error signature: an optimisation aimed at a deep call chain that changes nothing. Root cause: the vertical axis is call nesting. Remediation: rank by width only.

Reading left to right as a sequence. Error signature: a conclusion about what happened before what. Root cause: bars are usually ordered alphabetically, not chronologically. Remediation: treat the graph as an aggregate, and use a trace when ordering matters.

Acting on a framework frame. Error signature: a plan to optimise the request dispatcher. Root cause: the floor spans the full width because it is in every stack. Remediation: collapse the floor and read above it.

Not knowing which profile kind you have. Error signature: optimising a database driver that is doing nothing. Root cause: a wall-clock graph read as on-CPU. Remediation: check the metadata before interpreting.

Profiling a cold process. Error signature: a graph dominated by imports and module-level code. Root cause: the profile was taken during startup. Remediation: profile a warm process under load.

Frequently Asked Questions