Python Profiling and Performance Observability: A Production Guide

Metrics say a service got slower. Traces say which span got slower. Neither says which line of Python is responsible, and that is the gap profiling fills. This guide is for backend engineers and SREs holding a latency regression and a stack trace's worth of suspicion, and it links the deeper material: CPU profiling Python services, memory profiling and leak detection, concurrency and interpreter-lock observability, continuous profiling in production, and database and I/O performance observability. It assumes you already have request metrics and traces — profiling is the third question, asked after those two have narrowed the search.

Key decisions before you profile

• Wall-clock or on-CPU. On-CPU sampling shows where the interpreter burns cycles; wall-clock sampling includes the time threads spend blocked. A service that is slow because of a database needs the second and will look healthy in the first.
• Sampling or deterministic. Sampling costs a fixed small percentage regardless of workload. Deterministic profiling costs in proportion to call count, and Python programs make a great many calls.
• In-process or attached. An attached profiler needs no code change and can be pointed at the instance that is misbehaving right now. An in-process profiler can see application context — the current request, the current tenant — that an attached one cannot.
• One-off or continuous. A one-off profile answers a question you already have. Continuous profiling answers the question you will have at three in the morning about a process that has since been replaced.
• Whether memory is even the right axis. Resident memory, allocation rate and live-object count are three different problems with three different tools; picking the wrong one produces a very detailed answer about the wrong thing.

What a Profiler Actually Measures

A sampling profiler does one thing on a timer: it captures the current call stack of every thread, records it, and goes back to sleep. Do that a hundred times a second for thirty seconds and you have three thousand stacks. The number of times a function appears across those stacks is proportional to the time spent in it. That is the whole mechanism, and understanding it explains both the strengths and the failure modes.

Because samples are counted rather than timed, a profile is statistically accurate in aggregate and meaningless for any single call. A function that took four hundred milliseconds once, during a thirty-second profile at one hundred hertz, appears in roughly forty samples — enough to be visible. A function that took four hundred microseconds ten times appears, on average, in fewer than one. Rare slow things show up; frequent fast things do not, unless they are frequent enough to add up.

Because the stack is captured from outside the running code, a sampling profiler costs the same whether the program makes a thousand function calls a second or a million. This is the property that makes it usable in production, and it is the exact opposite of cProfile, which hooks every call and return and therefore costs the most precisely when the program is busiest. Sampling versus deterministic profilers works through the trade in detail.

Because a sample records what a thread is doing, not what it is waiting for, an on-CPU profile of a thread blocked in a socket read shows nothing at all. Wall-clock profiling includes blocked threads, and the difference between the two profiles is one of the most useful measurements available: a frame wide in wall-clock and narrow on-CPU is waiting; a frame wide in both is working.

Reading a Flame Graph Without Fooling Yourself

A flame graph is a summary of thousands of stacks, drawn so that identical stacks merge. The horizontal axis is sample count — that is, time. The vertical axis is call depth and carries no quantitative meaning whatsoever. This is the single most misread property of the format: a tall narrow tower is a deep call chain that cost almost nothing, and a short wide bar is a shallow function that cost everything.

Read it by scanning the widest bars from the bottom up, and stop at the first one that is code you can change. Below that point are your callers, which are usually the framework; above it are the callees, which are usually the standard library. The frame worth acting on is the widest one whose implementation is yours.

Two structural patterns are worth recognising immediately. A wide plateau of a single frame with nothing above it is a function doing its own work — a tight loop, a serialisation routine, a regular expression. A wide frame with dozens of narrow towers above it is a function called many times from many places; the cost is the call volume, not the function, and the fix is upstream. Reading flame graphs for Python services works through several real shapes and what each one meant.

One Python-specific caution: interpreter frames such as <listcomp>, <genexpr> and __import__ appear in stacks and can dominate a profile taken during startup. A profile of the first ten seconds of a process's life measures imports, not work. Profile a warm process.

Memory: Three Different Problems

"The service is using too much memory" describes at least three unrelated failures, and the tool that diagnoses one is useless against the others.

Resident set growth without live-object growth is fragmentation or allocator retention. Python's allocator holds freed blocks in arenas for reuse and only returns fully empty arenas to the operating system, so a process that once peaked at two gigabytes tends to stay near two gigabytes even when it now needs four hundred megabytes. This is not a leak, it is not fixable in application code, and chasing it with an object-graph tool wastes a day. Diagnosing resident memory growth in Python containers covers how to tell it apart from the real thing.

Live-object growth is a genuine leak: something holds a reference that should have been dropped. Module-level caches without eviction, a logging handler holding formatted records, and closures captured in a long-lived registry are the usual three. tracemalloc compares allocation snapshots and names the line that allocated the growth, which is usually enough to find the reference.

Allocation rate is a throughput problem wearing a memory costume. A handler that allocates two hundred megabytes per request and frees it all still produces garbage-collection pauses, cache pressure and latency that looks like anything except memory. The signal here is the garbage collection metrics, not the resident size.

Concurrency, the Interpreter Lock and the Event Loop

Python's performance failures are disproportionately concurrency failures, and they share a signature: the process is not busy and is nevertheless slow.

Under threads, the global interpreter lock serialises bytecode execution. Eight threads doing CPU-bound work in one process complete in roughly the time one thread would take, plus contention overhead. The observable form is CPU utilisation pinned near one core while the thread count is high and queues grow. Measuring interpreter lock contention in Python covers how to measure the wait rather than infer it.

Under asyncio, there is no lock to contend for, because there is only one thread — which is worse. Any synchronous call on the event loop thread stops every other coroutine, and the symptom appears as latency on unrelated endpoints. This is the measurement described in measuring asyncio event loop lag, and the profiling counterpart is in diagnosing blocked event loops in production.

Under a thread pool, the failure is saturation: every worker is busy, the queue grows without bound, and the latency an individual task reports is the queue wait rather than the work. A pool with no queue-depth metric is a place where latency hides, and instrumenting it is the subject of observing thread pool saturation.

Continuous Profiling and the Link to Traces

A one-off profile requires that the problem be happening while you watch. Production problems rarely cooperate, and the instance that misbehaved has usually been replaced by the time anyone looks.

Continuous profiling solves this by sampling constantly at a low rate and storing the result, so a profile exists for any past minute. The storage cost is small because profiles compress extremely well — the same stacks repeat — and the runtime cost of a well-implemented sampler at a hundred hertz is in the low single digits of one core. Running continuous profiling for Python services covers deployment; profiling overhead budgets and safety covers how to bound what it costs.

The higher-value capability is correlation. If the profiler records the active trace and span identifiers alongside each sample, the profile becomes queryable by request: show me the stacks sampled while this endpoint was running, or while this one slow trace was in flight. That turns "the service spends nine percent of its time in date parsing" into "the checkout endpoint spends forty percent of its time in date parsing, which is why it is the slow one". The mechanics — reading the current span context from a sampling thread without perturbing it — are covered in linking profiles to traces with span context, and it depends on the same context propagation machinery the tracing SDK already maintains.

Production Code Examples

An in-process wall-clock sampler that tags each sample with the active span, written to show the mechanism rather than to replace a real profiler.

# sampler.py — a minimal correlated sampler, ~60 lines of mechanism.
import sys
import threading
import time
from collections import Counter

from opentelemetry import trace

SAMPLES: Counter = Counter()
_stop = threading.Event()


def _stack_key(frame):
    """Collapse a frame chain into a single 'a;b;c' folded-stack key."""
    parts = []
    while frame is not None and len(parts) < 64:
        code = frame.f_code
        parts.append(f"{code.co_filename.rsplit('/', 1)[-1]}:{code.co_name}")
        frame = frame.f_back
    return ";".join(reversed(parts))


def _sample_once():
    # 1. One snapshot of every live thread's stack.
    frames = sys._current_frames()
    # 2. The span context is thread-local, so read it per thread id.
    span = trace.get_current_span()
    ctx = span.get_span_context()
    trace_id = f"{ctx.trace_id:032x}" if ctx.is_valid else "-"
    for thread_id, frame in frames.items():
        if thread_id == threading.get_ident():
            continue                      # never profile the sampler
        SAMPLES[(trace_id, _stack_key(frame))] += 1


def run(hz: int = 100):
    """Sample at a fixed rate until stopped. Cost is O(threads) per tick."""
    interval = 1.0 / hz
    while not _stop.wait(interval):
        _sample_once()


def start(hz: int = 100) -> threading.Thread:
    t = threading.Thread(target=run, args=(hz,), daemon=True, name="sampler")
    t.start()
    return t


def top(n: int = 5):
    for (trace_id, stack), count in SAMPLES.most_common(n):
        print(f"{count:6d}  trace={trace_id[:8]}  {stack.rsplit(';', 1)[-1]}")

Expected Output: after thirty seconds under load, the folded stacks ranked by sample count, with the request each belongs to.

  1184  trace=9f2a71c4  serializers.py:to_representation
   902  trace=9f2a71c4  dateutil.py:parse
   431  trace=3b18ee02  views.py:list_orders
   118  trace=-         psycopg.py:execute
    44  trace=-         gc.py:collect

Two readings follow immediately. Date parsing is the second-widest frame and belongs to one trace, so it is endpoint-specific rather than fleet-wide. And psycopg.execute is narrow here despite being the slowest span in the trace — because this is an on-CPU sampler and the database call was not on the CPU, exactly as described above.

A memory comparison that names the line responsible for growth rather than the objects that resulted from it:

# leakcheck.py — two snapshots, one difference.
import tracemalloc
import gc

tracemalloc.start(25)          # keep 25 frames so the caller is identifiable

baseline = tracemalloc.take_snapshot()
serve_requests(count=5_000)    # the workload under suspicion
gc.collect()                   # drop anything merely unreferenced
current = tracemalloc.take_snapshot()

# Group by the line that allocated, not by object type.
for stat in current.compare_to(baseline, "lineno")[:5]:
    print(f"{stat.size_diff / 1024:9.1f} KiB  {stat.count_diff:+7d} blocks  {stat}")

Expected Output: a single line accounting for nearly all growth, which is the ordinary shape of a real leak.

  84210.4 KiB   +50021 blocks  app/cache.py:31: size=84.4 MiB (+84.2 MiB), count=50021 (+50021)
    412.9 KiB     +311 blocks  app/serializers.py:88: size=1.2 MiB (+412 KiB), count=902 (+311)
     31.0 KiB      +12 blocks  logging/__init__.py:1620: size=88 KiB (+31 KiB), count=44 (+12)

Fifty thousand blocks retained by one line in a cache module, after a workload of five thousand requests, is ten retained objects per request that nothing ever removes — an unbounded dictionary, almost certainly keyed by something request-specific.

Choosing a Profiler for a Python Service

The Python ecosystem offers several profilers, and they are not interchangeable. The choice follows from one question — can this run against a process that is serving traffic right now?

Three rows deserve comment. py-spy is the default answer for an unhealthy production process precisely because the process needs to know nothing about it: it reads stacks from outside, so no restart, no import and no configuration change is needed, which matters when the thing you want to measure is happening now and will stop when the pod is replaced. Details are in profiling a live Python process.

cProfile remains the right tool for a benchmark, a test, or a batch job you can run twice — it gives exact call counts, and call counts are what identify a function called four thousand times when it should have been called four. Running it in a live service is the mistake; running it in a reproduction is often the fastest route to an answer. Using cProfile and pstats in production covers the narrow circumstances where it is defensible and how to bound the damage.

memray intercepts allocations and reports them with full stacks, which is dramatically more informative than tracemalloc and dramatically more expensive. It suits a staging reproduction or a short capture window on one instance, and it is covered in profiling memory with memray.

Whatever the tool, the output format worth standardising on is folded stacks or pprof, because both are consumed by every flame graph renderer and by most continuous profiling backends. A profiler whose output only its own viewer can read makes comparison across time — the thing that gives a profile meaning — needlessly difficult.

When the Time Is Not in Python at All

Most latency in a typical Python backend is spent waiting on something else, and the profiler's honest answer — an almost empty on-CPU graph — is frequently misread as a broken measurement. The productive move at that point is to stop profiling and start looking at the boundaries.

Database work is the usual destination. The three failures are a slow query, too many queries, and a connection pool that is exhausted, and they are distinguishable only if the spans exist. A single span per query with the statement and its duration turns "the endpoint is slow" into "one query takes 900 ms" or "this endpoint issues 340 queries"; without them, both look identical from the outside. Tracing slow SQL queries in Python covers the instrumentation, and detecting repeated query patterns with traces covers the query-count case, which is the most common and the least visible, because each individual query is fast.

Pool exhaustion deserves separate mention because its signature is deceptive: every query is fast, every request is slow, and CPU is flat. The waiting happens before the query span starts, in the call that acquires a connection, so a naive instrumentation shows nothing at all. Measuring the acquisition wait separately from the query duration is the only way to see it, and it is the subject of observing connection pool exhaustion.

Outbound HTTP has the same structure with an extra hazard: client libraries retry. A request that reports 2.4 seconds may be three attempts of 800 milliseconds, and the distinction matters enormously — the first is a slow dependency, the second is a flapping one. Instrumenting attempts rather than calls is covered alongside instrumenting aiohttp client requests.

Building a Profiling Practice

Profiling becomes useful when it stops being an emergency skill. Four habits do most of the work.

Keep a baseline. A profile of the service under normal load, taken monthly and stored with the release it belongs to, converts every future profile into a comparison. A frame that is eight percent of CPU means nothing; a frame that was three percent last month and is eight percent now means a change landed.

Profile before optimising and after. The single most common outcome of a confident optimisation is that the cost moves rather than disappears — a serialisation improvement that pushes the work into garbage collection, a caching layer that turns CPU time into lock contention. Only the second profile shows this.

Attach the profile to the incident. A profile captured during an incident and pasted into the review is worth more than any recollection of it, and it costs thirty seconds while the process is still alive. After the pod is replaced it cannot be recovered at all, which is the argument for continuous profiling stated as an operational habit rather than a feature.

Let the profile inform the instrumentation. A frame that keeps appearing wide in profiles is a candidate for its own span, so the next investigation can start from a trace instead of a profiler. Over time this migrates knowledge out of the profiling tool and into span attributes where everyone on call can see it, which is the difference between a service one person can debug and a service the team can.

Common Mistakes

Profiling the wrong process. A profile taken on a healthy replica describes a healthy replica. Attach to the instance whose latency is bad, which means the profiler has to be usable without a restart.

Profiling during startup. The first seconds of a Python process are dominated by imports and module-level initialisation. Every profile taken then agrees with every other, and none of them describes the service.

Reading depth as cost. The vertical axis of a flame graph is call nesting. A tower twenty frames high may account for two samples out of three thousand.

Using cProfile on a production hot path. Deterministic profiling multiplies runtime by a factor that depends on call volume, which in a framework-heavy Python service is large. It is a development tool, and the load it adds can itself cause the incident.

Mistaking allocator retention for a leak. Resident memory that rises and plateaus is usually the allocator keeping arenas. A leak keeps rising. The distinguishing evidence is the daily minimum, not the peak.

Trusting a single profile. Sampling is statistical, and a thirty-second window can catch an unrepresentative thirty seconds — a cache warm-up, one unusually large request, a background job that runs on the minute. Two profiles taken at different times that agree are evidence; one profile is a hypothesis.

Optimising a frame you cannot change. A wide frame inside a third-party library is a real cost and rarely a real target. The actionable question is which of your own call sites is producing that volume of calls, which means reading the frame below it rather than the frame itself.

Ignoring the waiting. An on-CPU profile of a service bottlenecked on a database is a nearly empty graph. That emptiness is the finding, and it is routinely read as "the profiler is broken".

Related Reading

• CPU profiling Python services — attaching a profiler, sampling rates, and what the numbers mean.
• Memory profiling and leak detection — telling retention, leaks and allocation churn apart.
• Concurrency and interpreter-lock observability — the three ceilings and how to measure each one.
• Continuous profiling in production — always-on profiles and correlating them with traces.
• Database and I/O performance observability — where the time goes when it is not in Python.
• Python metrics and instrumentation — the signal that tells you a profile is worth taking.
• Python Telemetry Pipelines and Delivery — log shipping, collector topology, delivery guarantees and telemetry cost.
• Python Logging Fundamentals — handlers, formatters, levels and structured records.

Frequently Asked Questions