Python Observability

Profiling a Live Python Process with py-spy

The instance that is misbehaving right now is the one worth measuring, and it is the one that a restart destroys. py-spy reads a running process's stacks from outside it, which means no restart, no import, and no change to the application. This page covers getting access to the process, the two modes worth knowing, and reading what comes back. It is a task article under CPU profiling Python services, part of the Python profiling and performance observability section.

Reading the interpreter from outside Two processes are shown side by side in the same namespace. The target runs a Python interpreter holding a list of thread states, each of which points to a chain of frame objects, each frame pointing to a code object with a file name and a function name. The profiler process attaches, locates the interpreter state in the target's memory, and walks those structures to produce a stack per thread. It repeats this on a timer. The target is briefly paused during each read and is otherwise unmodified: no module is imported into it, no hook is installed, and it has no way to know it is being observed. The note added is that this is also why the profiler needs permission to read another process's memory, which is the only real obstacle to using it. one process reading another's interpreter state target process — unmodified interpreter state → thread list thread → current frame frame → back-pointer chain code object → file, function reads memory, 100×/s py-spy — separate process walks the structures resolves names counts stacks no import into the target no restart, no code change the same mechanism is why it needs permission to read another process's memory
Nothing is injected into the target. That is the whole value of the approach and the whole reason it needs a capability most container runtimes drop.

Prerequisites

pip install "py-spy>=0.3.14,<0.5.0"

# a debug container sharing the target pod's process namespace
kubectl debug -it checkout-7d9f8c5b6-xk2lm \
  --image=ghcr.io/example/pyspy:0.4.0 --target=app --profile=general -- sh

Implementation

Step 1 — Get into the right namespace and find the process. The profiler must be able to see the target in its process table, which means sharing the process namespace. Inside an ephemeral debug container targeting the application container, the application is usually process one, but confirming rather than assuming saves a confusing minute.

py-spy dump --pid 1 2>/dev/null | head -5 || ps aux | grep -m1 python

Step 2 — Dump when the process appears stuck. A dump prints every thread's current stack once and exits. For a process that has stopped responding, this single command usually ends the investigation: the stack names the call it is waiting in, which is normally a lock acquisition, a socket read, or a native call that does not release the interpreter lock.

py-spy dump --pid 1 --locals

Expected Output: every thread, with the one that matters immediately obvious.

Process 1: gunicorn: worker [checkout]
Python v3.12.4

Thread 0x7f2a (active): "MainThread"
    _recv_bytes (multiprocessing/connection.py:216)
    recv (multiprocessing/connection.py:250)
    get (multiprocessing/queues.py:103)
    consume (app/worker.py:88)

Thread 0x7f31 (idle): "otel-export"
    wait (threading.py:331)
    _export (opentelemetry/sdk/trace/export/__init__.py:355)

Step 3 — Record when the process is slow rather than stuck. A recording samples over a window and aggregates. Sixty seconds at a hundred hertz is six thousand samples, which resolves anything consuming more than about one percent of the process's time and costs one or two percent of a core while it runs.

py-spy record --pid 1 --duration 60 --rate 100 \
  --format speedscope --output /tmp/checkout-oncpu.json

Step 4 — Take the wall-clock version too. Without the idle option, only samples where a thread was executing are counted, so a service blocked on a database produces a nearly empty profile. Including idle threads turns the same window into a picture of where wall-clock time went, and the difference between the two profiles is the blocking.

py-spy record --pid 1 --duration 60 --idle \
  --format speedscope --output /tmp/checkout-wallclock.json

Step 5 — Enable native frames when the Python frames stop explaining things. A process that is busy while its Python frames look idle is executing inside a C extension — a parser, a compression library, a database driver, a numerical routine. Native frames show what it is doing there, at the cost of a slower and more fragile capture.

py-spy record --pid 1 --duration 30 --native --output /tmp/native.svg

Step 6 — Write a format you can diff. A rendered flame graph is readable and not comparable. Speedscope or folded-stack output can be diffed against a stored baseline, which is what turns "this frame is eight percent" into "this frame was three percent last month".

Two recordings of one window The same sixty second window is recorded twice. The on-CPU profile is sparse: the process was executing for only eleven percent of the samples, and within that small slice serialisation and date parsing dominate. The wall-clock profile covers the full window, and the great majority of it is occupied by a database call that never appeared in the first profile because the thread was blocked rather than executing. Subtracting one from the other gives the blocked time per frame, which is the measurement neither profile states directly and which immediately identifies whether the service is slow because of its own code or because of something it is waiting for. one 60-second window, two recordings on CPU — 11% of samples ser the rest of the window is not represented at all wall clock — the whole window psycopg execute — blocked, not executing the subtraction is the finding wide in wall clock and narrow on CPU = waiting · wide in both = working the on-CPU profile alone would have sent somebody to optimise a serialiser that accounts for 7% of the latency
Neither profile states the blocked time. The difference between them does, and it is usually the first thing worth knowing.

Reading a dump during an incident

The dump output repays a little familiarity, because it is the fastest diagnostic available for a process that has stopped making progress and it is read under pressure.

The thread marked active is the one executing. In a process with a dozen threads, most will be idle in a wait or a poll. The active ones are where the work is, and if none is active while the service is unresponsive, the process is blocked on something outside itself.

A stack ending in a lock acquisition is a contention problem. Several threads whose innermost frame is the same acquire call, with one thread inside the critical section, is a lock bottleneck rendered as plainly as it can be. The frame below the acquisition names the lock's purpose.

A stack ending in a socket read is a dependency problem. The frames above it name which dependency. If the same read appears in a dump taken thirty seconds later, the dependency is not merely slow but unresponsive, and the timeout configuration on that call is the thing to look at.

A stack ending in native code, with no Python frame above it, is a C extension. This is where the native option earns its cost. Common cases are a regular expression with catastrophic backtracking, a compression or serialisation routine over a large object, and a database driver processing a very large result set.

Two dumps are better than one. Taking a second dump a few seconds after the first turns a snapshot into a comparison: frames that appear in both are where the process is stuck, and frames that change are where it is progressing slowly. That distinction is worth the extra command.

Making it available before it is needed

The recurring failure with this tool is not technical. It is that the first attempt to use it happens during an incident, discovers a missing capability, and turns into a conversation with a platform team while the service is degraded. Three preparations remove that entirely and take an hour between them.

Build and publish a profiler image. A small image containing the profiler and nothing else, versioned and available from the registry the cluster can pull from. Doing this during an incident means waiting for a build, and pulling a public image may be blocked by policy.

Agree and test the elevated debug profile. Attach to a healthy pod, take a dump, confirm it works, and write down the exact command. This both proves the arrangement and produces the one line somebody will need to paste at three in the morning.

Decide what is acceptable to capture. A profile contains file names, function names and — with the locals option — variable values, which can include data that should not leave the cluster. Deciding in advance whether profiles may be copied out, and where they may be stored, avoids an awkward conversation after the fact. In practice, function-level profiles are almost always fine and the locals option deserves more thought.

A fourth preparation is worth considering for services that are profiled regularly: a small endpoint, protected and internal, that triggers a recording and returns the path. It removes the container attachment step entirely, at the cost of the profiler being present in the application image, which is a trade some teams will take and others will not.

Three py-spy commands and when to use each A table of the three py-spy subcommands and when each helps. py-spy dump prints the current stack of every thread once, in about a second; use it when a process is hung or stuck to see exactly where. py-spy top shows a live, continuously updating view of the functions using the most CPU, like the top command; use it to watch a busy process in real time. py-spy record samples for a duration and writes a flame graph or speedscope file; use it to capture a profile for later analysis or to attach to an incident. The note says dump answers where is it stuck, top answers what is busy now, record answers where did the time go. command output answers py-spy dump every thread's stack, once where is it stuck? py-spy top live CPU-heavy functions what is busy right now? py-spy record flame graph or speedscope where did the time go? for a hung process, dump first — it takes a second and needs no analysis
Each subcommand answers a different incident question. A hang calls for dump; a slow service calls for record.

Configuration options

Flag When Effect
dump the process appears hung one stack per thread, immediately
record the process is slow aggregate over a window
--rate 100 default resolves events above ~1% of the window
--duration 60 default long enough to be representative
--idle latency investigations includes blocked threads
--native Python frames look idle shows C extension frames
--subprocesses prefork servers follows workers from the master
--format speedscope always worth it comparable and diffable output

Verification

Confirm the profiler is seeing the process you think it is, which is the step most often skipped and most often wrong in a multi-container pod.

# confirm the target's identity before trusting the profile
py-spy dump --pid 1 | head -3
cat /proc/1/cmdline | tr '\0' ' '; echo

Expected Output: the command line matching the application, not a shell, an init wrapper, or the wrong container's process.

Process 1: gunicorn: master [checkout]
Python v3.12.4
gunicorn --workers 4 --bind 0.0.0.0:8000 checkout.wsgi:application

A master process here means the recording should use the subprocess option, or it will describe a process whose only job is supervision.

Common mistakes

Profiling the master of a prefork server. Error signature: a profile containing only accept and supervision frames. Root cause: the work happens in worker processes. Remediation: add the subprocess option, or attach to a worker directly.

Recording without the idle option on a latency problem. Error signature: a nearly empty profile of a service that is demonstrably slow. Root cause: the threads are blocked, not executing. Remediation: record both views and compare, as in step 4.

Missing permission, discovered during an incident. Error signature: a permission denied error from the profiler while the service is down. Root cause: the capability is dropped by default and was never arranged. Remediation: agree the debug container profile in advance and test it once while nothing is wrong.

Interpreting a dump as an aggregate. Error signature: a conclusion drawn from one instant that a recording contradicts. Root cause: a dump is a single sample. Remediation: take two dumps, or record, when the question is about proportions rather than about where a process is stuck.

Frequently Asked Questions

Does py-spy need to be installed in the application image?

No, and it is better if it is not. It reads the target process's memory from outside, so it can run in a separate debug container that shares the process namespace, which keeps the application image free of tooling.

What permission does it need?

The ability to read another process's memory, which on Linux means the process trace capability or running as the same user with the appropriate kernel setting. In Kubernetes this usually means an ephemeral container with an elevated security profile, agreed with the platform team in advance.

Does it pause my service?

Briefly, for each sample, while it reads the target's memory. At a hundred samples per second the accumulated pause is well under a percent of wall-clock time, and in practice the overhead is not distinguishable from noise.

Why does the output show no Python frames?

Usually because the interpreter is a build py-spy cannot introspect, or the process is executing inside a native extension and native frames are not enabled. Adding the native option shows the C frames, which is often exactly where a mysteriously busy process turns out to be.

Related Guides