Measuring GIL Contention in Python

The interpreter lock is the first thing blamed when a threaded Python service stops scaling and the thing least often confirmed. It is a real limit for CPU-bound Python work and an irrelevant one for most web services, whose threads spend their time waiting on I/O with the lock released. This page covers confirming it is the bottleneck before anything is rewritten. It is a task article under concurrency and GIL observability, part of the Python profiling and performance observability section.

Prerequisites

pip install "py-spy>=0.3.14,<0.5.0" \
            "prometheus-client>=0.20.0,<1.0.0"

Implementation

Step 1 — Check CPU against thread count. This single comparison rules the lock in or out for most services. A process with sixteen worker threads, a rising latency and CPU utilisation flat at almost exactly one core is serialised: sixteen threads want to run Python and one at a time is allowed to. The same process at thirty percent of one core is not lock-bound, however many threads it has — it is waiting on something else.

# per process, in cores; a plateau at ~1.0 under rising load is the signature
rate(process_cpu_seconds_total{service="report-builder"}[1m])

Step 2 — Take both profiles and compare. An on-CPU profile shows which Python is executing. A wall-clock profile with idle threads included shows every thread's state, including threads that are ready to run and waiting for the lock. The frames that appear heavily in the second and lightly in the first are where threads are parked; when that parking is in the interpreter's lock-acquisition path rather than in a socket read, contention is confirmed.

py-spy record --pid 1 --duration 60 --output /tmp/oncpu.svg
py-spy record --pid 1 --duration 60 --idle --output /tmp/wall.svg

Step 3 — Measure scheduling delay directly. The contention is the gap between a thread becoming ready and actually running. A probe thread that asks to wake at a fixed interval and records how late it runs measures that gap — the same technique as event loop lag, applied to the interpreter's own scheduler. Under no contention the probe wakes within a fraction of a millisecond; under heavy contention it waits for the switch interval of every other runnable thread.

import threading, time
from prometheus_client import Histogram

LOCK_WAIT = Histogram(
    "interpreter_schedule_delay_seconds",
    "How late a ready thread actually ran",
    buckets=(0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1))

def _probe(interval: float = 0.05):
    while True:
        due = time.perf_counter() + interval
        time.sleep(interval)                 # releases the lock
        # Re-acquiring it to run this line is the contended step.
        LOCK_WAIT.observe(max(0.0, time.perf_counter() - due))

threading.Thread(target=_probe, daemon=True, name="lock-probe").start()

Expected Output: a delay distribution that shifts right as CPU-bound threads are added.

threads=1   p50 0.08 ms   p99 0.31 ms
threads=4   p50 4.9 ms    p99 14.8 ms
threads=16  p50 19.6 ms   p99 71.2 ms

Step 4 — Vary the thread count under steady load. The confirming experiment: hold the offered load constant and change the number of worker threads. If throughput rises with threads, the lock is not the constraint. If throughput stays flat and the scheduling delay rises, it is. This takes minutes in a staging environment and removes any remaining doubt.

Step 5 — Check that the hot frames are Python. A CPU-heavy process whose hot frames are inside a compression library, a numerical routine or a cryptography call is usually not lock-bound, because those extensions release the lock during their native work and run in parallel. The distinction matters because the remedy differs: lock-bound Python needs processes, while a native-heavy workload may already be getting the parallelism it can.

Step 6 — Decide the remedy from the measurement. Confirmed contention on CPU-bound Python means moving that work into processes — a process pool for the heavy operations, or more worker processes with fewer threads each. Anything else means the lock was a red herring, and the concurrency guide's other two limits are where to look next.

Why it is blamed more often than it is guilty

The interpreter lock has a reputation that exceeds its actual impact on typical services, and understanding why saves investigation time.

Most Python backend work is I/O-bound. A request handler that queries a database, calls two downstream services and serialises a response spends perhaps five percent of its wall-clock time executing Python and the rest waiting. During that waiting the lock is released. With twenty threads, each holding the lock for five percent of the time, the expected contention is low, and the measurement in step 3 typically shows sub-millisecond scheduling delay.

The lock becomes a real constraint when the ratio shifts. Serialising very large responses, parsing large payloads, running business logic over big collections in Python, rendering templates with heavy loops — each of these moves time from waiting to executing, and past a certain point the threads start queuing for the lock. The measurement makes the shift visible as it happens, which is more useful than any rule about which workloads are affected.

A second reason for the reputation is that the lock's symptom — CPU capped at one core — resembles other problems. A single-threaded bottleneck anywhere in the process, a lock in application code, or a connection pool of size one all produce a similar picture from a distance. The scheduling delay probe distinguishes the interpreter lock from these specifically, because it measures the interpreter's own scheduling rather than any application resource.

The practical upshot is to measure first. A service that has been rewritten to use processes because of assumed lock contention, when the real constraint was a database pool, has gained a more complex deployment and nothing else. The experiment in step 4 costs minutes and prevents that outcome.

Contention that looks like the interpreter lock but is not

Several application-level constraints produce a picture close enough to lock contention to be mistaken for it, and the scheduling delay probe is what tells them apart.

A lock in application code. A threading.Lock around a shared cache, a counter or a lazily-initialised client serialises every thread that touches it. CPU is low rather than pinned, because the waiting threads are blocked on a lock the interpreter releases while waiting. The wall-clock profile shows threads parked on the application lock's acquisition, with the lock's owner visible in the frame beneath it, and the interpreter's own scheduling delay stays low.

A single-connection resource. A client object with an internal connection that allows only one request at a time — some SDK clients behave this way by default — serialises threads through that connection. The symptom is similar: throughput flat in thread count, low CPU. The frame where threads wait is inside the client library.

Logging to a slow handler. A synchronous handler writing to a slow sink holds the handler's own lock during the write, so every thread that logs queues behind it. This is surprisingly common and appears as threads parked in the logging module, which is the pattern non-blocking logging with QueueHandler exists to remove.

In all three, CPU is low and the probe reports little delay — the interpreter is free, and threads are waiting on something the application arranged. That combination points away from the interpreter lock and towards whatever the wall-clock profile shows threads waiting on.

Configuration options

Verification

The fix is verified by repeating the experiment with the new arrangement and seeing throughput scale.

# the same load test, before and after moving CPU-bound work to a process pool
for mode in threads processes; do
  RUN_MODE=$mode python loadtest.py --rps 400 --duration 60 | tail -1
done

Expected Output: throughput and latency both improving with processes, and scheduling delay falling back to sub-millisecond.

threads    throughput 118 req/s   p99 2.41 s   schedule_delay_p99 71.2 ms
processes  throughput 402 req/s   p99 0.19 s   schedule_delay_p99 0.4 ms

Common mistakes

Assuming the lock without measuring. Error signature: a service rewritten to use processes with no improvement. Root cause: the constraint was elsewhere — a pool, a database, a blocked loop. Remediation: run the CPU-versus-threads check and the thread count experiment first.

Adding threads to fix throughput. Error signature: latency worsening slightly as threads are added. Root cause: more competitors for the same lock, plus switching overhead. Remediation: add processes for CPU-bound work.

Blaming the lock for native-heavy CPU. Error signature: high CPU on a process whose hot frames are in a compression or numeric library. Root cause: those extensions release the lock and are already parallel. Remediation: check the hot frames before concluding.

Tuning the switch interval as a fix. Error signature: a small change in latency distribution and no change in throughput. Root cause: the interval redistributes waiting rather than removing it. Remediation: treat it as a latency adjustment for mixed workloads, not a contention fix.

Reading one process in a prefork fleet. Error signature: conflicting conclusions from different workers. Root cause: uneven load across workers. Remediation: run the check on several workers, or on the busiest one deliberately.

Frequently Asked Questions