Concurrency and GIL Observability in Python

A Python service that is slow while its CPU is idle is almost always hitting a concurrency limit, and Python has three distinct ones: the interpreter lock that serialises bytecode across threads, the single thread that runs an asyncio event loop, and the fixed size of a worker pool. Each produces a different signature and each needs a different measurement. This guide covers telling them apart and measuring each directly. It is part of the Python profiling and performance observability section. The focused articles in this topic are Diagnosing Blocked Event Loops in Production, Measuring GIL Contention in Python and Observing Thread Pool Saturation.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "py-spy>=0.3.14,<0.5.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0"

Concept and architecture

The three limits share a symptom and have nothing else in common, so it is worth being precise about each.

The interpreter lock permits one thread at a time to execute Python bytecode in a process. Threads release it during blocking system calls — socket reads, file operations, sleeps — so I/O-bound threads run concurrently in practice. CPU-bound threads do not: eight of them in one process complete in roughly the time one would take, plus switching overhead. The ceiling is therefore about one core of Python execution per process, and it appears only for CPU-bound work.

The event loop runs every coroutine in an asyncio service on a single thread. There is no lock contention between coroutines because there is only one thread, and there is also nothing to rescue them if one misbehaves: any synchronous call on the loop thread — a blocking driver, a large JSON parse, a synchronous log handler — stops every other coroutine until it returns. The ceiling is the loop thread's availability, and the cost of exceeding it lands on unrelated requests.

A worker pool has a fixed number of workers and an unbounded queue by default. When every worker is busy, new work waits, and the wait happens before the task's own timing begins. The ceiling is the pool's size, and exceeding it is invisible to any measurement that starts when the task starts.

What unites them is the observable outcome: latency rises, throughput stops rising, and the host looks under-used. What separates them is where the waiting happens — for the lock, for the loop, or in the queue — and each of those can be measured directly.

Step-by-step implementation

Step 1 — Compare CPU utilisation with configured concurrency. This is the one-minute check that classifies the problem. A threaded process with sixteen workers using exactly one core while latency climbs is lock-bound. An asyncio process at twenty percent CPU with high latency everywhere is loop-bound. A pool whose worker count equals its busy count, continuously, is saturated.

# CPU used against the one-core ceiling of a single interpreter
rate(process_cpu_seconds_total{service="checkout"}[5m])

Step 2 — Measure event loop lag in asyncio services. A sampler that sleeps for a known interval and records how late it wakes measures the delay every coroutine is experiencing. It is the single most useful metric for an async service and it costs one coroutine. The details are in measuring asyncio event loop lag.

import asyncio, time
from prometheus_client import Histogram

LOOP_LAG = Histogram("event_loop_lag_seconds", "Scheduling delay",
                     buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0))

async def lag_sampler(interval: float = 0.1):
    while True:
        due = time.perf_counter() + interval
        await asyncio.sleep(interval)
        LOOP_LAG.observe(max(0.0, time.perf_counter() - due))

Step 3 — Measure queue wait for pools. The time between submission and the moment a worker begins is the number that a saturated pool hides. Wrapping the submission records both, and the difference between them is the saturation.

import time
from concurrent.futures import ThreadPoolExecutor
from prometheus_client import Histogram

QUEUE_WAIT = Histogram("pool_queue_wait_seconds", "Time from submit to start", ["pool"])
RUN_TIME = Histogram("pool_task_seconds", "Time from start to finish", ["pool"])

def instrumented_submit(pool: ThreadPoolExecutor, name: str, fn, *args):
    submitted = time.perf_counter()
    def run():
        started = time.perf_counter()
        QUEUE_WAIT.labels(name).observe(started - submitted)
        try:
            return fn(*args)
        finally:
            RUN_TIME.labels(name).observe(time.perf_counter() - started)
    return pool.submit(run)

Step 4 — Take a wall-clock profile to see the waiting. An on-CPU profile of a lock-bound or loop-bound service shows relatively little, because the waiting consumes no processor. Including blocked threads shows where each thread is waiting, and for lock contention it shows threads parked in the lock's acquisition path.

py-spy record --pid 1 --duration 60 --idle --output /tmp/wall.svg

Step 5 — Record the concurrency limit as a metric. The configured pool size, the worker count and the thread count are facts about the service that belong on the same dashboard as the latency. A saturation is instantly legible when busy workers and configured workers are drawn on the same axis, and invisible when one of them lives in a configuration file.

Step 6 — Change the model when the limit is structural. Tuning around a structural ceiling produces small gains. CPU-bound work that is lock-bound needs processes; synchronous calls on an event loop need to move to a thread pool or be replaced with async equivalents; a saturated pool needs either more workers — if the resource behind it can take them — or back-pressure so callers fail fast rather than queue indefinitely.

Configuration reference

Async and concurrency considerations

The measurements above interact with the concurrency model they measure, and three interactions are worth knowing.

The loop lag sampler is itself a coroutine, so it measures lag only when it gets to run. A loop blocked for two seconds delays the sampler by two seconds and it records exactly that, which is the point. But a loop blocked indefinitely produces no samples at all, and a lag histogram that simply stops receiving observations is the signature of a hung loop rather than a healthy one. Alerting on the absence of lag samples, not only on their value, covers that case.

Metrics recording from worker threads takes a short lock inside the metrics library. At normal rates this is negligible; at very high rates from many threads it becomes a small contention point of its own, which is a strange thing to find in a profile of a service whose problem is contention. Recording per batch rather than per item, where the work is very fine-grained, avoids it.

Under a prefork server, each worker process has its own interpreter lock, its own event loop if it runs one, and its own pools. Every measurement here is per process, and aggregating across workers hides the case where one worker is saturated while others idle — which is common when a load balancer's connection affinity concentrates traffic. Recording the worker identity alongside these metrics, at least temporarily, makes uneven distribution visible, and it pairs with the multiprocess collection described in Prometheus multiprocess mode with Gunicorn.

Production code examples

A single module that exposes all three measurements, so the classification can be read from one dashboard:

# concurrency_metrics.py
import asyncio
import os
import threading
import time

from prometheus_client import Gauge, Histogram

THREADS = Gauge("process_threads", "Live threads in this process")
LOOP_LAG = Histogram(
    "event_loop_lag_seconds", "How late the loop ran a scheduled wakeup",
    buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5))
LOOP_LAST_SAMPLE = Gauge("event_loop_last_sample_timestamp_seconds",
                         "When the lag sampler last ran")


def _thread_counter(interval: float = 5.0) -> None:
    while True:
        THREADS.set(threading.active_count())
        time.sleep(interval)


async def _lag_sampler(interval: float = 0.1) -> None:
    while True:
        due = time.perf_counter() + interval
        await asyncio.sleep(interval)
        LOOP_LAG.observe(max(0.0, time.perf_counter() - due))
        # 1. The absence of this advancing is itself the signal of a hung loop.
        LOOP_LAST_SAMPLE.set_to_current_time()


def install(loop: asyncio.AbstractEventLoop | None = None) -> None:
    threading.Thread(target=_thread_counter, daemon=True, name="thread-counter").start()
    if loop is not None:
        loop.create_task(_lag_sampler())

Expected Output: during a blocked-loop incident, lag spikes and CPU stays low — the classification made by two numbers.

event_loop_lag_seconds p99       1.84
process_cpu_seconds rate         0.21
process_threads                  11
event_loop_last_sample age       0.3 s

The alert rules that encode the classification, so on-call is told which limit rather than only that something is slow:

# lock-bound: CPU pinned at one core with a thread pool configured larger than one
rate(process_cpu_seconds_total[5m]) > 0.9 and process_threads > 4

# loop-bound: lag high, CPU low
histogram_quantile(0.99, rate(event_loop_lag_seconds_bucket[5m])) > 0.1
  and rate(process_cpu_seconds_total[5m]) < 0.5

# loop hung: the sampler has stopped running at all
time() - event_loop_last_sample_timestamp_seconds > 10

# pool-bound: tasks are waiting to start
histogram_quantile(0.99, rate(pool_queue_wait_seconds_bucket[5m])) > 0.5

Expected Output: one alert, naming the limit.

EventLoopBlocked  firing  service=checkout  lag_p99=1.84s  cpu=0.21

Choosing a concurrency model from measurements

Most services choose their concurrency model at creation and never revisit it, which is reasonable until the model becomes the ceiling. The measurements above make the revisit a decision rather than a rewrite prompted by an incident.

If the work is mostly waiting on I/O, threads and asyncio both work. The interpreter lock is released during blocking calls, so a threaded service spends most of its time with the lock free. asyncio handles very high connection counts more economically, and it is less forgiving: one synchronous call ruins it. A threaded service degrades gracefully when somebody adds a blocking call; an async one degrades sharply, and the degradation lands on every request in the process rather than on the one that made the call, which makes it much harder to attribute. For teams without strong discipline around what runs on the loop, threads are the lower-risk choice even where asyncio is theoretically better.

If the work is CPU-bound Python, processes are the answer. No arrangement of threads makes CPU-bound Python faster in a standard build, and asyncio makes it worse by adding a single point of blockage. A prefork server, a process pool for the heavy operations, or moving the work into a native extension that releases the lock are the options, in rough order of disruption.

If the work is mixed, separate it. The usual shape is an I/O-bound request path with occasional CPU-heavy operations — report generation, image processing, large serialisations. Keeping the request path in threads or asyncio and sending the heavy work to a process pool gives each its appropriate model, and the queue wait metric on that pool becomes the capacity signal for the heavy work.

The measurement that decides between these is not a benchmark but the ratio of on-CPU time to wall-clock time for a representative request, which the two profiles from step 4 provide directly. A ratio near one means CPU-bound; a ratio near zero means waiting. Most web requests sit well below a tenth, which is why most services never hit the interpreter lock and why so many investigations that start by blaming it end somewhere else.

Connection pools are pools too

The pool saturation pattern applies to every bounded resource, not only to thread pools, and the most common instance in practice is a database connection pool.

A connection pool has a fixed size and callers wait when all connections are in use. The waiting happens in the acquisition call, before the query begins, so a query span shows a fast query and the request is nevertheless slow. The signature and the remedy are identical to a thread pool: measure acquisition wait separately from query duration, and treat a growing acquisition wait as saturation. Observing connection pool exhaustion covers the instrumentation for the common drivers.

What makes connection pools worse than thread pools is that the obvious fix — a larger pool — moves the limit to the database, which has its own ceiling shared across every service. A pool sized up to relieve one service's saturation can exhaust the database's connection limit for all of them. Sizing connection pools is therefore a fleet-level calculation: the sum of every service's maximum connections across every replica must fit inside what the database accepts, with headroom, and the measurements from each service's pool are the inputs to that sum.

What changed with free-threaded builds

Recent Python releases offer an optional build without the interpreter lock, and it is worth being clear about what that does to the picture above.

It removes the first limit for workloads that can use it: CPU-bound threads in one process can execute in parallel, and the signature of CPU pinned at one core disappears. That is a real change for services that were forced into multiple processes solely to use more than one core.

It does not change the second or third limits at all. An event loop is still one thread, and a synchronous call on it still stops every coroutine; a pool is still a fixed number of workers with a queue in front. Those failures are about the concurrency model's structure, not about the lock.

It also introduces contention elsewhere. Shared data structures that the lock used to protect implicitly now need their own synchronisation, and a hot shared dictionary or counter can become a serialisation point of its own. The measurement changes from "CPU pinned at one core" to "threads waiting on a specific lock", which a wall-clock profile shows in the same way. The method in this guide — classify first, then measure the specific wait — transfers unchanged.

For most services the practical position is that the measurements remain correct under either build, and the free-threaded one simply moves the first ceiling for the workloads that were hitting it. Confirming that a service is actually lock-bound, rather than loop-bound or pool-bound, is what decides whether that build would help at all.

Common mistakes

Adding threads to a lock-bound service. Error signature: throughput unchanged, latency slightly worse. Root cause: CPU-bound Python threads serialise on the interpreter lock. Remediation: move the CPU-bound work to processes.

Profiling a loop-bound service on CPU. Error signature: a nearly empty profile and a slow service. Root cause: the loop is blocked, not busy. Remediation: measure loop lag, and profile with idle threads included.

Timing tasks from inside the task. Error signature: normal task durations during a latency incident. Root cause: queue wait happens before the timer starts. Remediation: measure submission-to-start separately.

An unbounded queue in front of a pool. Error signature: latency rising for minutes after load subsides. Root cause: a backlog built during the peak that has to drain. Remediation: bound the queue and shed or reject when full, so callers fail fast rather than wait indefinitely.

Averaging across workers. Error signature: one saturated worker invisible in fleet-wide metrics. Root cause: per-process limits hidden by aggregation. Remediation: record per worker during investigation.

Sizing a connection pool in isolation. Error signature: one service's pool increase followed by connection refusals across several services. Root cause: the database's connection limit is shared, and per-service pool sizes were chosen without reference to it. Remediation: sum maximum connections across services and replicas, and size each pool within that budget.

Alerting on lag value only. Error signature: no alert while the loop is completely hung. Root cause: a hung loop produces no lag samples. Remediation: alert on the age of the last sample as well.

Frequently Asked Questions