Observing Thread Pool Saturation

A thread pool that has run out of workers does not report an error and does not report slow tasks. Tasks wait in a queue before they start, the waiting is added to whoever submitted them, and every measurement taken inside the task looks normal. This page covers making that wait visible, the gauges that turn saturation into a line touching a ceiling, and the decision between growing the pool and bounding its queue. It is a task article under concurrency and GIL observability, part of the Python profiling and performance observability section.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0" \
            "opentelemetry-api>=1.27.0,<2.0.0"

Implementation

Step 1 — Wrap submission so queue wait is measured. The executor gives no hook for when a task starts, so the wrapper records the submission time and measures the gap at the top of the task itself. The difference between submission and start is the queue wait, and it is the number that changes when the pool saturates.

# instrumented_pool.py
import threading
import time
from concurrent.futures import Future, ThreadPoolExecutor

from opentelemetry import context as otel_context
from prometheus_client import Gauge, Histogram

QUEUE_WAIT = Histogram("pool_queue_wait_seconds", "Submit to start", ["pool"],
                       buckets=(0.001, 0.005, 0.025, 0.1, 0.25, 0.5, 1, 2.5, 5, 10))
RUN_TIME = Histogram("pool_task_run_seconds", "Start to finish", ["pool"])
BUSY = Gauge("pool_busy_workers", "Workers currently running a task", ["pool"])
SIZE = Gauge("pool_max_workers", "Configured pool size", ["pool"])
QUEUED = Gauge("pool_queued_tasks", "Tasks submitted but not started", ["pool"])


class InstrumentedPool:
    def __init__(self, name: str, max_workers: int):
        self._name = name
        self._pool = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix=name)
        SIZE.labels(name).set(max_workers)

    def submit(self, fn, *args, **kwargs) -> Future:
        submitted = time.perf_counter()
        ctx = otel_context.get_current()          # 1. capture the caller's trace context
        QUEUED.labels(self._name).inc()

        def run():
            started = time.perf_counter()
            QUEUED.labels(self._name).dec()
            QUEUE_WAIT.labels(self._name).observe(started - submitted)
            BUSY.labels(self._name).inc()
            token = otel_context.attach(ctx)      # 2. restore it inside the worker
            try:
                return fn(*args, **kwargs)
            finally:
                otel_context.detach(token)
                BUSY.labels(self._name).dec()
                RUN_TIME.labels(self._name).observe(time.perf_counter() - started)

        return self._pool.submit(run)

Step 2 — Gauge busy workers against the pool size. Two lines on one chart — workers busy and workers configured — turn saturation into something visible at a glance: the first line touching the second and staying there. An average utilisation figure hides this, because a pool that is saturated half the time and idle the other half averages to a healthy-looking fifty percent.

Step 3 — Gauge the queue length. Queue length is the earliest indicator that arrivals exceed capacity, and its trend is more informative than its value. A queue that grows during a peak and drains afterwards is coping; one that grows without draining is a pool that has fallen permanently behind.

Expected Output: the three measurements during a saturation event.

pool_max_workers{pool="pdf"}          4
pool_busy_workers{pool="pdf"}         4
pool_queued_tasks{pool="pdf"}       118
pool_queue_wait_seconds p99         6.8
pool_task_run_seconds p99           0.041

Every task runs in forty-one milliseconds and every caller waits nearly seven seconds. Only the first three lines explain the last two.

Step 4 — Carry trace context into the worker. Context is stored per thread, so a task running on a pool thread starts with an empty context unless one is attached explicitly. The wrapper above captures and restores it, so the task's spans join the originating trace and the queue wait appears as a visible gap between the submitting span and the task span — which is the view that makes saturation obvious to anyone reading the trace. The mechanics generalise to any pool boundary, as covered in propagating context across thread and process pools.

Step 5 — Bound the queue when growing the pool will not help. The default executor's queue is unbounded, so overload accumulates as backlog and latency rather than as errors. A semaphore around submission converts it: when the queue is full, submission fails immediately, the caller can return an error or a degraded response, and the backlog never exceeds what the pool can clear in a reasonable time.

class BoundedPool(InstrumentedPool):
    def __init__(self, name: str, max_workers: int, max_queued: int):
        super().__init__(name, max_workers)
        self._slots = threading.BoundedSemaphore(max_workers + max_queued)

    def submit(self, fn, *args, **kwargs) -> Future:
        if not self._slots.acquire(blocking=False):
            raise RuntimeError(f"{self._name} pool full")   # fail fast
        future = super().submit(fn, *args, **kwargs)
        future.add_done_callback(lambda _: self._slots.release())
        return future

Growing the pool or bounding it

When the measurements show saturation, there are two responses and they suit different situations.

Grow the pool when the resource behind it has capacity. If the workers call a service or a database that is lightly loaded, more workers mean more concurrent calls and more throughput. The check is whether the downstream resource's own latency stays flat as the pool grows; if it rises, the pool was not the real constraint and growing it has moved the queue somewhere else.

Do not grow the pool when the work is CPU-bound Python. Additional threads contend for the interpreter lock, as described in measuring GIL contention in Python, and throughput stays flat. The right change is a process pool.

Bound the queue when overload is possible and waiting is worse than failing. A request that will time out after ten seconds gains nothing from sitting in a queue for nine of them. Rejecting it immediately lets the caller retry elsewhere, degrade gracefully or report the problem honestly, and it prevents the backlog that makes recovery slow. This is the right default for any pool on a request path.

Leave the queue unbounded only for work that must eventually complete and is not latency-sensitive. Background tasks that can run late but must run belong on a durable queue anyway; an in-memory unbounded queue is a place where work is lost when the process exits.

The measurements decide which case applies. Queue wait growing while downstream latency is flat says grow the pool; queue wait growing while downstream latency also grows says the pool is protecting something that is already saturated, and bounding it is the kinder option for everyone.

Choosing the pool size from the measurements

Pool sizes are usually set once, from a guess, and never revisited. The measurements above make a better choice possible, and the reasoning is straightforward once the numbers exist.

Little's law gives the relationship: the average number of busy workers equals the arrival rate multiplied by the average run time. A pool receiving eighty tasks per second, each running for fifty milliseconds, keeps four workers busy on average. That is the floor, not the answer — a pool sized exactly at its average load saturates whenever arrivals bunch up, which they always do.

The useful sizing question is what queue wait is acceptable at the peak. The run time histogram gives the service time; the arrival rate at the busiest minute of the busiest hour gives the load; and the gap between the average busy count and the configured ceiling is the headroom that absorbs bursts. Most request-path pools want the busy count at peak to sit around seventy percent of the ceiling, which leaves room for bursts without queueing becoming the dominant latency.

Two constraints override that arithmetic. The downstream resource's capacity is a hard ceiling: a pool of fifty workers calling a service that accepts twenty concurrent requests has thirty workers queueing somewhere else. And for CPU-bound work the useful ceiling is the number of cores, whatever the arrival rate, because workers beyond that only contend.

Re-running the sizing when the run time distribution shifts — after a dependency slows down, or a feature adds work per task — catches the case where a pool that was comfortably sized last quarter is now saturating every afternoon. The run time histogram changing is the trigger; the queue wait histogram is the confirmation.

Configuration options

Verification

Drive the pool past its capacity in staging and confirm each measurement moves the way it should.

import time
from instrumented_pool import BoundedPool

pool = BoundedPool("probe", max_workers=4, max_queued=8)
rejected = 0
for _ in range(100):
    try:
        pool.submit(time.sleep, 0.2)
    except RuntimeError:
        rejected += 1
print(f"rejected {rejected} of 100")

Expected Output: the bound engaging, and the gauges reflecting it.

rejected 88 of 100
pool_busy_workers{pool="probe"}   4
pool_queued_tasks{pool="probe"}   8

Common mistakes

Timing only inside the task. Error signature: normal task durations during a latency incident. Root cause: queue wait happens before the timer starts. Remediation: measure from submission, as in step 1.

Averaging utilisation. Error signature: a pool reported at fifty percent that saturates every afternoon. Root cause: an average hides alternating saturation and idleness. Remediation: chart busy workers against the ceiling, at full resolution.

Growing the pool for CPU-bound work. Error signature: no throughput gain from more workers. Root cause: interpreter lock contention. Remediation: move the work to a process pool.

Leaving the queue unbounded on a request path. Error signature: latency that stays high for minutes after load subsides. Root cause: a backlog that must drain. Remediation: bound the queue and fail fast when it is full.

Losing trace context at the pool. Error signature: worker spans appearing as unrelated root traces. Root cause: context is per thread and was not carried across. Remediation: capture at submission and attach inside the worker.

Frequently Asked Questions