Profiling Overhead Budgets and Safety

Continuous profiling is only defensible if its cost is small, known and bounded. "Small" is the claim every vendor makes; "known" requires measuring it honestly on your own services; and "bounded" requires an agreed ceiling that somebody checks. This page covers all three, plus the safety properties an agent must have before it goes near production. It is a task article under continuous profiling in production, part of the Python profiling and performance observability section.

Prerequisites

pip install "prometheus-client>=0.20.0,<1.0.0"

A load generator capable of steady, repeatable traffic is the other requirement, since every comparison below depends on holding load constant.

Implementation

Step 1 — Agree the ceiling before deploying. An always-on component with no written ceiling drifts: somebody enables wall-clock mode for an investigation and never disables it, somebody raises the sampling rate for a finer picture, a library upgrade doubles the per-sample cost. A figure agreed in advance — two percent of CPU per request and no measurable change in p99 latency is common — turns each of those into a visible breach rather than a slow creep.

Step 2 — Measure per request. Total CPU is dominated by traffic. A rollout that coincides with a traffic increase looks expensive, and one that coincides with a decrease looks free. Dividing CPU by request count removes traffic from the comparison and leaves the per-request cost of serving, which is the number the profiler changes.

# CPU-seconds per request, the figure the budget is written against
sum by (instance, profiling) (rate(process_cpu_seconds_total{service="checkout"}[30m]))
  /
sum by (instance, profiling) (rate(http_server_requests_total{service="checkout"}[30m]))

Step 3 — Compare like with like, at the same time. The cleanest comparison runs the agent on some instances of a service and not others, under the same load balancer, over the same period. Anything that affects the service — a dependency slowing down, a traffic mix shifting — affects both groups equally, and the difference between them is the profiler.

Expected Output: a per-request difference comfortably inside the budget, and no movement in tail latency.

profiling=off   cpu/request 4.12 ms   p99 212 ms
profiling=on    cpu/request 4.19 ms   p99 214 ms
overhead        +1.7% cpu              +0.9% p99 (within noise)

Step 4 — Reduce scope before reducing rate. When a service is over budget, the capture cost is usually the cause, and it scales with the number of threads walked on every tick. Excluding threads that are never interesting — idle pool workers, exporter threads, the profiler's own — often halves the cost without losing any information. Only after that is lowering the sampling rate worth considering, because it reduces resolution everywhere.

PROFILER_OPTIONS = {
    "sample_rate_hz": 100,
    "exclude_idle_threads": True,          # 1. threads blocked in select/poll are skipped
    "exclude_thread_name_prefixes": [      # 2. never interesting, always walked otherwise
        "otel-", "prometheus-", "profiler-",
    ],
    "wall_clock": False,                   # 3. enable per investigation, not permanently
    "upload_period_s": 60,
}

Step 5 — Verify the agent drops rather than blocks. Stop the profiling backend in staging, hold load steady, and watch latency. An agent whose upload path can block the application — synchronously, or by exhausting a shared resource like a thread pool — turns a profiling backend outage into an application incident. This is the single most important safety property and it must be tested rather than assumed.

Step 6 — Alert on budget breaches. The comparison from step 3 can run continuously on a small canary group and alert when the per-request difference exceeds the budget. That is what catches the drift described in step 1, and it costs only the canary.

Safety properties an agent must have

Overhead is one axis of risk. The other is what the agent does when something goes wrong, and four properties are worth verifying for any agent before it runs in production.

It must never block the application. Uploads happen on a background thread, into a bounded buffer, and when the buffer is full the oldest profile is discarded. Any design in which the application waits for the profiling backend — even indirectly, through a shared executor or a lock — is disqualifying, and step 5 is how to check.

It must bound its own memory. An agent that accumulates profiles while its backend is down, without a limit, is a memory leak that appears only during an outage elsewhere. The buffer's maximum size should be known and small.

It must survive interpreter changes gracefully. An in-process agent that reads interpreter internals can break on a new Python release. The acceptable failure is that it disables itself and logs once; the unacceptable one is that it crashes the process. Testing the agent on a new interpreter version before upgrading production is the only way to know which you have.

It must not expose what it captures inappropriately. Profiles contain function names and file paths, and with some options they contain local variable values. Deciding where profiles are stored, who can read them and whether local values are captured at all is a data governance question with the same shape as deciding what goes in log records, and it deserves the same attention.

An agent that satisfies all four, within an overhead budget that is measured and alerted on, is safe to run everywhere. One that fails any of them is a production risk regardless of how small its sampling overhead is.

Overhead against value

A budget is a ceiling, not a target, and it is worth being clear about what the overhead buys so the ceiling is set sensibly rather than as low as possible.

The value of continuous profiling is dominated by rare events: the regression caught after a deploy, the incident diagnosed from a profile of a pod that no longer exists, the fleet-wide cost that turned out to be one serialisation library. None of these happens daily, and each one, when it does happen, is typically worth far more than a year of the overhead. A two percent overhead on a service costing a few thousand a month in compute is a small number next to a single incident shortened by an hour.

That argues against driving the overhead towards zero at the expense of value. A sampling rate lowered to twenty hertz to save a percent of CPU produces profiles that can no longer resolve the thirty-millisecond operation that turns out to matter, and the saving is lost the first time an investigation fails for lack of resolution. Wall-clock mode disabled everywhere to save cost is reasonable; span context disabled to save cost removes most of the reason to run the profiler at all.

The better framing is that the budget protects the service from the profiler, and the configuration within the budget should maximise what the profiler can answer. Spending the budget on span context and a useful sampling rate, while excluding threads that never contribute anything, is the allocation that usually works best — and it is the one most often reversed by an overly literal reading of "reduce overhead".

Configuration options

Verification

Run the drop-not-block test as part of adopting any new agent version.

# steady load, backend stopped for five minutes, latency watched throughout
kubectl scale deploy/profile-backend --replicas=0
python loadtest.py --rps 300 --duration 300 --report p99
kubectl scale deploy/profile-backend --replicas=2

Expected Output: latency unchanged while the backend is absent, and the agent's own counters showing profiles dropped rather than queued without bound.

p99 with backend     214 ms
p99 without backend  216 ms
profiler_profiles_dropped_total  5
profiler_buffer_bytes             262144   (the configured maximum)

Common mistakes

No written ceiling. Error signature: overhead that doubled over six months without anyone deciding it should. Root cause: configuration drift with nothing to compare against. Remediation: agree the budget and alert on breaches.

Before-and-after measurement. Error signature: an overhead figure that varies wildly between rollouts. Root cause: traffic and other changes confounded with the profiler. Remediation: measure side by side, per request.

Lowering the rate first. Error signature: coarser profiles and still over budget. Root cause: capture cost driven by thread count rather than rate. Remediation: exclude idle and uninteresting threads first.

Untested failure behaviour. Error signature: an application incident during a profiling backend outage. Root cause: an upload path that could block. Remediation: run the drop-not-block test before rollout and on every agent upgrade.

Wall-clock mode left on. Error signature: a service over budget that was within it last month. Root cause: an investigation setting never reverted. Remediation: enable wall-clock per investigation with an expiry, and alert on the budget.

Frequently Asked Questions