Continuous Profiling in Production for Python

An on-demand profile requires the problem to be happening while somebody is watching, on an instance that still exists. Production rarely cooperates: the slow minute was at three in the morning, and the pod that had it has been replaced twice since. Continuous profiling removes that constraint by sampling constantly at a low rate and keeping the result, so a profile exists for any recent minute of any instance. This guide covers how it works for Python, what it costs, and how it joins with traces. It is part of the Python profiling and performance observability section. A focused article in this topic goes further: Running Continuous Profiling for Python Services.

Prerequisites

The profiler agent depends on the backend chosen; the application-side requirement is that resource attributes and span context are available to it.

pip install "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "py-spy>=0.3.14,<0.5.0"

# the same resource attributes the traces carry, so the signals join
export OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,service.version=2026.09.18,deployment.environment=prod"

Concept and architecture

Continuous profiling is the sampling profiler from CPU profiling Python services with three changes: it never stops, it ships its output somewhere, and it labels that output well enough to be queried.

The sampler runs constantly at a modest rate. A hundred samples per second is enough to characterise a service within a minute and cheap enough to leave on. Each collection window produces one profile — a set of stacks with counts — which is aggregated in the process before being sent, so the network cost is one small payload per window rather than one per sample.

Profiles are labelled with the same attributes as every other signal. Service, version, environment, instance. This is what makes profiles comparable across time and deployments: a query can ask for the profile of the checkout service at version N and compare it with version N−1, which is how a regression is located without anyone reproducing it.

Samples carry span context. The active trace and span identifiers, read from the thread's context at the moment of sampling, are attached to each stack. This is the capability that distinguishes continuous profiling from a merely always-on profiler: a profile becomes filterable by request, so the stacks sampled while one slow trace was executing can be isolated from the rest of the process.

There are two ways to collect. An in-process agent imports into the application, has direct access to span context and to Python's own frame objects, and costs slightly more in the process. An out-of-process agent reads stacks from outside, like py-spy, costs nothing inside the application and must read span context from the process's memory, which is harder and not always supported. Most Python deployments use an in-process agent for the context alone.

Step-by-step implementation

Step 1 — Agree the overhead budget first. An always-on component needs an explicit ceiling, measured rather than assumed. Two percent of a core per process is a common figure; the sampling rate and window are then chosen to fit it. Measuring before and after on a representative service is the only way to know, because the cost varies with stack depth and thread count. Profiling overhead budgets and safety covers the measurement.

Step 2 — Configure the agent with the fleet's resource attributes. Using the same attributes as traces and metrics is what makes the signals join. A profile labelled with a different spelling of the service name is a profile nobody will find from a trace.

# profiling_setup.py — sketch of an in-process agent's configuration
import os

PROFILER_CONFIG = {
    "service": os.environ["OTEL_SERVICE_NAME"],
    "version": os.environ.get("SERVICE_VERSION", "unknown"),
    "environment": os.environ.get("ENVIRONMENT", "dev"),
    "sample_rate_hz": 100,
    "upload_period_s": 60,
    "include_span_context": True,
    "endpoint": os.environ.get("PROFILER_ENDPOINT", "http://localhost:4040"),
}

Step 3 — Record span context with each sample. The sampler reads the current span for the thread it is sampling, which is available because the tracing SDK stores it in a context variable per thread. With it, the profile store can filter by trace identifier, span identifier, or any attribute propagated to the span — route, tenant, feature flag. The mechanics are in linking profiles to traces with span context.

Step 4 — Deploy to one service and verify the overhead. Before a fleet rollout, run one service with the agent for a day and compare its CPU, latency and memory against the same service without it. The numbers should be within the budget from step 1; if they are not, the sampling rate is the first thing to reduce.

# the agent's overhead, compared across otherwise identical deployments
avg by (profiling) (rate(process_cpu_seconds_total{service="checkout"}[1h]))

Step 5 — Compare by version after every deploy. The highest-value routine use: diff the new version's aggregated profile against the previous one for the same service, and flag any frame whose share grew beyond a threshold. This finds regressions that affect CPU without yet affecting latency, which is the period during which they are cheapest to fix.

Step 6 — Set retention like any other telemetry. Recent profiles at full resolution for a week or two, older ones aggregated to hourly or daily for a longer period. The value of old profiles is almost entirely comparative, so aggregated retention loses little.

A final architectural point concerns where the profile data goes. Sending it to the same backend as traces, where one exists that accepts profiles, makes the span-to-profile link a native feature rather than an integration. Sending it to a separate profiling store works equally well for the profiles themselves and requires the two systems to share identifiers — the trace identifier recorded with each sample, and the service and version attributes on each profile — for the link to be constructed. Either is fine; what matters is that the identifiers match, because a profile that cannot be reached from a trace is a profile almost nobody will look at.

Configuration reference

Async and concurrency considerations

Continuous profiling interacts with Python's concurrency models in ways that affect both what it shows and what it costs.

For threaded services the sampler walks every thread's stack on each tick, so its cost rises with thread count. A service with two hundred threads — common with generous thread pools — pays noticeably more than one with twenty. Excluding idle threads, or threads known to be uninteresting such as exporter threads, keeps the cost proportional to the work that matters.

For asyncio services the picture is different: there is one thread running the loop, and every coroutine's stack hangs beneath the loop's own frames. A sampler that understands tasks can attribute samples to the coroutine that was running, which is far more readable than a profile dominated by loop machinery. Span context is particularly valuable here, because it is the only reliable way to attribute a sample to a request when the stack itself is shared by every request.

For prefork servers, each worker process runs its own sampler. That multiplies the per-process overhead by the worker count, which is worth including in the budget, and it means each worker's profile is labelled with its own instance identity. Aggregating across workers for a service-level view is the default query; separating them is how uneven load or a single misbehaving worker becomes visible.

Wall-clock sampling deserves a specific note. It shows blocked time, which is often the more useful view for a service bottlenecked on I/O, and it costs more because every thread is sampled regardless of state. Running on-CPU continuously and enabling wall-clock selectively — per service, or during investigations — is a common compromise.

Production code examples

A post-deploy check that compares the new version's profile with the previous one and reports frames that grew:

# profile_regression.py — run by the deploy pipeline after rollout settles.
import collections
import sys

import requests

STORE = "http://profiles.observability.svc:4040"
THRESHOLD = 0.02          # flag frames whose share grew by more than two points


def frame_shares(service: str, version: str, window: str = "30m") -> dict[str, float]:
    resp = requests.get(f"{STORE}/api/v1/profile", params={
        "service": service, "version": version, "window": window,
        "format": "folded"}, timeout=30)
    resp.raise_for_status()
    counts = collections.Counter()
    for line in resp.text.splitlines():
        stack, _, n = line.rpartition(" ")
        counts[stack.split(";")[-1]] += int(n)
    total = sum(counts.values()) or 1
    return {frame: n / total for frame, n in counts.items()}


def main(service: str, old: str, new: str) -> int:
    before, after = frame_shares(service, old), frame_shares(service, new)
    regressions = sorted(
        ((after.get(f, 0) - before.get(f, 0), f) for f in set(before) | set(after)),
        reverse=True)
    flagged = [(d, f) for d, f in regressions if d > THRESHOLD]
    for delta, frame in flagged:
        print(f"REGRESSION {frame}: {before.get(frame, 0):.1%} -> {after.get(frame, 0):.1%}")
    return 1 if flagged else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:4]))

Expected Output: a flagged frame and a non-zero exit, so the pipeline can require acknowledgement.

REGRESSION parse: 2.9% -> 8.9%

A query pattern that goes from a slow trace to the code that ran inside it — the capability that justifies recording span context:

# from a trace identifier to the stacks sampled during it
resp = requests.get(f"{STORE}/api/v1/profile", params={
    "service": "checkout",
    "trace_id": "9f2a71c4f0b84c2e9d5f1a7b3c8e6d02",
    "format": "folded",
}, timeout=30)
top = sorted((int(l.rpartition(" ")[2]), l.rpartition(" ")[0].split(";")[-1])
             for l in resp.text.splitlines())[-3:]
for count, frame in reversed(top):
    print(f"{count:5d}  {frame}")

Expected Output: the stacks from inside one slow request, rather than from the whole process.

   41  parse
   12  to_representation
    3  execute

Making it part of the workflow

Continuous profiling that nobody opens is an overhead with no return. Three habits make it pay.

Link from traces to profiles. A trace viewer that offers "show profile for this span" turns profiling from a specialist activity into something any engineer does while reading a slow trace. Most of the value arrives the first time somebody clicks through from a span that is inexplicably slow and sees the stack that explains it.

Diff after every deploy, automatically. The regression check above takes minutes to set up and runs forever. It finds the class of problem that is otherwise found at the next traffic peak, and it attributes it to a specific release rather than to "something recently".

Keep a monthly baseline per service. Aggregated profiles over a month, per service, form the reference that makes any individual profile interpretable. A frame's share means little in isolation and a great deal compared with its share last month.

What does not work is treating continuous profiling as an incident tool only. By the time an incident is declared the relevant minute has passed, and the value comes from the profile that was taken before anyone knew it was needed — which is exactly what the always-on arrangement provides, and exactly what is wasted if nobody looks until something is on fire.

What to do with the data on an ordinary day

Most discussion of continuous profiling focuses on incidents, and most of its value arrives on days when nothing is wrong. Four routine uses account for the bulk of it.

Cost attribution. A fleet-wide profile aggregated by service and then by top-level frame answers "where does our CPU spend actually go", which is a question finance asks about infrastructure and engineering usually cannot answer. The answer is frequently surprising — a logging formatter, a serialisation library, a retry loop — and it directs optimisation effort towards changes that reduce the bill rather than towards whatever the team last found interesting.

Library upgrade review. Comparing profiles before and after a dependency upgrade shows whether the new version is cheaper or more expensive in this service's actual usage, which benchmarks published by the library's authors cannot. An upgrade that makes a common call twenty percent more expensive is worth knowing about before it reaches every service.

Capacity planning. CPU per request, broken down by frame, says which parts of the service will dominate as traffic grows. A service where one serialisation frame is forty percent of CPU will hit its scaling limit on that frame first, and knowing that in advance turns a capacity crunch into a planned optimisation.

Onboarding. A profile of a service under normal load is one of the fastest ways for a new engineer to understand what a service actually does, as opposed to what its documentation says. The widest frames are, almost by definition, the code paths that matter.

Choosing between agents

The choice between an in-process and an out-of-process agent is less about quality than about trade-offs that differ by service.

An in-process agent is imported by the application, runs a sampling thread inside it, and reads frames and span context directly. It has the richest context, the simplest correlation with traces, and a small cost inside the process that the application's own metrics will show. Its risk is that it is code running inside your service: a bug in the agent, or an incompatibility with a new interpreter version, becomes a bug in your service.

An out-of-process agent reads the application's memory from outside, as py-spy does. It adds nothing to the application's dependencies and cannot crash it. It needs elevated permissions to read another process's memory, and reading span context from outside is considerably harder — some agents support it for specific interpreter versions, others not at all.

A reasonable default is in-process for services where per-request attribution matters, which is most of them, and out-of-process for services where adding a dependency is costly or the interpreter version is unusual. The overhead is comparable either way, so the decision rests on context versus isolation rather than on performance.

Common mistakes

No overhead budget. Error signature: a fleet-wide rollout followed by a CPU increase nobody expected. Root cause: the cost was assumed rather than measured. Remediation: agree a ceiling, measure on one service, then roll out.

Different labels from traces. Error signature: profiles that cannot be found from a trace. Root cause: the profiler's service name or version spelled differently from the tracing resource. Remediation: drive both from the same environment variables.

No span context. Error signature: whole-process profiles that cannot answer questions about one endpoint. Root cause: context recording disabled. Remediation: enable it; per-request filtering is most of the value.

Wall-clock mode everywhere. Error signature: overhead well above budget on services with many threads. Root cause: every thread sampled on every tick. Remediation: run on-CPU continuously and wall-clock selectively.

Retaining full resolution forever. Error signature: a profiling store whose cost grows steadily and whose old data nobody queries in detail. Root cause: no aggregation policy. Remediation: keep recent profiles at full resolution and roll older ones up to hourly or daily, since their value is comparative.

Profiling only a canary. Error signature: a regression visible fleet-wide that the canary's profile did not show. Root cause: the canary received a traffic mix unlike production's. Remediation: profile across the fleet at a low rate, rather than one instance at a high one.

Nobody opens it. Error signature: a profiling bill with no investigations to show for it. Root cause: profiles are not linked from the places engineers already look. Remediation: link from traces and automate the post-deploy diff.

Frequently Asked Questions