Running Continuous Profiling for Python Services

The argument for continuous profiling is that the profile exists before anybody knows it will be needed. That only holds if the agent is actually running on every instance, costing what was agreed, and labelled well enough to be found. This page covers the deployment details that decide all three: where the agent starts under each process model, which settings matter, how to roll it out, and how to notice when it has quietly stopped. It is a task article under continuous profiling in production, part of the Python profiling and performance observability section.

Prerequisites

The specific agent package depends on the chosen backend; the application requirements are the same for all of them.

pip install "opentelemetry-sdk>=1.27.0,<2.0.0"

export OTEL_SERVICE_NAME=checkout
export SERVICE_VERSION=2026.09.18
export ENVIRONMENT=prod
export PROFILER_ENDPOINT=http://localhost:4040

Implementation

Step 1 — Start the agent where the work happens. For a single-process service, at application startup. For a prefork server, in each worker after the fork — the master's sampler thread is not copied into the workers, so an agent started in the master profiles only the master. For a Celery worker pool, in the worker process initialisation signal, for the same reason. Getting this wrong produces no error at all; it produces profiles of a process that does nothing, which is worse than no profiles because it looks like data.

# gunicorn.conf.py
import os

def post_fork(server, worker):
    # 1. One sampler per worker, created after the fork so its thread exists here.
    from profiling_agent import start
    start(
        service=os.environ["OTEL_SERVICE_NAME"],
        version=os.environ["SERVICE_VERSION"],
        environment=os.environ["ENVIRONMENT"],
        instance=f"{os.environ.get('HOSTNAME', 'local')}-w{worker.age}",
        endpoint=os.environ["PROFILER_ENDPOINT"],
        sample_rate_hz=100,
        upload_period_s=60,
    )

Step 2 — Drive every label from the environment the tracer already uses. The profile's service, version and environment must match the tracing resource exactly, or a profile cannot be found from a trace. Reading both from the same variables makes that structural rather than a matter of discipline. The version label in particular is what makes post-deploy comparison possible, so it must change with every release.

Step 3 — Measure one service before extending. Run the agent on one service for a full day — including its peak — and compare CPU per request, latency percentiles and resident memory against the same service without it. The difference is the real overhead, and it should sit inside the budget agreed in profiling overhead budgets and safety.

# CPU per request, with and without the agent, same service
sum by (profiling) (rate(process_cpu_seconds_total{service="checkout"}[1h]))
  /
sum by (profiling) (rate(http_server_requests_total{service="checkout"}[1h]))

Expected Output: a difference of a percent or two.

profiling="off"   0.00412 CPU-seconds per request
profiling="on"    0.00419 CPU-seconds per request   (+1.7%)

Step 4 — Extend by value, not alphabetically. The services investigated most often, the ones with the most complex code paths and the ones with the most expensive CPU footprint return the most from profiling. Starting there means the rollout pays for itself early, and the evidence it produces makes the case for the rest.

Step 5 — Confirm the agent never blocks the application. Stop the profiling backend in staging and verify the application's latency does not move. A well-behaved agent buffers briefly and drops, exactly as a telemetry exporter should; one that blocks on upload is a production risk regardless of its sampling overhead.

Step 6 — Alert on gaps. A service that has not uploaded a profile in several minutes has either never been configured or has an agent that failed. Both look identical to a service that was never profiled, so the only way to notice is to compare the set of services and instances uploading against the set that should be.

# instances expected to profile that have not uploaded recently
count by (service) (up{job="app"} == 1)
  unless on(service)
count by (service) (time() - profile_last_upload_timestamp_seconds < 300)

Settings that matter and settings that do not

Profiling agents expose many options, and only a few of them have much effect on either the value or the cost.

Sample rate matters. It sets both the resolution and the overhead, roughly proportionally. A hundred hertz is the common default and the right starting point; lower rates are reasonable for services with very large thread counts, where the per-tick cost of walking every thread is higher.

Upload period matters less than it seems. Shorter periods give finer time resolution in the store, which helps when correlating with a short incident, at the cost of more requests to the backend. Anything between ten and sixty seconds works; the choice is about the backend's request budget more than about the application.

Wall-clock mode matters a great deal. It shows blocked time, which is often the more useful picture, and it samples every thread regardless of state, which costs more. Running on-CPU continuously and enabling wall-clock per service where waiting is the usual question is the common compromise.

Span context matters most of all. It is the difference between a whole-process picture and one that can be narrowed to a request, and it is the capability described in linking profiles to traces with span context. Leaving it off saves a small amount of overhead and removes most of the value.

Stack depth limits rarely matter. Python stacks in web services are deep but not pathologically so, and a default limit of a hundred or more frames almost never truncates anything meaningful. Lowering it to save overhead trades away readability for very little.

Process models other than prefork

The post-fork rule generalises, and each common process model has its own place for the agent to start.

A single-process async server. One process, one loop, one sampler. Start the agent in the application's startup hook, and make sure it records span context, because every coroutine shares the loop thread and the stack alone cannot say which request a sample belongs to.

A Celery worker pool. The pool's child processes are forked from the worker's main process, so the same rule applies: start the agent in the child initialisation signal. A Celery worker also runs long-lived tasks whose profiles are more interesting per task than per minute, and labelling samples with the task name through span context makes that view available.

A process pool inside a service. Work sent to a ProcessPoolExecutor runs in child processes that the main process's agent does not cover. If that work is where the CPU goes — which is usually why it is in a process pool — the pool's initialiser is where a second agent belongs, labelled so its profiles are identifiable as the pool's rather than the service's.

A serverless function. An execution environment is frozen between invocations, so a sampler thread cannot run on a schedule, and uploading at the end of each invocation adds to billed time. Most continuous profiling approaches do not suit this model well, and occasional on-demand profiling of a reproduction is usually the better tool.

In every case the verification from above applies: count the instances uploading and compare with the number of processes doing work. A mismatch always means an agent started in the wrong process.

Configuration options

Verification

Confirm every expected instance is uploading and that the profiles contain the application's own frames rather than only machinery.

# instances that uploaded in the last five minutes, by service
curl -s "http://profiles.observability.svc:4040/api/v1/instances?since=5m" \
  | python3 -c 'import json,sys; [print(s["service"], s["instances"]) for s in json.load(sys.stdin)]'

Expected Output: instance counts matching the deployment's replica and worker counts.

checkout 16
inventory 8
pricing 12

Sixteen instances for a service with four replicas of four workers is correct. Four would mean the agent started in the master of each replica rather than in its workers — the failure from step 1, visible in one number.

Common mistakes

Starting the agent in the prefork master. Error signature: profiles showing only accept loops and supervision. Root cause: the sampler thread does not exist in forked workers. Remediation: start it in a post-fork hook.

Labels that differ from the tracer's. Error signature: no link from traces to profiles. Root cause: the profiler reads a different service name or version. Remediation: drive both from the same environment variables.

Rolling out fleet-wide without measuring. Error signature: a fleet CPU increase discovered on the bill. Root cause: the overhead was assumed. Remediation: measure one service for a full day first.

An agent that blocks on upload. Error signature: application latency rising during a profiling backend outage. Root cause: synchronous upload on a path the application waits for. Remediation: verify drop-on-failure behaviour in staging before rollout.

No gap alert. Error signature: a service discovered months later to have never uploaded. Root cause: a failed agent looks like an unprofiled service. Remediation: compare uploading instances against running instances continuously.

Frequently Asked Questions