Python Observability

Zero-Code Instrumentation with opentelemetry-instrument

The fastest way to get traces from an existing Python service is to change nothing in its code: install the OpenTelemetry distro and the instrumentation packages for the libraries it uses, and prefix its start command with opentelemetry-instrument. The launcher configures the SDK from environment variables and patches each supported library before the application imports it, so the web framework, database driver and HTTP clients produce spans immediately. This page covers what the launcher actually does, configuring it by environment, the prefork problem that catches most production deployments, and where automatic instrumentation needs a manual complement. It is a task article under OpenTelemetry SDK setup, part of the distributed tracing and OpenTelemetry in Python section.

What happens before your code runs The launcher's sequence is drawn in four steps. First it reads environment variables and configures the tracer, meter and logger providers, exporters, sampler and propagators. Second it finds every installed instrumentation package through Python's entry points and calls each one's instrument method, which patches its target library — the web framework, the database driver, the HTTP client, the cache client. Third it executes the original start command as if it had been run directly. Fourth, when the application imports those libraries, it receives the patched versions, and every request, query and outbound call produces a span without any application code referring to OpenTelemetry. The note records that the ordering is the whole mechanism: libraries imported before patching are not instrumented, which is why the launcher must wrap the process start rather than be imported partway through. opentelemetry-instrument gunicorn app:app 1 · configure SDK from OTEL_* env providers, exporters 2 · patch libraries every installed instrumentation 3 · run the command exactly as before 4 · app imports patched versions spans appear the ordering is the whole mechanism a library imported before patching is not instrumented which is why the launcher wraps the process start instead of being imported partway through and why step 1 happening in a prefork master matters so much
Everything the launcher does happens before the application's first import. That timing is both why it works without code changes and why prefork servers need extra care.

Prerequisites

pip install "opentelemetry-distro>=0.48b0,<1.0.0" \
            "opentelemetry-exporter-otlp>=1.27.0,<2.0.0"

# add instrumentation packages matching the libraries already installed
opentelemetry-bootstrap -a install

Implementation

Step 1 — Install the distro and let bootstrap choose instrumentations. The distro provides the launcher and sensible defaults. opentelemetry-bootstrap inspects the installed packages and installs the matching instrumentation for each — Flask, Django, FastAPI, psycopg, SQLAlchemy, requests, httpx, redis and many more. Running it in the image build, after the application's dependencies are installed, keeps the two in step.

Step 2 — Wrap the start command. The service's command is unchanged apart from the prefix. The launcher configures everything, then hands over to the original command.

opentelemetry-instrument uvicorn myservice.app:app --host 0.0.0.0 --port 8000

Step 3 — Configure through the environment. Everything the launcher needs comes from environment variables — the same set every OpenTelemetry SDK understands, so the configuration reads the same across languages — which makes the same image correct in every environment and keeps telemetry configuration in the deployment rather than in code.

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="service.version=2026.09.18,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.10
export OTEL_PROPAGATORS=tracecontext,baggage
export OTEL_PYTHON_LOG_CORRELATION=true            # trace ids on stdlib log records
export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis  # too chatty for this service

Expected Output: spans from libraries the application never mentions OpenTelemetry in.

GET /orders/{order_id}                 SERVER   212 ms
  SELECT orders                        CLIENT    41 ms   db.system=postgresql
  GET https://pricing.internal/quote   CLIENT   138 ms   http.response.status_code=200

Step 4 — Handle prefork servers explicitly. Under Gunicorn the launcher configures the SDK in the master. The master then forks, and the batch span processor's export thread exists only in the master — workers create spans into a queue nothing drains. The reliable fix is to re-initialise the tracer provider in a post-fork hook, which also gives each worker its own instance identifier. The same concern and fix appear in Gunicorn and Uvicorn worker logging.

# gunicorn.conf.py
def post_fork(server, worker):
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk.resources import Resource

    provider = TracerProvider(resource=Resource.create())     # env-driven, per worker
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)

Step 5 — Add manual spans where only the application knows the meaning. Automatic spans describe library calls: a SELECT, a POST. They cannot describe what those calls were for. A few manual spans — around "apply discount rules", "reserve stock", "calculate shipping" — turn a trace from a list of queries into a description of what the service did, and they are cheap to add because the SDK is already configured.

from opentelemetry import trace
tracer = trace.get_tracer("checkout")

def place_order(cart):
    with tracer.start_as_current_span("place order") as span:
        span.set_attribute("order.item_count", len(cart.items))
        with tracer.start_as_current_span("reserve stock"):
            reserve(cart)
        with tracer.start_as_current_span("charge payment"):
            charge(cart)
Library spans and business spans The same request is shown twice. With automatic instrumentation alone, the trace shows the server span and beneath it seven client spans: three SELECT statements, two UPDATE statements, one outbound POST and one cache call. Every span is correct, and nothing says what the service was trying to do; an engineer must infer from the statements that this was placing an order, reserving stock and charging a card. With three manual spans added — place order, reserve stock, charge payment — the same library spans are grouped under named business operations, and the trace reads as a description of the work, with the time for each business step visible directly. The note records that automatic instrumentation provides the breadth and a handful of manual spans provides the meaning, and that the two are complementary rather than alternatives. the same request, before and after three manual spans automatic only POST /orders SELECT, SELECT, UPDATE, POST, UPDATE — correct, and silent about purpose plus three manual spans POST /orders place order reserve stock charge payment the same library spans, grouped by meaning automatic instrumentation provides breadth; manual spans provide meaning the launcher gets a service traced in an afternoon three well-placed manual spans make the trace readable by someone who did not write the code
Library spans say what was called. Business spans say what was being done. A trace needs both to be read quickly by someone unfamiliar with the code.

Where zero-code instrumentation stops

The launcher is an excellent starting point and has limits worth knowing before relying on it entirely.

It only covers libraries with instrumentation packages. Most popular frameworks, drivers and clients are covered. An internal HTTP client wrapper, a less common database driver, or a library that talks to its backend over a custom protocol produces no spans. The trace then has gaps exactly where those libraries were used.

It can be too broad. Some instrumentations produce a span per call, and some calls happen thousands of times per request — cache lookups, fine-grained ORM operations. The resulting traces are large and expensive, and the useful spans are buried. Disabling specific instrumentations, or tuning them through their own options, is part of adopting the launcher in production rather than an afterthought.

It configures once, at start. Anything that needs to differ per request, per tenant or per operation — sampling for a debug flag, extra attributes from application context — needs a span processor or code, not environment variables. The patterns in using baggage for tenant and feature context are one way to add that without touching business logic.

It hides the configuration. A service configured entirely by the launcher has no code that says what its telemetry setup is. That is convenient and occasionally confusing: an engineer debugging missing spans may not realise the SDK was configured at all, or by what. Documenting the environment variables alongside the service, or moving to an explicit configuration module once the setup stabilises, keeps it understandable. Debugging missing spans in Python covers diagnosing the cases where something went wrong.

Many teams follow a natural progression: start with the launcher for immediate coverage, add manual spans for business operations, and eventually move the SDK configuration into code where it can be tested and reviewed, keeping the automatic library instrumentation throughout.

Rolling it out across a fleet

The launcher's appeal is strongest at fleet scale, where it can instrument dozens of services without dozens of code changes. A few practices make that rollout go smoothly.

Put it in the base image. A shared base image that installs the distro, runs bootstrap after the service's dependencies, and sets the start command's prefix means every service built on it is instrumented by default. Services opt out rather than in, which is the direction that produces consistent coverage.

Set the fleet-wide environment in one place. Exporter endpoint, propagators, sampler and log correlation are the same for most services and belong in a shared deployment template. Only the service name and version differ per service, and those come from the build. Keeping the shared settings centralised means a change to the collector endpoint or the sample rate is one edit, not forty.

Start with a low sample rate and raise it deliberately. Automatic instrumentation of an unfamiliar service can produce more spans per request than anyone expects, particularly with ORM and cache instrumentation enabled. A rollout at a few percent, followed by a look at spans per request per service as described in estimating telemetry volume from a Python service, shows which services need instrumentations disabled before the rate is raised.

Verify each service once. A single request through each service after rollout, checked for a connected trace with the expected library spans, catches the prefork problem, missing instrumentation packages and propagation gaps between services. Services that fail the check are usually few and each has one of those three causes.

Rolling it out across a fleet Four stages for rolling out zero-code instrumentation across many Python services. First, one low-risk service runs under opentelemetry-instrument in staging, with the console exporter, to confirm spans appear and startup time is unaffected. Second, the same service goes to production exporting to the collector, and its latency and error rates are compared with the week before. Third, the wrapper moves into a shared base image or entrypoint so every service gets it on its next deploy, with an environment variable to disable it. Fourth, services add manual spans for business operations on top. The note says the disable switch is what makes the rollout safe. from one service to the whole fleet 1 · one service staging, console exporter check spans and startup 2 · production export to the collector compare latency, errors 3 · base image wrapper in the entrypoint env var to disable 4 · manual spans business operations on top of library spans the disable switch is what makes a fleet-wide rollout safe OTEL_SDK_DISABLED=true turns it off without a rebuild
Start with one service, prove it in production, then move the wrapper into the base image with an off switch.

Configuration options

Variable Example Effect
OTEL_SERVICE_NAME checkout resource identity
OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4317 where spans go
OTEL_TRACES_SAMPLER parentbased_traceidratio head sampling strategy
OTEL_TRACES_SAMPLER_ARG 0.10 the ratio
OTEL_PROPAGATORS tracecontext,baggage cross-service context
OTEL_PYTHON_LOG_CORRELATION true trace ids on log records
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS redis,sqlite3 turn off noisy ones
OTEL_PYTHON_EXCLUDED_URLS healthz,readyz skip health checks

Verification

Confirm which instrumentations loaded and that workers export.

OTEL_TRACES_EXPORTER=console opentelemetry-instrument python -c "
import requests; requests.get('https://example.org')" 2>&1 | grep '"name"'

Expected Output: a span from a library the command never instrumented explicitly.

    "name": "GET",

Common mistakes

Libraries imported before the launcher runs. Error signature: some libraries traced and others not. Root cause: a module imported during an earlier import hook or a site customisation. Remediation: the launcher must wrap the process start.

Relying on the launcher under Gunicorn. Error signature: no spans from workers in production. Root cause: the export thread exists only in the master. Remediation: re-initialise the provider in post_fork.

Instrumentation packages out of step with libraries. Error signature: an instrumentation that silently does nothing after an upgrade. Root cause: bootstrap run against a different dependency set. Remediation: run bootstrap in the image build after dependencies are installed.

Health checks traced at full rate. Error signature: a large share of stored traces are probes against health endpoints. Root cause: framework instrumentation traces every route by default. Remediation: exclude health and readiness paths with the excluded-URLs setting, so the sample budget is spent on real traffic.

Every instrumentation enabled. Error signature: huge traces dominated by cache or ORM spans. Root cause: accepting all defaults. Remediation: disable or tune the noisy ones.

No manual spans at all. Error signature: traces that list queries but do not say what the request was doing. Root cause: treating automatic instrumentation as complete. Remediation: a few manual spans around business operations.

Frequently Asked Questions

What does opentelemetry-instrument actually do?

It runs before your application: it configures the SDK from environment variables — providers, exporters, sampler, propagators — and loads every installed instrumentation package, which patches its target library. Then it runs your command as normal. When your code imports Flask or psycopg, it gets the patched version.

Do I need to change application code?

No, for the libraries covered by instrumentation packages. You change the start command and the environment. Business-level spans — naming what the application is doing rather than which library call it made — still need code.

Does it work with Gunicorn?

It configures the SDK in the master before forking, and the export threads do not survive the fork, so spans created in workers may never be exported. The standard fix is to re-initialise the SDK in a post-fork hook rather than relying on the launcher alone.

How do I disable one instrumentation?

Set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS to a comma-separated list of instrumentation names. Useful when an instrumentation is too noisy — a cache client producing thousands of spans per request, for example.

Related Guides