Instrumenting Flask with OpenTelemetry

Flask is WSGI, which makes tracing it structurally simple: one middleware wraps the application, one span covers each request, and everything else hangs off it. This page covers the wiring, the two settings that keep the data useful, and the prefork detail that catches every gunicorn deployment. It builds on instrumenting Python web frameworks, part of the distributed tracing and OpenTelemetry in Python section.

Prerequisites

pip install "flask>=3.0.0,<4.0.0" \
            "opentelemetry-sdk>=1.27.0,<2.0.0" \
            "opentelemetry-instrumentation-flask>=0.48b0,<1.0.0" \
            "opentelemetry-instrumentation-requests>=0.48b0,<1.0.0" \
            "opentelemetry-instrumentation-psycopg>=0.48b0,<1.0.0" \
            "opentelemetry-exporter-otlp-proto-grpc>=1.27.0,<2.0.0" \
            "gunicorn>=21.2.0,<23.0.0"

export OTEL_SERVICE_NAME=orders-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_PYTHON_EXCLUDED_URLS="healthz,readyz,metrics"
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

Implementation

Step 1 — Build the provider first. FlaskInstrumentor obtains a tracer from whatever provider is registered at the time the middleware handles a request; registering afterwards works, but building the provider first removes any ordering doubt.

# observability/tracing.py
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

def build_provider() -> TracerProvider:
    provider = TracerProvider(resource=Resource.create({
        "service.name": os.environ["OTEL_SERVICE_NAME"],
        "service.version": os.environ.get("SERVICE_VERSION", "0"),
        "deployment.environment": os.environ.get("DEPLOY_ENV", "dev"),
    }))
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(insecure=True)))
    trace.set_tracer_provider(provider)
    return provider

Step 2 — Instrument the app and its dependencies.

# app.py
from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from observability.tracing import build_provider

def create_app() -> Flask:
    build_provider()

    app = Flask(__name__)
    FlaskInstrumentor().instrument_app(app)        # one server span per request
    RequestsInstrumentor().instrument()            # outbound HTTP becomes child spans
    PsycopgInstrumentor().instrument(
        enable_commenter=True,                     # trace id into the SQL comment
        commenter_options={"db_driver": True},
    )

    @app.route("/orders/<int:order_id>")
    def get_order(order_id: int):
        return {"id": order_id}

    return app

instrument_app(app) rather than the global instrument() is the form to prefer: it attaches to one application object, which is testable and does not surprise anyone importing the module.

Step 3 — Exclude the probes. A readiness probe every second across twenty replicas is 1.7 million spans a day that answer nothing.

export OTEL_PYTHON_EXCLUDED_URLS="healthz,readyz,metrics"

Excluded requests produce no span at all — not an unsampled one. That is the right treatment for traffic whose only property of interest is whether it returned 200, which a metric already tells you.

Step 4 — Add the span attributes only you know. The instrumentation adds the HTTP semantic conventions; the business dimensions are yours to add.

from opentelemetry import trace

@app.route("/orders/<int:order_id>")
def get_order(order_id: int):
    span = trace.get_current_span()               # the server span created by the middleware
    span.set_attribute("order.id", order_id)
    span.set_attribute("tenant.id", g.tenant_id)

    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("price.calculate") as child:
        child.set_attribute("price.rules_version", RULES_VERSION)
        return compute(order_id)

Attributes on a span are safe at high cardinality — an order ID on a span is fine, the same value as a metric label is not. That asymmetry is the point of controlling label cardinality in Prometheus.

Step 5 — Wire gunicorn's post_fork hook.

# gunicorn.conf.py
def post_fork(server, worker):
    from observability.tracing import build_provider
    build_provider()                               # one provider per worker, after the fork

def worker_exit(server, worker):
    from opentelemetry import trace
    provider = trace.get_tracer_provider()
    if hasattr(provider, "force_flush"):
        provider.force_flush(timeout_millis=3000)  # do not lose the last batch
        provider.shutdown()

gunicorn -c gunicorn.conf.py --workers 4 --bind 0.0.0.0:8000 "app:create_app()"

Configuration options

Verification

curl -s localhost:8000/orders/42 > /dev/null
curl -s localhost:8000/healthz > /dev/null

Expected Output (Collector debug exporter):

Span #0
    Name           : GET /orders/<int:order_id>
    Kind           : Server
    Attributes:
         -> http.request.method: Str(GET)
         -> http.route: Str(/orders/<int:order_id>)
         -> http.response.status_code: Int(200)
Span #1
    Name           : SELECT orders
    Kind           : Client
    Parent ID      : 00f067aa0ba902b7

Two properties to check. The server span's name is the route template with the converter in it, not /orders/42 — one span name per route, not per ID. And there is no span at all for /healthz, which confirms the exclusion is applied rather than merely configured.

Common mistakes

No traces at all under gunicorn

Error signature: the app is instrumented, requests succeed, and the backend has nothing. Root cause: the provider was built in the master before the fork, so no worker has a live exporter thread. Remediation: build it in post_fork, and flush in worker_exit.

Span names contain IDs

Error signature: thousands of distinct span names such as GET /orders/42. Root cause: the span was named from the path rather than from url_rule, usually because a custom middleware created it before Flask matched the route. Remediation: let FlaskInstrumentor own the server span; if you need one earlier, name it from the route after matching and rename with span.update_name.

Health checks dominate the backend

Error signature: the trace backend's storage is mostly two-millisecond probe spans. Root cause: exclusions were never configured; sampling reduces them proportionally, which is not enough. Remediation: set OTEL_PYTHON_EXCLUDED_URLS so they produce no span at all.

Which instrumentations to install

The framework instrumentation gives you a server span and nothing inside it, and a trace with one bar per request answers no question worth asking. The instrumentations that fill it in matter more than the one that starts it.

The database driver is the highest-value addition in almost every service, because database time is where request time usually goes and because the span carries the statement — with parameters stripped — which turns "slow request" into "this query was slow". psycopg, sqlalchemy, pymysql and redis all have instrumentations, and installing the one for your driver takes a line.

The HTTP client is second, for the same reason applied to downstream services: an outbound call span shows which dependency took the time, and it is also the mechanism that propagates trace context onward, so the downstream service's spans join the same trace.

The task queue, where one exists, because a request that queues work and returns tells you very little without the worker's side of it.

The cache, which is lower value per span and occasionally very high value: a cache client instrumentation is what shows that a request made forty cache calls where it should have made one.

from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.instrumentation.celery import CeleryInstrumentor

RequestsInstrumentor().instrument()
PsycopgInstrumentor().instrument(enable_commenter=True)
RedisInstrumentor().instrument()
CeleryInstrumentor().instrument()

The span-count consequence is worth noting: each addition multiplies the volume, and the total is what the BatchSpanProcessor queue has to carry. Instrumenting everything and then sampling is usually the right order, because a sampled complete trace is far more useful than an unsampled partial one.

Manual spans, and where they earn their place

Automatic instrumentation covers the I/O boundaries and nothing about your own logic, which means a request that spends 200 milliseconds in a pricing calculation shows a 200-millisecond gap between two child spans and no explanation. A manual span around the part you suspect is what closes that gap.

The rule of thumb is one span per unit of work a reader would recognise as a step: validate, price, persist, notify. Finer than that produces a waterfall nobody can read; coarser leaves gaps. Attributes on those spans are where domain identity belongs — the order ID, the tenant, the rules version — because span attributes are not subject to the cardinality constraint that governs metric labels.

One more consideration when adding instrumentations: each one is a dependency whose version has to move with the SDK's. The instrumentation packages track the core libraries closely, and a mismatched pair produces an import error at startup rather than a subtle failure — which is the better outcome, and is still a reason to upgrade the whole set together rather than individually.

SQL commenter, and its one caveat

enable_commenter=True on the database instrumentation appends the trace context as a SQL comment, so the database's own slow-query log can be joined to the trace that produced the query. That is a genuinely useful link and it is nearly free.

The caveat is statement-level connection pooling — PgBouncer in transaction or statement mode being the common case — where the comment defeats prepared-statement caching because every statement text is now unique. In that deployment the commenter costs more than it returns, and the option should be off. Anywhere else it is worth having on.

Related

• Instrumenting Python web frameworks — the parent guide: the middleware model across frameworks.
• Instrumenting Django with OpenTelemetry — the same problem in Django's middleware stack.
• Tracing gRPC services in Python — the non-HTTP equivalent.
• Instrumenting Flask with Prometheus metrics — the metrics signal for the same app.
• Exporters and the OpenTelemetry Collector — where these spans go once they leave the worker.

Frequently Asked Questions