Python Observability

Instrumenting Django with Prometheus Metrics

Django's middleware stack makes request metrics straightforward, and two details decide whether the result is useful: which label carries the route, and whether the scrape counts itself. This page covers both, plus database timing through the execute wrapper and the multiprocess behaviour every gunicorn deployment meets. It builds on Prometheus client instrumentation, part of the Python metrics and instrumentation section.

Where the middleware sits, and what it can see there Django's request path drawn from the outside in, with the metrics middleware placed outermost. On the way in, the request passes through the metrics middleware first, then security, session and authentication middleware, then URL resolution, then any view middleware, and finally reaches the view. On the way out it returns through the same layers in reverse. Two consequences are marked. First, because the metrics middleware is outermost, it observes every request including those that a middleware further in short-circuits — an authentication redirect, a rate-limit rejection, a security block — which are exactly the requests a per-view decorator would miss. Second, the route label is only available after URL resolution, which happens inside the metrics middleware's inward call, so it must be read on the way out rather than on the way in; reading it early yields None for every request and collapses the whole metric into one series. outermost, so nothing escapes it metrics middleware — timer starts, in-flight +1 security · session · authentication may short-circuit here — a redirect, a 403, a rate limit URL resolution resolver_match becomes available from here on the view on the way out read resolver_match.route observe latency, count never on the way in reading the route on the way in yields None for every request — one series, and nothing in the code says why
Outermost placement catches the requests other middleware rejects. Reading the route on the way out is what makes the label exist at all.

Prerequisites

pip install "django>=5.0,<6.0" \
            "prometheus-client>=0.20.0,<1.0.0" \
            "gunicorn>=21.2.0,<23.0.0"

export PROMETHEUS_MULTIPROC_DIR=/tmp/django_prom
mkdir -p "$PROMETHEUS_MULTIPROC_DIR" && rm -f "$PROMETHEUS_MULTIPROC_DIR"/*

Implementation

Step 1 — Define the instruments once. Constructing a metric twice raises Duplicated timeseries, and Django settings modules can be imported more than once.

# observability/metrics.py — the single owner of every instrument
from prometheus_client import Counter, Gauge, Histogram

REQUESTS = Counter(
    "django_http_requests_total", "Total requests",
    ["method", "route", "status"],
)
LATENCY = Histogram(
    "django_http_request_duration_seconds", "Request latency",
    ["method", "route"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
IN_FLIGHT = Gauge(
    "django_http_requests_in_progress", "Requests being handled",
    ["method"], multiprocess_mode="livesum",
)
DB_QUERY = Histogram(
    "django_db_query_duration_seconds", "Database query duration",
    ["alias", "operation"],
    buckets=(0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0),
)

Step 2 — Write the middleware. Read the route on the way out, and fold unmatched requests into one value.

# observability/middleware.py
import time
from django.db import connections
from .metrics import REQUESTS, LATENCY, IN_FLIGHT, DB_QUERY

class MetricsMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        started = time.perf_counter()
        IN_FLIGHT.labels(request.method).inc()
        try:
            with connections["default"].execute_wrapper(self._time_query):
                response = self.get_response(request)
        finally:
            IN_FLIGHT.labels(request.method).dec()

        route = self._route(request)
        elapsed = time.perf_counter() - started
        LATENCY.labels(request.method, route).observe(elapsed)
        REQUESTS.labels(request.method, route, response.status_code).inc()
        return response

    @staticmethod
    def _route(request) -> str:
        match = getattr(request, "resolver_match", None)
        return match.route if match is not None else "<unmatched>"   # 404s, static, blocks

    @staticmethod
    def _time_query(execute, sql, params, many, context):
        alias = context["connection"].alias
        operation = sql.split(None, 1)[0].upper() if sql else "UNKNOWN"
        started = time.perf_counter()
        try:
            return execute(sql, params, many, context)
        finally:
            DB_QUERY.labels(alias, operation).observe(time.perf_counter() - started)

Register it outermost — first in the list, since Django processes MIDDLEWARE top-down on the way in:

# settings.py
MIDDLEWARE = [
    "observability.middleware.MetricsMiddleware",     # first = outermost
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    ...
]

The _time_query wrapper labels by operation — SELECT, INSERT, UPDATE — and never by statement. A statement label is unbounded by construction and would also put query text, including bound values, into a metric name.

Step 3 — Serve /metrics outside the middleware stack. A Django view would pass through MetricsMiddleware and count every scrape.

# wsgi.py
import os
from django.core.wsgi import get_wsgi_application
from werkzeug.middleware.dispatcher import DispatcherMiddleware
from prometheus_client import CollectorRegistry, make_wsgi_app, multiprocess

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
django_app = get_wsgi_application()

def metrics_app(environ, start_response):
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)      # merge per-worker files
    return make_wsgi_app(registry)(environ, start_response)

target = metrics_app if os.environ.get("PROMETHEUS_MULTIPROC_DIR") else make_wsgi_app()
application = DispatcherMiddleware(django_app, {"/metrics": target})
A scrape that counts itself The metrics endpoint mounted two ways. As a Django view, a scrape enters through the middleware stack exactly like any other request, so the metrics middleware starts a timer, increments the in-flight gauge, and on the way out records a latency observation and a request count for the metrics route itself. At a fifteen-second scrape interval that is four self-generated requests per minute per replica, permanently, appearing in the totals as a route nobody ever called and adding a latency distribution that reflects exposition rendering rather than any user-facing work. Mounted through a WSGI dispatcher beside the Django application, the scrape never enters the middleware stack at all: it is routed to the client's exposition app by path before Django is involved, so it produces no metrics of its own while still sharing the port, the bind address and whatever network policy protects the service. a scrape every 15 s — does it appear in your own metrics? as a Django view scrape the metrics middleware counted, timed, and in the in-flight gauge 4 requests/minute/replica, forever for a route nobody called as a WSGI mount scrape the exposition app routed by path, before Django is involved no middleware, no self-counting same port, same bind, same network policy the self-counting version is not harmful so much as permanently misleading — a route in every dashboard that represents monitoring and a latency series describing how long it takes to render your own registry
Not harmful, permanently misleading: a route in every dashboard that represents your monitoring rather than your traffic.

Step 4 — Handle gunicorn's workers. Each prefork worker has its own registry unless the multiprocess directory is set.

# gunicorn.conf.py
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)     # retire the dead worker's files

gunicorn -c gunicorn.conf.py --workers 4 --bind 0.0.0.0:8000 wsgi:application

Counters and histograms sum across workers automatically; gauges need an explicit multiprocess_mode, which is why IN_FLIGHT declares livesum. The full mechanism is in Prometheus client instrumentation.

Query timing without touching a single model One Django request's database activity measured through the connection execute wrapper. The wrapper is installed for the duration of the request by the metrics middleware, so every query the request runs passes through it regardless of which model, manager or raw call produced it. The request shown runs four queries: a session lookup, a select for the main object, a second select for related rows, and a commit. Each is timed and labelled by connection alias and by the first word of the statement — SELECT, INSERT, UPDATE, COMMIT — which is a label set with a handful of values rather than one per statement. The alternative labelling, by statement text, is marked as the trap: it is unbounded, and it also places query text, including any bound literal values, into metric labels that are stored and displayed. one request, four queries, no model code touched execute_wrapper active for the whole request — installed by the middleware SELECT session SELECT orders SELECT items COMMIT each observed into django_db_query_duration_seconds{alias, operation} label by operation SELECT · INSERT · UPDATE · COMMIT a handful of values, forever enough to answer "are writes slow?" label by statement one series per distinct SQL string unbounded — and it stores query text including any literal values in it
The wrapper covers every query the request makes, including the ones your code did not write. Labelling by operation is what keeps that affordable.

Configuration options

Option Where Purpose Notes
Middleware position settings.MIDDLEWARE see every request first in the list
resolver_match.route middleware bounded route label read on the way out
<unmatched> fallback middleware 404s and blocks one series, not none
execute_wrapper middleware query timing label by operation, never statement
/metrics mount wsgi.py avoid self-counting DispatcherMiddleware
PROMETHEUS_MULTIPROC_DIR env worker aggregation cleared on boot
multiprocess_mode gauges cross-worker reduction livesum for in-flight
child_exit gunicorn clean dead workers mark_process_dead

Verification

curl -s localhost:8000/orders/1 > /dev/null
curl -s localhost:8000/orders/2 > /dev/null
curl -s localhost:8000/metrics | grep -E 'django_http_requests_total|django_db_query'

Expected Output:

django_http_requests_total{method="GET",route="orders/<int:pk>/",status="200"} 2.0
django_db_query_duration_seconds_bucket{alias="default",le="0.01",operation="SELECT"} 4.0
django_db_query_duration_seconds_count{alias="default",operation="SELECT"} 4.0

Three properties confirm the setup: two different IDs produced one series with the URL pattern as the route, the database queries were measured without any model code being touched, and there is no series at all for the /metrics path — because the scrape never entered Django.

Common mistakes

Every route label is None

Error signature: one series with route="None" covering all traffic. Root cause: resolver_match was read before URL resolution — on the way in rather than the way out. Remediation: read it after get_response returns, and fold the genuinely unmatched requests into one fallback value.

One series per primary key

Error signature: series count grows with traffic and the store starts rejecting writes. Root cause: the label was taken from request.path. Remediation: use resolver_match.route, which is the pattern. The budgeting method is in controlling label cardinality in Prometheus.

Totals fluctuate between scrapes

Error signature: counters go down as often as up, and every value looks about a quarter too small. Root cause: four gunicorn workers with four private registries. Remediation: set PROMETHEUS_MULTIPROC_DIR, render through MultiProcessCollector, and add the child_exit hook.

Beyond request metrics

Request counts and latency are the baseline; the metrics that make a Django service genuinely observable are usually the ones specific to what it does. Four categories are worth adding deliberately.

Business outcomes. Orders placed, payments declined, signups completed — counters incremented at the point the outcome is decided, labelled by outcome and nothing else. These are the metrics anyone outside engineering asks about, and they are also the fastest way to detect an incident that does not manifest as an error: requests succeeding while the thing they were supposed to do stopped happening.

Queue and job state. Where the project uses Celery or a similar worker, the depth of each queue and the age of the oldest waiting item. Depth alone is ambiguous — a deep queue draining quickly is fine — and age is the number that says whether work is actually stuck.

Cache effectiveness. Hits and misses by cache name, which is two counters and answers "is the cache doing anything" without a profiling session. A hit ratio that quietly drops after a deploy is a common and otherwise invisible cause of a latency regression.

External dependency health. Latency and error counters for each outbound service, labelled by the dependency rather than by the endpoint. The tracing signal covers individual calls better; the metric is what an alert can be built on.

ORDERS = Counter("orders_total", "Orders by outcome", ["outcome"])
CACHE = Counter("cache_operations_total", "Cache operations", ["cache", "result"])
QUEUE_AGE = Gauge("queue_oldest_item_age_seconds", "Age of the oldest queued item", ["queue"])
Metric family Labels Answers
Business outcomes outcome did the service do its job
Queue depth and age queue is work getting stuck
Cache hits and misses cache, result is the cache working
Dependency latency and errors dependency which one is degraded

Cardinality discipline in a Django project

Two Django-specific patterns account for most cardinality accidents.

The first is labelling by anything derived from a model instance: a user, an organisation, a product. Each is unbounded in principle and large in practice, and the metric that results grows with the business rather than with the code. The rule is that a label's value set should be enumerable from the source, not from the database.

The second is a per-view metric created inside the view, which produces one instrument per view rather than one instrument with a view label — and, if the view is a class-based view instantiated per request, one instrument per request, which raises Duplicated timeseries on the second one. Instruments belong at module scope in one place, and the view identity belongs in a label.

The one legitimate exception is a small, fixed set that happens to be a model field: a subscription tier with four values, a region with six. Those are enumerable, stable, and worth having — the test is whether adding a row to the table can add a series.

A note on where these live: business counters belong in the same instruments module as the request metrics, not scattered across the apps that increment them. One module means one place to review the label sets, and it avoids the duplicate-registration error that appears the first time two apps declare a metric with the same name.

Serving the endpoint safely

Two details about the exposition endpoint in a Django deployment. It should not be routed through the URL configuration if the WSGI mount is available, for the self-counting reason above. And it should be protected: the exposition names every route, every dependency, every queue and the build version, which is a useful map for anyone probing the service. A network policy, an ingress rule, or a separate port bound to an internal interface all work; what does not work is relying on obscurity, because /metrics is the first path anything scanning will try.

Frequently Asked Questions

Do I need django-prometheus?

No. The official prometheus_client plus about forty lines of middleware gives you request counts, latency, in-flight requests and database timing, with label choices you control. django-prometheus is a reasonable convenience wrapper, and writing the middleware yourself keeps the route label — the one that decides your cardinality — under your own review rather than a library's defaults.

Where do I get a bounded route label in Django?

request.resolver_match.route, which is the URL pattern string such as orders// rather than the concrete path. It is set after URL resolution, so it is available in the response phase of middleware and is None for requests that never matched — 404s, static files, and anything a middleware short-circuited — which must be folded into a single fallback value.

Why should the metrics endpoint not be a Django view?

Because a Django view passes through your middleware, so every scrape counts itself: a fifteen-second scrape interval adds four requests a minute to the totals for a route nobody called. Mounting the client's WSGI app beside Django through a dispatcher keeps the exposition outside the middleware stack entirely, and inherits the same port and network policy.

How do I measure database time without instrumenting every query?

Use Django's database instrumentation hook, connection.execute_wrapper, which wraps every query executed inside its context and gives you the duration without touching any model code. Register it in middleware so the wrapper is active for the request's lifetime, and label by alias and operation rather than by statement, since a statement label is unbounded.

Related Guides