Python Observability

Defining SLIs from Python Request Metrics

A service level indicator is a measurement of what users experience, expressed as the fraction of events that were good. For a Python web service, the two that matter first are availability — the fraction of requests that did not fail — and latency — the fraction that completed within a threshold. Both can be computed from metrics most services already export: a request counter and a latency histogram. This article shows the queries, the recording rules and the decisions behind them. It belongs to SLOs, alerts and dashboards from Python metrics in the Python metrics and instrumentation section.

Two ratios from two metrics On the left, the request counter http_requests_total with labels route, method and status. Over five minutes the service handled 30 000 valid requests, of which 29 970 were not 5xx. The availability SLI is 29 970 divided by 30 000, or 99.90 percent. On the right, the latency histogram with buckets at 0.1, 0.3 and 1 second. Over the same five minutes, the bucket at 0.3 seconds counted 29 610 requests and the histogram count was 30 000. The latency SLI is 29 610 divided by 30 000, or 98.70 percent of requests under 300 milliseconds. A note says both are ratios of counts, so both sum across instances and average across time correctly, which is what makes them usable as indicators. availability http_requests_total{route, method, status} valid requests (5 m)30 000 not 5xx29 970 SLI = 99.90 % latency ≤ 300 ms http_request_duration_seconds · le 0.1, 0.3, 1 _count (5 m)30 000 _bucket{le="0.3"}29 610 SLI = 98.70 % both are ratios of counts they sum across instances and average across time correctly — a percentile does neither
Availability from the counter's status label; latency from the histogram's bucket at the threshold. Two divisions, and the indicators exist.

Prerequisites

A service exporting a request counter and a latency histogram — through prometheus_client, as in instrumenting FastAPI with Prometheus metrics, or through the OpenTelemetry SDK. A Prometheus-compatible backend to evaluate recording rules.

curl -s localhost:8000/metrics | grep -E '^http_request(s_total|_duration_seconds_bucket)' | head -5

Implementation steps

Step 1 — Check the labels and buckets. The counter needs a status label (or a status_class label with values like 2xx and 5xx) and a route label with templated routes, not raw paths. The histogram needs a boundary at the latency threshold. If the threshold is 300 milliseconds and the buckets are 0.25, 0.5, the indicator will be interpolated — change the buckets first.

Step 2 — Write the availability ratio. Good events over valid events, with health checks and the metrics endpoint excluded from both.

sum(rate(http_requests_total{service="orders-api", route!~"/health|/metrics", status!~"5.."}[5m]))
/
sum(rate(http_requests_total{service="orders-api", route!~"/health|/metrics"}[5m]))

Step 3 — Write the latency ratio. The bucket counter at the threshold over the total count. Because histogram buckets are cumulative, the le="0.3" bucket already counts every request that took 300 milliseconds or less.

sum(rate(http_request_duration_seconds_bucket{service="orders-api", route!~"/health|/metrics", le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{service="orders-api", route!~"/health|/metrics"}[5m]))

A choice hides here: should failed requests count against latency? A 500 returned in ten milliseconds is fast and bad. Most teams measure latency over successful requests only, adding status!~"5.." to both sides, so that failures are counted once, in availability, rather than improving the latency SLI by being quick.

Step 4 — Record at every window the consumers need. Burn-rate alerts use 5 minutes, 30 minutes, 1 hour and 6 hours; dashboards and budget calculations use 30 days. Recording rules compute each once.

groups:
  - name: orders-api-sli
    interval: 30s
    rules:
      - record: sli:orders_api:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{service="orders-api",route!~"/health|/metrics",status!~"5.."}[5m]))
          / sum(rate(http_requests_total{service="orders-api",route!~"/health|/metrics"}[5m]))
      - record: sli:orders_api:latency300:ratio_rate5m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{service="orders-api",route!~"/health|/metrics",status!~"5..",le="0.3"}[5m]))
          / sum(rate(http_request_duration_seconds_count{service="orders-api",route!~"/health|/metrics",status!~"5.."}[5m]))

The same pair repeats with [30m], [1h], [6h]; the 30-day ratio is better computed from the 5-minute series with avg_over_time, which is much cheaper than a thirty-day rate. Strictly, averaging ratios weights quiet periods the same as busy ones; for a precise 30-day figure, record the good and valid rates separately and divide their 30-day sums.

Percentiles do not add up; ratios do Two instances of a service. Instance A handled 9 000 requests with a p99 of 120 milliseconds and 8 991 under 300 milliseconds. Instance B handled 1 000 requests with a p99 of 900 milliseconds and 950 under 300 milliseconds. Averaging the two p99 values gives 510 milliseconds, which is not the p99 of anything — the true fleet p99 is far lower because instance A carries nine times the traffic. Summing the ratios' numerators and denominators gives 9 941 out of 10 000, or 99.41 percent under 300 milliseconds, which is exactly the fleet's value. The note says this is why SLIs are defined as ratios: they are the only form that combines across instances, routes and time windows without distortion. two instances, one fleet instance A · 9 000 requests p99 120 ms · 8 991 under 300 ms instance B · 1 000 requests p99 900 ms · 950 under 300 ms average of p99s = 510 ms not the p99 of anything (8 991 + 950) / 10 000 = 99.41 % exactly the fleet's value ratios combine across instances, routes and windows without distortion — which is why SLIs use them
A percentile describes one population and cannot be merged. A ratio of counts can be summed across any grouping and stays exact.

Checking the indicator against reality

An indicator is a claim about user experience, and the claim can be tested. Plot a month of the availability and latency SLIs next to the incident log. Every incident users noticed should appear as a dip; every dip should correspond to something users noticed, or to something they would have noticed with more traffic.

Mismatches are the useful part. An incident that did not move the indicator means it is measuring the wrong requests — perhaps the failures happened at a load balancer that returned errors before reaching the Python process, or the affected route was excluded, or the failures were 200 responses with an error body. A dip nobody noticed may mean the indicator counts something users do not care about, such as a background polling endpoint, or that it is working as intended and caught a problem early. Each mismatch either fixes the definition or earns the indicator some trust.

Two classes of failure are commonly invisible to in-process indicators. Requests that never reach the process — rejected at the ingress, timed out in a queue, lost to a crashed pod — are not counted at all. And responses that are technically successful but wrong — an empty list instead of the user's orders, a stale cache — look good to any status-code-based indicator. The first is addressed by measuring at the ingress as well; the second needs a correctness check, often a synthetic probe that asserts on content.

Indicators for OpenTelemetry metrics

With the OpenTelemetry SDK, the histogram instrument exports buckets according to the view's explicit boundaries, and a Prometheus-compatible backend exposes them under the familiar _bucket, _count and _sum suffixes. The queries are the same once names are mapped: an instrument named http.server.request.duration typically appears as http_server_request_duration_seconds, and the status code attribute as http_response_status_code.

Two details differ. The default OpenTelemetry bucket boundaries for duration are in seconds and may not include the threshold, so a view setting them explicitly is needed, as in choosing histogram buckets for latency SLOs. And if the SDK exports delta temporality, the backend must accumulate before rate gives meaningful results; delta vs cumulative temporality covers when that happens.

How many indicators, and at what granularity

A service-wide availability SLI and a service-wide latency SLI are the right place to start, and for many services they are enough. The pressure to add more comes from the averaging problem: a route that fails completely but carries one percent of traffic moves the service-wide availability by one percentage point, which may or may not cross the objective, while every user of that route is fully affected.

Per-route indicators solve that and create another problem. A route handling a few requests a minute produces a ratio that swings between zero and one on individual requests, and alerts on it fire on single failures. The practical compromise is a short list of routes that matter disproportionately — checkout, login, the public API's core resource — each with its own indicator and objective, and everything else covered by the service-wide figure.

A related decision is whether to split by client type. A mobile app and a web front end may call the same routes with different expectations, and a partner integration may have contractual latency terms. When different groups of callers have genuinely different requirements, an indicator per group, distinguished by a bounded label such as the calling application, reflects that. When they do not, one indicator is simpler and less noisy.

Whatever the granularity, the indicator's name and definition should be written down next to the recording rule: which requests, which status codes, which threshold, and why. Six months later, when someone asks why a particular incident did not breach the objective, the answer is in that definition, and without it the rule is just a query nobody wants to change.

Failures an in-process SLI cannot see A table of failure types and whether an availability indicator computed inside the Python process records them. An exception returning a 500 from the handler: recorded. A dependency timeout surfaced as a 503: recorded. A request rejected by the load balancer because no pod was ready: not recorded, the process never saw it. A worker killed by the Gunicorn timeout mid-request: not recorded, the request never completed. A 200 response with wrong or empty content: not recorded, the status looks successful. The note says the ingress covers the first two gaps and a synthetic probe that checks content covers the third. failure in-process SLI covered by handler raises → 500 recorded — dependency timeout → 503 recorded — rejected at the load balancer missed ingress metrics worker killed mid-request missed ingress metrics 200 with wrong content missed content-checking probe a process can only count the requests it finished
The process records what reached it and completed. Rejections, killed workers and wrong-but-successful answers need other vantage points.

Configuration options

Decision Common choice Reason
Bad event (availability) 5xx client errors are not the service's
Excluded routes health, readiness, metrics frequent, cheap, always succeed
Latency population successful requests failures counted once
Threshold a bucket boundary exact ratio
Recording windows 5m, 30m, 1h, 6h the burn-rate alert windows
30-day figure sums of good and valid precise budget
Route-level SLIs a few critical routes not every route

Verification

Generate a known mix — say 1 000 requests, 10 of which return 500 and 50 of which sleep past the threshold — against a test instance, then query the 5-minute recording rules:

sli:orders_api:availability:ratio_rate5m
sli:orders_api:latency300:ratio_rate5m

Expected Output: availability at 0.99 and latency near 0.95 — within the rounding of rate over the window. Values that disagree point to a label that is not what it seems: a status recorded as "500" in one place and 500 in another, or a route template that differs from the one in the selector.

Common mistakes

Raw paths as the route label. Error signature: an SLI query that times out, or a series count in the millions. Root cause: identifiers in URLs becoming label values. Remediation: the framework's matched route template.

Threshold between buckets. Error signature: an indicator that moves when the request mix changes. Root cause: interpolation. Remediation: a boundary at the threshold.

Averaging percentiles. Error signature: fleet latency figures that do not match any instance. Root cause: percentiles do not combine. Remediation: ratios of bucket counts.

Fast failures improving latency. Error signature: latency SLI rising during an outage. Root cause: quick 500s counted as fast requests. Remediation: measure latency over successful requests.

An indicator never checked against incidents. Error signature: an SLO that was met during an outage users complained about. Root cause: the indicator measured the wrong thing. Remediation: compare a month of it with the incident log before alerting on it.

Probes left in the denominator. Error signature: availability that barely dips during a real outage. Root cause: health checks and scrapes, which always succeed, outnumbering affected requests. Remediation: exclude them by route on both sides of the ratio.

Status labels with mixed formats. Error signature: a ratio above one, or a good-event count of zero. Root cause: some code paths record "500" and others "5xx" or an integer. Remediation: one helper that records the status label everywhere.

Frequently Asked Questions

Why a ratio instead of p99 latency?

A ratio of fast requests to all requests can be summed across instances, averaged over any window, compared with an objective and turned into an error budget. A percentile can do none of those correctly — percentiles from different instances or windows cannot be combined.

Can I compute SLIs from OpenTelemetry metrics?

Yes. An OpenTelemetry histogram exported to a Prometheus-compatible backend produces the same bucket and count series, and the queries are identical once the metric names are known. With delta temporality the backend has to convert to cumulative first.

Should the availability SLI count 4xx responses as failures?

Usually not. Client errors are mostly caused by the client, and counting them lets a misbehaving client spend the service's budget. Some teams count 429 or specific 4xx codes that indicate a service problem.

How do I handle a route with very little traffic?

Low-traffic routes produce noisy ratios, since one failure out of ten requests is ten percent. Grouping them into the service-wide SLI, or using longer windows for them, avoids alerts driven by single requests.

What if the latency threshold is not a bucket boundary?

Add a boundary at the threshold. A ratio computed from interpolated buckets is an estimate that shifts with the distribution of requests inside the bucket, which makes the indicator drift for reasons unrelated to users.

Related Guides