SLOs, Alerts and Dashboards from Python Metrics

A Python service that exports request counters and latency histograms has the raw material for the most useful alerting there is: alerts that fire when users are affected, and stay silent when they are not. Getting there takes three steps — indicators that measure what users experience, objectives that say how good is good enough, and burn-rate alerts that fire when the objective is at risk. This guide covers all three, and the dashboards that sit around them. It is part of the Python metrics and instrumentation section, with detail in defining SLIs from Python request metrics, writing alert rules for Python services and burn-rate alerts and error budgets.

Prerequisites

A Python service exporting, through either prometheus_client or the OpenTelemetry SDK:

• a request counter labelled by route and status class, and
• a latency histogram whose buckets include each latency threshold used in an objective.

Both are covered in instrumenting FastAPI with Prometheus metrics and the Prometheus client instrumentation topic. A Prometheus-compatible backend evaluates the rules; the examples use PromQL.

Concept and architecture

Symptoms, not causes. The central idea behind SLO-based alerting is that pages should fire on what users experience, not on what might be causing it. High CPU is a cause; a service can run at ninety percent CPU all day with every request fast. A full connection pool is a cause; the pool can be full and queued requests still complete in time. Conversely, a service can fail users while every resource metric looks healthy — a bad deploy returning errors, a dependency timing out, a misconfigured route. Alerting on causes produces both false pages and missed incidents. Alerting on symptoms produces neither, and the cause metrics become what they are best at: diagnosis, once a symptom page has fired.

An indicator is a ratio of good events to valid events. For availability, good events are requests that did not fail because of the service — typically, responses that are not 5xx — and valid events are all requests, perhaps excluding health checks. For latency, good events are requests that completed within a threshold. Both are ratios of counts, so both come from counters, and the latency one comes from the histogram's bucket counter at the threshold. Expressing latency as a ratio rather than a percentile is what makes it aggregatable, alertable and budgetable; a percentile is none of those.

An objective is a target for the ratio over a window. Ninety-nine point nine percent of requests succeed, measured over thirty days. The window matters as much as the target: thirty days is long enough to absorb a bad hour and short enough to reflect the service's current state.

The error budget is what the objective allows to fail. At 99.9 percent, one request in a thousand may fail. Over thirty days at a hundred requests a second, that is roughly 260 000 failed requests. The budget is a quantity to spend — on deploys, experiments, maintenance — and the rate at which it is spent is what alerts watch.

Burn rate is the spending speed relative to sustainable. A burn rate of one spends the budget exactly over the window. A burn rate of 14.4 spends two percent of a thirty-day budget in one hour and exhausts it in about two days. Alerts on burn rate fire according to how fast the objective is being consumed, which is the question a person being paged needs answered.

Step-by-step implementation

Step 1 — Write the indicators as recording rules. Recording rules compute the ratio once, at several window lengths, so alerts and dashboards read a cheap precomputed series.

groups:
  - name: orders-api-sli
    rules:
      - record: sli:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{service="orders-api",status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="orders-api"}[5m]))
      - record: sli:latency_300ms:ratio_rate5m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{service="orders-api",le="0.3"}[5m]))
          /
          sum(rate(http_request_duration_seconds_count{service="orders-api"}[5m]))

The same rules repeat for 30m, 1h, 6h and 3d, the windows the alerts use.

Step 2 — Choose objectives. Start from what the service achieves today, measured over the last month, and set the objective slightly below it. An objective the service already misses pages constantly; one far below its performance never pages at all. Tighten it later, deliberately, when there is a reason to.

Step 3 — Alert on burn rate with two windows. A page when the budget burns at 14.4× over both the last hour and the last five minutes; a ticket at 6× over six hours and thirty minutes. The long window ensures the problem is significant; the short one ensures it is still happening, so the alert resolves promptly after a fix.

      - alert: OrdersApiAvailabilityBurnFast
        expr: |
          (1 - sli:availability:ratio_rate1h) > (14.4 * 0.001)
          and
          (1 - sli:availability:ratio_rate5m) > (14.4 * 0.001)
        labels: {severity: page}

Step 4 — Route cause alerts to tickets. Pool saturation, memory growth, GC pauses and event-loop lag remain valuable. They become tickets or dashboard panels — things to look at in working hours, or during an incident a symptom alert has already opened.

Step 5 — Build the dashboard around the indicators. Top row: each indicator against its objective, and the remaining budget. Below it: traffic, errors, latency distribution and saturation — the four signals that explain a change in the indicators.

Configuration reference

Async and concurrency considerations

The indicators are only as honest as the point where requests are measured. A Python service measuring latency inside the application handler excludes the time a request waited before a worker picked it up — in the Gunicorn backlog, in a saturated thread pool, or behind a blocked event loop. When the service is overloaded, that wait is the largest part of what users experience, and the in-process indicator does not see it.

Two mitigations exist. Measuring at the load balancer or ingress captures the queueing, at the cost of a second metrics source. Measuring in-process but alerting as well on a saturation signal that captures the wait — event-loop lag, thread-pool queue depth, as described in observing thread pool saturation — covers it indirectly. For services where overload is a realistic failure mode, the ingress measurement is the more reliable indicator.

Multi-process servers need aggregated metrics, or the indicators are computed from a random worker's share of traffic and fluctuate meaninglessly. Metrics in multi-process Python servers covers the setup; the ratio itself is robust to missing workers only if every worker's numerator and denominator are both missing, which unaggregated scraping does not guarantee.

Production code examples

A complete PrometheusRule for one service's availability objective, including all the windows the alerts need:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: {name: orders-api-slo}
spec:
  groups:
    - name: orders-api-sli
      interval: 30s
      rules:
        - record: sli:availability:errors_ratio_rate5m
          expr: 1 - (sum(rate(http_requests_total{service="orders-api",status!~"5.."}[5m])) / sum(rate(http_requests_total{service="orders-api"}[5m])))
        - record: sli:availability:errors_ratio_rate1h
          expr: 1 - (sum(rate(http_requests_total{service="orders-api",status!~"5.."}[1h])) / sum(rate(http_requests_total{service="orders-api"}[1h])))
        - record: sli:availability:errors_ratio_rate30m
          expr: 1 - (sum(rate(http_requests_total{service="orders-api",status!~"5.."}[30m])) / sum(rate(http_requests_total{service="orders-api"}[30m])))
        - record: sli:availability:errors_ratio_rate6h
          expr: 1 - (sum(rate(http_requests_total{service="orders-api",status!~"5.."}[6h])) / sum(rate(http_requests_total{service="orders-api"}[6h])))
    - name: orders-api-slo-alerts
      rules:
        - alert: OrdersApiErrorBudgetBurnFast
          expr: sli:availability:errors_ratio_rate1h > 14.4 * 0.001 and sli:availability:errors_ratio_rate5m > 14.4 * 0.001
          labels: {severity: page}
          annotations:
            summary: "orders-api is spending its 30-day error budget at over 14x"
        - alert: OrdersApiErrorBudgetBurnSlow
          expr: sli:availability:errors_ratio_rate6h > 6 * 0.001 and sli:availability:errors_ratio_rate30m > 6 * 0.001
          labels: {severity: ticket}

Expected Output: during a deploy that fails two percent of requests, the fast-burn alert fires within about five minutes of the error rate crossing the threshold, and resolves within five minutes of the rollback.

Dashboards that answer the first question

The first question during any incident is whether users are affected, and how badly. A dashboard built around SLOs answers it in the top row, before anyone has to interpret a graph.

A few layout rules keep such dashboards useful:

• One dashboard per service, with the same layout for every service, so an engineer who knows one knows all of them.
• Budgets as numbers, not only graphs. "Thirty percent over budget" is read in a second; a burn-down line takes longer.
• Breakdowns by route, since most incidents affect a subset of routes, and the route label is what locates them.
• Links from each panel to the traces and logs for the same service and window, using the correlation described in correlating logs, traces and metrics.

What the error budget is for

The error budget is often introduced as an alerting mechanism, and its more important role is as a decision rule. With budget to spare, a team can deploy freely, run experiments and accept some risk; the objective is being met and users are, by the team's own definition, adequately served. With the budget exhausted, the team slows down: fewer risky changes, more reliability work, until the window rolls forward and budget returns.

This turns a recurring argument — ship faster versus be more careful — into a measurement both sides agreed to in advance. It also makes the choice of objective consequential. An objective set too tight exhausts the budget on normal variation and stalls delivery for no user benefit; one set too loose never constrains anything. Revisiting objectives quarterly, against what users actually complained about, keeps them calibrated.

For a Python service specifically, the budget policy often decides practical questions: whether a dependency upgrade can go out on a Friday, whether a new feature flag rollout proceeds to the next stage, whether a performance regression found in profiling needs fixing this sprint. The budget does not answer those alone, and it gives the discussion a shared number.

Deciding what counts as a valid event

Most arguments about an indicator come down to the denominator. Which requests count at all?

Health checks and readiness probes should usually be excluded. They are frequent, cheap and almost always succeed, so including them inflates availability and dilutes the effect of real failures. A service handling ten requests a second, with a probe every second from three sources, has almost a quarter of its traffic from probes. Metrics scrapes should be excluded for the same reason.

Requests rejected deliberately need a decision. A 429 from rate limiting that protects the service from an abusive client is the service working; a 429 from a limit set too low for legitimate traffic is the service failing its users. Many teams count 429s as good by default and track them separately, revisiting if they rise.

Requests cancelled by the client — a user closing the page, an upstream timeout — are neither good nor bad from the service's point of view, but they often indicate latency. Recording them with a distinct status label, rather than letting them appear as errors or vanish, keeps the choice open.

Finally, some routes matter more than others. A checkout endpoint failing is not the same as a settings page failing, even at the same rate. Separate objectives for a handful of critical routes, alongside the service-wide one, keep a problem on an important route from disappearing into the average. The route label on the request counter makes this a matter of adding a selector.

SLOs for work that is not a request

Background jobs, queue consumers and scheduled tasks have users too, and the same approach applies with different events. For a Celery queue, the good event might be a task completing successfully within its freshness target — an email sent within five minutes of the order — and the valid event every task enqueued. For a nightly batch, the good event is the batch finishing by its deadline, and the indicator is a count of days rather than requests.

These indicators come from the metrics described in collecting metrics from Celery workers and logging from cron and batch jobs. The burn-rate arithmetic is identical; the windows are sometimes longer, since a daily job produces thirty events a month and short windows contain none.

Introducing SLOs to an existing service

A service with years of cause-based alerts cannot switch overnight, and trying usually fails: the team loses trust in the new alerts before they are tuned, and the old ones come back. A gradual path works better.

First, compute the indicators and put them on the dashboard, without alerting. A month of history shows what the service actually achieves and whether the indicators agree with the team's sense of how the service has been doing. Disagreements are informative — an indicator that looked fine during a known incident is measuring the wrong thing.

Second, add burn-rate alerts as tickets rather than pages, and compare them with the existing pages for a few weeks. Every time a cause alert pages without a burn alert, that is a candidate for demotion; every time a burn alert fires without a cause alert, that is an incident the old alerts missed.

Third, promote the fast-burn alert to a page and demote the cause alerts it has proved redundant. Keep a small number of cause alerts that predict imminent failure the indicators cannot see yet — a disk about to fill, a certificate about to expire — since those are genuinely urgent and have no user-facing symptom until it is too late.

Common mistakes

Latency objectives on percentiles. Error signature: an objective like "p99 under 300 ms" that cannot be turned into a budget. Root cause: percentiles do not aggregate or subtract. Remediation: the fraction of requests under 300 ms, from the bucket counter.

A threshold between bucket boundaries. Error signature: an indicator that shifts when traffic shape changes. Root cause: interpolation within a bucket. Remediation: a boundary exactly at the threshold.

4xx counted as failures. Error signature: budget spent by clients sending bad requests. Root cause: client errors in the bad-event count. Remediation: count 5xx, and perhaps 429 when the service throttles unfairly.

Paging on causes. Error signature: pages for high CPU on healthy services, silence during real outages. Root cause: cause metrics treated as symptoms. Remediation: tickets for causes, pages for burn rate.

Single-window burn alerts. Error signature: alerts that fire on one bad minute, or that stay firing an hour after the fix. Root cause: one window cannot be both fast and stable. Remediation: pair a long and a short window.

Objectives nobody owns. Error signature: SLOs breached for weeks without action. Root cause: no team committed to the budget policy. Remediation: fewer objectives, each with an owner and an agreed response.

Probes in the denominator. Error signature: availability that barely moves during an outage. Root cause: health checks and scrapes, which almost always succeed, diluting real traffic. Remediation: exclude them by route in the recording rule.

Indicators measured only in-process. Error signature: a latency SLO that stays green while users see timeouts during overload. Root cause: queueing before the handler is invisible to it. Remediation: measure at the ingress as well, or pair the indicator with a saturation alert.

Replacing every alert at once. Error signature: a team that turns the old alerts back on after a month. Root cause: new alerts untuned and untrusted. Remediation: run burn alerts as tickets alongside the old ones first.

Frequently Asked Questions