Burn-Rate Alerts and Error Budgets
An objective of 99.9 percent availability over thirty days allows one request in a thousand to fail. That allowance — the error budget — is a quantity, and the speed at which it is spent says how urgent a problem is. A service failing one percent of its requests is spending the budget ten times faster than sustainable, and will run out in three days. A burn-rate alert pages on that speed. This article works through the arithmetic, the multi-window alerts built on it, and budget tracking for a Python service. It belongs to SLOs, alerts and dashboards from Python metrics in the Python metrics and instrumentation section.
Prerequisites
SLI recording rules at 5-minute, 30-minute, 1-hour, 6-hour and 3-day windows, as described in defining SLIs from Python request metrics, and the alert routing from writing alert rules for Python services.
The arithmetic
The budget is one minus the objective. For 99.9 percent, it is 0.001 — a tenth of a percent of requests over the window.
The burn rate over a period is the observed error rate divided by the budget. An error rate of 0.5 percent against a budget of 0.1 percent is a burn rate of five.
Time to exhaustion is the window divided by the burn rate. At a burn rate of five, a thirty-day budget lasts six days.
Budget spent in a period is the burn rate times the period divided by the window. At a burn rate of 14.4 for one hour, the fraction spent is 14.4 × 1 ÷ 720 = 0.02, two percent.
That last relation is how thresholds are chosen. Rather than picking burn rates directly, decide what spending would justify each response, and derive the rate:
| Response | Budget spent | In | Burn rate | Long window | Short window |
|---|---|---|---|---|---|
| Page | 2 % | 1 hour | 14.4 | 1 h | 5 m |
| Page | 5 % | 6 hours | 6 | 6 h | 30 m |
| Ticket | 10 % | 3 days | 1 | 3 d | 6 h |
The first row catches severe incidents fast. The second catches sustained moderate ones that would never reach 14.4 but still spend a meaningful fraction of the budget within a working day. The third catches slow leaks — a burn rate just above one, spending the budget a little faster than sustainable — which are not urgent but will breach the objective if nobody looks.
Why one window is not enough
An alert on the one-hour error rate alone has two problems. It is slow to clear: after a ten-minute outage is fixed, the one-hour rate stays above threshold for up to fifty more minutes, and the alert keeps firing while the responder is trying to confirm the fix. And it is slow to fire on sudden total failures, since the hour must accumulate enough errors.
An alert on the five-minute rate alone has the opposite problem: it fires on every short spike, including those that spend a negligible fraction of the budget.
Requiring both — the long window over the threshold, and the short window over the same threshold — combines their strengths. The long window says the problem is significant. The short window says it is still happening. When the problem stops, the short window clears within minutes and the alert resolves, even though the long window is still elevated. The short window is conventionally one twelfth of the long one.
Implementation
Recording rules for the error ratio at each window — sli:orders_api:availability:errors_rate5m and its siblings, each one minus the availability ratio — then one alert per row of the table:
groups:
- name: orders-api-burn
rules:
- alert: OrdersApiBudgetBurnPage
expr: |
(
sli:orders_api:availability:errors_rate1h > (14.4 * 0.001)
and sli:orders_api:availability:errors_rate5m > (14.4 * 0.001)
) or (
sli:orders_api:availability:errors_rate6h > (6 * 0.001)
and sli:orders_api:availability:errors_rate30m > (6 * 0.001)
)
for: 1m
labels: {severity: page, service: orders-api}
- alert: OrdersApiBudgetBurnTicket
expr: |
sli:orders_api:availability:errors_rate3d > (1 * 0.001)
and sli:orders_api:availability:errors_rate6h > (1 * 0.001)
labels: {severity: ticket, service: orders-api}
The same structure applies to the latency SLI, with its own budget — for a 99 percent latency objective, the budget is 0.01, and the thresholds multiply that instead.
Tracking the budget remaining
Alerts say the budget is being spent fast. The dashboard should also say how much is left. Over a thirty-day rolling window:
- record: slo:orders_api:availability:budget_remaining
expr: |
1 - (
(1 - (sum(increase(http_requests_total{service="orders-api",status!~"5.."}[30d]))
/ sum(increase(http_requests_total{service="orders-api"}[30d]))))
/ 0.001
)
A value of 0.62 means sixty-two percent of the budget remains; a negative value means the objective has been breached over the window. Evaluating a thirty-day increase is expensive, so this rule is best evaluated every few minutes rather than every thirty seconds, in a group with a longer interval.
Low-traffic services
Burn-rate alerts assume enough events that a ratio is meaningful. A service handling one request a minute has five requests in a five-minute window; a single failure is a twenty percent error rate and a burn rate of two hundred. Three approaches help. Adding a minimum count to the expression — and sum(increase(http_requests_total[1h])) > 100 — suppresses alerts on too little data. Longer windows accumulate more events. Synthetic probes, running every few seconds, give the denominator a floor and double as an end-to-end check. For internal Python services with sporadic traffic, a combination of the minimum count and probes works well.
A budget policy the team agrees in advance
The budget is only useful if it changes what the team does. A budget policy, written and agreed before it is needed, says what happens at each level of remaining budget. Without one, an exhausted budget is a number on a dashboard; with one, it is a decision already made.
The bands and responses vary between teams, and three properties matter more than the specific thresholds. The policy must be agreed by the people who own both feature delivery and reliability, or it will be overridden the first time it is inconvenient. It must be proportionate — an exhausted budget pauses risky changes, not all work. And it should be revisited when the objective changes, because a policy tuned for 99.9 percent is too strict or too loose at other targets.
For Python services, the most common budget spenders are deploys — a bad release, a dependency upgrade that changes behaviour — and dependencies outside the team's control. Annotating the budget graph with deploy markers makes the first visible immediately. For the second, a budget spent mostly by a dependency is an argument for timeouts, retries with backoff and graceful degradation, or for an objective on the dependency itself.
Configuration options
| Setting | Value | Notes |
|---|---|---|
| Budget | 1 - objective |
0.001 for 99.9 % |
| Fast page | 14.4× over 1 h and 5 m | 2 % in an hour |
| Sustained page | 6× over 6 h and 30 m | 5 % in six hours |
| Ticket | 1× over 3 d and 6 h | slow leak |
| Short window | long ÷ 12 | recency |
for |
1 m or none | windows already smooth |
| Budget remaining | 30 d increase ratio |
slower evaluation group |
| Low traffic | minimum count, probes | avoid single-request pages |
Verification
Use promtool test rules with a synthetic series failing 2 percent of requests: the page alert must fire within about five minutes. A series failing 0.3 percent must produce the ticket but never the page. A series with a two-minute burst at 50 percent and otherwise clean must produce neither, which proves the long window filters blips. Then check the budget-remaining rule against a hand calculation over a known input.
Common mistakes
Thresholds chosen as error rates. Error signature: the same alert paging constantly on one service and never on another. Root cause: a fixed error rate ignores each objective's budget. Remediation: burn-rate multiples of the budget.
A single window. Error signature: pages on blips, or alerts firing long after the fix. Root cause: one window cannot be both significant and recent. Remediation: pair long and short windows.
No slow-burn ticket. Error signature: an objective breached at month end with no alert along the way. Root cause: only fast-burn pages. Remediation: the three-day ticket.
Budget computed from averaged ratios. Error signature: a remaining budget that disagrees with a hand count. Root cause: averaging five-minute ratios weights quiet hours equally. Remediation: divide thirty-day sums of good and valid events.
Ignoring low traffic. Error signature: pages caused by one failed request at night. Root cause: tiny denominators. Remediation: a minimum count or synthetic probes.
No budget policy. Error signature: an objective breached for three consecutive months with no change in how the team works. Root cause: the budget measured but not connected to decisions. Remediation: a written policy with agreed responses per band.
Calendar windows gamed by timing. Error signature: risky changes clustered just after the monthly reset. Root cause: a budget that refills all at once. Remediation: a rolling window, which recovers only as bad events age out.
Frequently Asked Questions
What is a burn rate?
The rate at which the error budget is being consumed, relative to the rate that would exactly exhaust it over the SLO window. A burn rate of one uses the whole budget in the window; a burn rate of ten uses it in a tenth of the window.
Where does the 14.4 come from?
It is the burn rate that spends two percent of a thirty-day budget in one hour: two percent of thirty days is 14.4 hours, and spending that in one hour is 14.4 times the sustainable rate. Two percent in an hour is a common threshold for paging.
Why use two windows per alert?
The long window makes the alert significant — a brief spike cannot fill it. The short window makes the alert current — once the problem stops, the short window clears within minutes and the alert resolves, rather than staying on for the length of the long window.
What if my service has very little traffic?
Burn rates become noisy, because a single failure is a large fraction of a small window. Longer windows, a minimum request count in the expression, or synthetic traffic that keeps the denominator meaningful all help.
Does the error budget reset?
With a rolling window, it never resets; it recovers gradually as bad events age out of the window. A calendar window resets at the period boundary, which is simpler to report but can encourage risky changes just after a reset.