Python Observability

Writing Alert Rules for Python Services

An alert rule is a promise that a person will be interrupted when, and only when, something needs them. Most alerting problems come from breaking one half of that promise — pages that fire when nothing is wrong, or silence during a real outage. This article covers the rules a Python service needs, the handful of mechanics that decide whether they keep the promise — for durations, missing data, labels, annotations — and how to test them before an incident does. It belongs to SLOs, alerts and dashboards from Python metrics in the Python metrics and instrumentation section.

What pages, what tickets, what is only watched Three columns. Pages, which wake someone: fast error-budget burn on availability, fast budget burn on latency, the service's metrics absent for five minutes, and a disk or certificate within a day of failure. Tickets, handled in working hours: slow budget burn, memory growth trending toward the container limit, rising worker restarts, a dependency's error rate climbing, and certificates expiring within two weeks. Dashboards only, never alerting: CPU usage, GC pause time, event-loop lag, connection pool utilisation and request rate — these are diagnostic signals consulted once a page has fired. A note says the page column should stay short enough to be memorised; if it grows past a handful per service, some of its entries are causes. page availability fast burn latency fast burn metrics absent 5 min disk / cert < 1 day short enough to memorise ticket slow burn memory trending to limit worker restarts rising dependency errors climbing cert < 14 days working hours dashboard only CPU GC pauses event-loop lag pool utilisation request rate diagnosis, once paged if the page column grows past a handful per service, some of its entries are causes
Pages are for user impact and imminent failure. Causes are tickets or dashboard panels — still valuable, just not at three in the morning.

Prerequisites

Recording rules for the service's SLIs, as in defining SLIs from Python request metrics, a Prometheus server evaluating rules, and Alertmanager routing on a severity label. promtool, shipped with Prometheus, for testing.

Implementation steps

Step 1 — Define severities and routes. Two severities are usually enough: page, routed to the on-call rotation, and ticket, routed to a queue. Alertmanager's routing tree matches on the label, so a rule's severity is the only thing that decides whether it wakes someone.

# alertmanager.yml (excerpt)
route:
  receiver: tickets
  routes:
    - matchers: [severity="page"]
      receiver: oncall
      group_by: [alertname, service]
      group_wait: 30s
      repeat_interval: 2h

Step 2 — Write symptom alerts against recording rules. Burn-rate alerts, explained fully in burn-rate alerts and error budgets, become short expressions over precomputed series.

groups:
  - name: orders-api-alerts
    rules:
      - alert: OrdersApiAvailabilityBudgetBurn
        expr: |
          (1 - sli:orders_api:availability:ratio_rate1h) > 14.4 * 0.001
          and
          (1 - sli:orders_api:availability:ratio_rate5m) > 14.4 * 0.001
        for: 2m
        labels: {severity: page, service: orders-api}
        annotations:
          summary: "orders-api is failing {{ $value | humanizePercentage }} of requests"
          dashboard: "https://grafana.internal/d/orders-api"
          runbook: "https://runbooks.internal/orders-api/availability"

Step 3 — Choose for deliberately. Every alert fires at least for after its condition first becomes true. For burn-rate alerts that already use a five-minute window, a short for — one or two minutes — is enough to skip a single bad evaluation. For threshold alerts on noisy series, a longer for avoids flapping. A for longer than users would wait before noticing defeats the alert.

Step 4 — Alert on missing data. Comparisons against missing series return nothing, and nothing never fires. Two rules cover it: one for the scrape target being down, and one for a key series disappearing while the target stays up — a metrics endpoint that serves only some metrics after a bad deploy.

      - alert: OrdersApiMetricsAbsent
        expr: absent(up{job="orders-api"} == 1)
        for: 5m
        labels: {severity: page, service: orders-api}
        annotations:
          summary: "No orders-api target has been scrapeable for 5 minutes"
      - alert: OrdersApiRequestMetricMissing
        expr: absent(rate(http_requests_total{service="orders-api"}[10m]))
        for: 10m
        labels: {severity: ticket, service: orders-api}

Step 5 — Write cause alerts as tickets. Memory growth is a good example: it predicts an out-of-memory kill but has no user impact until then. A predict_linear on the container's working set, as in diagnosing RSS growth in Python containers, gives a ticket hours before the kill.

      - alert: OrdersApiMemoryTrendToLimit
        expr: |
          predict_linear(container_memory_working_set_bytes{container="orders-api"}[6h], 12 * 3600)
          > on(pod) kube_pod_container_resource_limits{container="orders-api",resource="memory"}
        for: 30m
        labels: {severity: ticket, service: orders-api}
An alert's timeline A timeline of a burn-rate alert. At minute zero, a deploy starts failing two percent of requests. The five-minute rate crosses the threshold at about minute two, because the window needs time to fill. Rule evaluation every thirty seconds sees the condition true and the alert enters pending. The for clause of two minutes holds it pending until minute four, when it fires. Alertmanager's group_wait of thirty seconds batches related alerts, and the page arrives at about minute four and a half. The deploy is rolled back at minute twelve; the five-minute window drains and the condition turns false at about minute sixteen, and the resolved notification follows. A note says each delay — window fill, for, group wait — is a setting, and their sum is how long users are affected before anyone knows. 041216 min deploy failing 2 % of requests pending firing page sent · ~4.5 min window fill ~2 min → for 2 min → group_wait 30 s rollback · window drains each delay is a setting — their sum is how long users are affected before anyone knows
Detection time is the window fill, plus the for, plus Alertmanager's grouping wait. Each is tunable; none is free.

Annotations that shorten the incident

The alert's text is the first thing an engineer reads, often on a phone, often half awake. It should answer three questions without a click: what is wrong, how badly, and where to look next. A summary with the current value — "failing 2.3% of requests" rather than "availability SLO violated" — answers the first two. Links to the service's dashboard and runbook answer the third.

A runbook link is only as good as the runbook, and the best runbooks for Python services are short: the dashboard panels to check in order, the two or three most common causes with the signal that distinguishes each, and the safe first actions — roll back the last deploy, scale out, disable a feature flag. They are updated after every incident whose response was slower than it should have been.

Labels carry routing information and should be kept to what routing needs: severity, service, team. Adding high-cardinality labels to alerts — a pod name, a route — splits one incident into many notifications. Those details belong in annotations or in the dashboard the annotation links to.

Testing rules with promtool

Alert rules are code, and untested code fails when it is needed. promtool test rules feeds synthetic series into the rules and asserts on which alerts are firing at which times.

# orders-api-alerts.test.yml
rule_files: [orders-api-rules.yml]
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'http_requests_total{service="orders-api",route="/orders",status="200"}'
        values: '0+980x40'
      - series: 'http_requests_total{service="orders-api",route="/orders",status="500"}'
        values: '0+20x40'
    alert_rule_test:
      - eval_time: 15m
        alertname: OrdersApiAvailabilityBudgetBurn
        exp_alerts:
          - exp_labels: {severity: page, service: orders-api}

promtool test rules orders-api-alerts.test.yml

Expected Output: a pass, and a failure with a diff of expected and actual alerts whenever a rule or a recording rule changes in a way that stops the alert firing.

Unit Testing:  orders-api-alerts.test.yml
  SUCCESS

A second test case with healthy traffic — no 500s — should assert that no alert fires. Together they prove both halves of the promise.

Alerts specific to Python services

The generic rules apply to any service. A few failure modes are particular to how Python services run, and each has a signal worth a ticket, occasionally a page.

Worker timeouts. Gunicorn kills a worker that does not respond within its timeout, and the request it was handling vanishes without an application error. A counter of worker aborts, as described in Prometheus multiprocess mode with Gunicorn, makes these visible. A handful a day is a ticket; a sustained rate usually shows up in the availability burn as well, which pages.

Event-loop blocking. An asyncio service with a blocked loop serves nothing while it is blocked, and its request metrics simply stop moving — no errors, no latency, because nothing completes. Event-loop lag above a few hundred milliseconds for several minutes is a strong ticket candidate, and the latency burn alert catches the user impact.

Stuck background work. Celery queues whose oldest task is aging, or scheduled jobs that have not reported success within their expected interval, have no request traffic to burn a budget. A freshness alert — time since the last successful run exceeding twice the schedule — is the symptom alert for work that is not a request, and it often warrants a page when the work is user-facing.

Export failures. An OpenTelemetry SDK that cannot reach its collector drops spans and metrics silently. The exporter's own failure counters, covered in detecting dropped spans and metrics, deserve a ticket: nothing is wrong for users, and the next incident will be harder to investigate.

Each of these is written the same way as the rules above — a recording rule where the expression is reused, a severity that decides routing, and a unit test that proves it fires.

What a good page contains A table of the parts of a page notification and an example of each done well. The summary names the service and the symptom with its current value: orders-api failing 2.3 percent of requests. The severity says whether to act now: page. The impact says who is affected: checkout for all users. The dashboard link opens the service dashboard at the alert's time window. The runbook link opens the steps for this specific alert. The note says a responder should know what is wrong and where to look before unlocking a laptop. part example summary orders-api failing 2.3% of requests severity page impact checkout, all users dashboard service dashboard at the alert's window runbook steps for this alert, in order the responder should know what is wrong and where to look before opening a laptop
A notification that states the symptom, its size and where to look saves the first ten minutes of every incident.

Configuration options

Element Recommendation Why
Severities page, ticket routing decides who is woken
Page rules burn rate, absent, imminent failure user impact only
for on burn alerts 1–2 min windows already smooth
for on thresholds 5–15 min avoid flapping
Absent alerts target down + key series missing silence is not health
Annotations summary with value, dashboard, runbook answer what, how bad, where
Labels severity, service, team low cardinality
Tests promtool test rules in CI prove fire and no-fire

Verification

Beyond unit tests, fire each page alert once in a staging environment, end to end: inject errors, confirm the page arrives with a working dashboard and runbook link, and confirm it resolves after the injection stops. A rule that passes unit tests but routes to a receiver nobody watches fails silently, and only an end-to-end test finds it.

Common mistakes

Paging on CPU and memory. Error signature: an on-call rotation that ignores pages. Root cause: causes paging without user impact. Remediation: tickets for causes, pages for symptoms.

No absent-data alert. Error signature: a crashed service with a green dashboard. Root cause: no data, no comparison, no alert. Remediation: absent rules for the target and key series.

A for longer than the budget allows. Error signature: pages arriving after the incident is visible on social media. Root cause: a long for stacked on a long window. Remediation: short for on windowed burn alerts.

High-cardinality alert labels. Error signature: one incident producing forty notifications. Root cause: pod or route labels splitting the group. Remediation: route by service; put detail in annotations.

Untested rules. Error signature: an alert that never fired during an incident it was written for. Root cause: a typo in a metric name or a changed label. Remediation: promtool tests for fire and no-fire cases.

Alerts on raw expressions. Error signature: slow rule evaluation and alerts that disagree with the dashboard. Root cause: the same ratio written slightly differently in two places. Remediation: alert and graph from the same recording rule.

No freshness alert for background work. Error signature: a nightly job silently failing for a week. Root cause: no request traffic, so no burn alert. Remediation: time since last success against its schedule.

Frequently Asked Questions

What should a Python service page on?

On error-budget burn for its availability and latency objectives, and on a small number of imminent failures that have no user symptom yet — a disk about to fill, a certificate about to expire. Everything else is better as a ticket.

What does the for clause do?

It requires the expression to stay true for that duration before the alert fires. It suppresses one-off spikes, at the cost of delaying every alert by at least that long, so it should be shorter than the delay users would tolerate.

Why alert on absent metrics?

Because a comparison against a series that does not exist returns nothing, and nothing does not fire. A service that crashes, stops being scraped, or loses its metrics endpoint looks exactly like a service with no errors unless something alerts on the absence.

How do I test alert rules?

promtool test rules takes a file of synthetic input series and expected alerts at given times. It runs in CI and proves each alert fires when it should and stays silent when it should not, without waiting for a real incident.

Should each route have its own alert?

Only a few critical routes. A per-route alert for every route produces many noisy alerts on low-traffic routes; one service-wide burn alert with the route in the dashboard link finds the affected route just as quickly.

Related Guides