Alerting on error budget burn rate

Threshold alerts ask whether something looks bad right now: error rate above 1% for five minutes, p99 latency above 800 ms. They are easy to write and hard to live with. Set the threshold low and on-call is paged for blips no user noticed. Set it high and a slow leak runs for days before anyone looks.

Burn-rate alerts ask a more useful question: at the current rate of failure, will this service miss its reliability objective? The answer tells you both whether to act and how urgently.

From objective to budget

Start with a service level objective (SLO) for something users notice. For instance: 99.9% of checkout requests succeed over a rolling 30 days. The error budget is the complement. In that window, 0.1% of requests may fail without breaking the promise.

Burn rate is how fast that budget is being spent compared with plan:

burn rate = error ratio ÷ (1 − SLO)
 
 0.1% errors, 99.9% SLO  →  1     budget lasts 30 days
1.44% errors, 99.9% SLO  →  14.4  2% of budget per hour

The share of the budget consumed in a window is burn rate × window ÷ SLO period. At a burn rate of 14.4 for one hour, that is 14.4 × 1 h ÷ 720 h = 2%. This is what makes burn rate useful: it converts any error ratio into how much of the month it cost.

Choosing what to measure

The service level indicator should describe the user's experience, not the server's. Two practical rules:

• Count only failures the service caused. That means 5xx responses and timeouts. A 404 for a product that does not exist, or a 401 for a wrong password, is the service working correctly.
• Measure as close to the user as you can. Load balancer or ingress metrics catch failures the application never sees, such as crashed pods and reset connections.

Latency objectives work the same way. "99% of requests complete within 400 ms" counts every slow request as a failed one, and the same alerting applies.

Recording the error ratio

Burn-rate alerts compare error ratios over several windows. Record each ratio once instead of recomputing raw counters in every alert:

groups:
  - name: slo:checkout
    rules:
      - record: slo:error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="checkout"}[5m]))
      - record: slo:error_ratio:rate1h
        expr: |
          sum(rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{service="checkout"}[1h]))
      # ...and the same for 30m, 6h and 3d

Windows of several days are expensive to compute from raw counters. Generators such as Sloth and Pyrra produce the full rule set from a short SLO definition. They are worth adopting once you have more than a handful of objectives.

Multi-window, multi-burn-rate alerts

A single window is never enough:

• A long window is slow to notice a sharp outage, and slow to clear after it ends.
• A short window is noisy.

Pairing them solves both problems. The long window proves the problem is significant. The short window, about a twelfth of its length, proves it is still happening.

Each alert also uses a burn rate that matches its urgency. The thresholds below are the widely used set from Google's SRE Workbook for a 30-day objective:

The fast-burn page in Prometheus:

- alert: CheckoutErrorBudgetFastBurn
  # 14.4x burn: 2% of a 30-day budget in one hour.
  expr: |
    (
      slo:error_ratio:rate1h > (14.4 * 0.001)
      and slo:error_ratio:rate5m > (14.4 * 0.001)
    )
  labels:
    severity: page
  annotations:
    summary: Checkout is burning its 30-day error budget at 14x or more
    runbook: https://runbooks.internal/checkout/error-budget-burn

The six-hour page and the three-day ticket have the same shape, with their own windows and burn rates.

Low-traffic services

Ratios get jumpy when traffic is low. At 20 requests an hour, a single failure is a 5% error ratio and an instant page. There are three common remedies:

• Require a minimum request count in the alert expression before the ratio is trusted.
• Group several small endpoints into one objective, so the denominator means something.
• Add synthetic probes, so there is always enough traffic to measure.

Write the budget policy down

Alerts are half the value. The other half is agreeing in advance what happens when the budget runs out. A common policy is that releases other than reliability fixes pause until the 30-day window recovers.

Without that agreement, the SLO is just a dashboard. With it, reliability work and feature work are traded off using a number both sides accepted beforehand.

What changes for on-call

• Pages correspond to user-visible impact, not to a CPU graph crossing a line.
• Slow burns become tickets, handled in working hours.
• Alerts clear soon after the short window recovers, so stale pages stop piling up.
• The objective that drives alerting also informs planning. When the budget is spent, reliability work takes priority.