Alerts worth waking up for

This is a sample post, here so the layout can be checked. Delete content/posts/sample-alerts-worth-waking-for.md and its cover when you are done with it.

Every alerting setup I have inherited had the same shape: hundreds of rules, a channel nobody reads, and an on-call engineer who has learned that most pages resolve themselves. That last part is the real failure. Once a pager is usually wrong, it is ignored exactly when it is right.

Alert on symptoms, not causes

A node at 90% memory is not an incident. It is a fact. It becomes an incident when requests start failing or slowing down, and that may never happen.

The question worth encoding is always the same: is someone having a bad time? Which gives a short list.

• Requests failing beyond the error budget burn rate
• Latency past the point users abandon the page
• Work queued faster than it is drained
• A scheduled job that did not run at all

Everything else is a dashboard. Dashboards are for when you are already looking; alerts are for when you are not.

Burn rate, not thresholds

A fixed threshold like "error rate above 1%" fires on a thirty-second blip and stays quiet through a slow bleed that eats the month's budget. Burn rate asks how fast the budget is going:

groups:
  - name: slo-burn
    rules:
      # Fast burn: 2% of a 30-day budget in an hour. Page.
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="checkout"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: Checkout is burning its error budget 14x too fast
          runbook: https://runbooks.internal/checkout-errors

      # Slow burn: quiet but expensive. A ticket, not a page.
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          (
            sum(rate(http_requests_total{job="checkout",code=~"5.."}[6h]))
            /
            sum(rate(http_requests_total{job="checkout"}[6h]))
          ) > (6 * 0.001)
        for: 30m
        labels:
          severity: ticket

Two rules, two urgencies. The fast one wakes someone. The slow one does not, because nothing is on fire, but it still gets looked at on a Tuesday.

Make the trace the first thing you reach for

An alert that says something is wrong without saying where costs you the first ten minutes of every incident. Instrument once with OpenTelemetry and that gap closes, because the alert can link straight to exemplar traces:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def charge(order_id: str, amount_cents: int) -> Receipt:
    with tracer.start_as_current_span("payment.charge") as span:
        # High-cardinality identifiers belong on the span, not in a metric
        # label, where they would multiply the series count.
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.amount_cents", amount_cents)
        try:
            return gateway.charge(order_id, amount_cents)
        except GatewayTimeout:
            span.set_status(trace.StatusCode.ERROR, "gateway timeout")
            raise

The attribute split is the part people get wrong. Order ids on a metric label produce a series per order and eventually take Prometheus down. On a span they cost nothing and are exactly what you want when debugging one bad checkout.

The test that matters

For every rule that can page a human, answer two questions:

1. If this fires at 3am, is there something a person can do right now?
2. If they do nothing until morning, does it get worse?

If either answer is no, it is not a page. Make it a ticket, or delete it. A smaller set of alerts that are always worth reading beats a complete set nobody trusts.