Alerting on ML

Monitoring is only useful if the right person notices the right problem at the right time. Alerts do that. But badly designed alerts either wake people for nothing, or stay silent while models fail. In this lesson, you will see how to write alerts, that people trust, including a small simulation of how they behave.

• Alerts tap people on the shoulder
• Noisy alerts get ignored
• Design them carefully

Alert rules

Here are two alert rules, in the Prometheus format. The first watches latency. It computes the ninety ninth percentile, over five minutes, and fires when it exceeds two hundred milliseconds, for ten minutes in a row. It is labelled severity page, so it wakes a person, and it links to a runbook. The second watches drift. It fires when the population stability index of charges stays above point two five, for a whole day. It is only a ticket, because drift needs a decision, not a midnight call. Notice the for clauses, they filter out blips.

groups:
- name: churn-model
  rules:
  - alert: PredictionLatencyHigh
    expr: histogram_quantile(0.99, rate(prediction_seconds_bucket[5m])) > 0.2
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "p99 latency above 200 ms for 10 minutes"
      runbook: "https://wiki.example.com/runbooks/churn-latency"
  - alert: FeatureDriftHigh
    expr: feature_psi{feature="charges"} > 0.25
    for: 1d
    labels:
      severity: ticket
    annotations:
      summary: "charges has drifted: review, maybe retrain"

How an alert behaves

A rule is easier to understand when you watch it work. Here is a tiny simulation, with eleven days of accuracy. The alert fires only if accuracy stays below eighty percent, for three days in a row. And it resolves only when accuracy climbs back above eighty five percent, a gap called hysteresis. Look at the output. Day three dips once, but the next day recovers, so nothing fires. The accuracy stays low, from day five, and the alert fires on day seven. It stays firing, even on day eight, when accuracy is point seven nine, and only resolves on day nine. No flapping, no noise.

# Evaluate an alert rule: fire only if the problem lasts, resolve only when clearly better.
accuracy_by_day = [0.88, 0.87, 0.79, 0.84, 0.78, 0.77, 0.76, 0.79, 0.86, 0.87, 0.88]

THRESHOLD, FOR_DAYS, RESOLVE_ABOVE = 0.80, 3, 0.85
below = 0
firing = False
for day, accuracy in enumerate(accuracy_by_day, 1):
    below = below + 1 if accuracy < THRESHOLD else 0
    if not firing and below >= FOR_DAYS:
        firing = True
        print(f"day {day:2d}  accuracy {accuracy:.2f}  >>> ALERT FIRES (below {THRESHOLD} for {FOR_DAYS} days)")
    elif firing and accuracy >= RESOLVE_ABOVE:
        firing = False
        print(f"day {day:2d}  accuracy {accuracy:.2f}  >>> alert resolved")
    else:
        print(f"day {day:2d}  accuracy {accuracy:.2f}  {'(firing)' if firing else ''}")

day  1  accuracy 0.88
day  2  accuracy 0.87
day  3  accuracy 0.79
day  4  accuracy 0.84
day  5  accuracy 0.78
day  6  accuracy 0.77
day  7  accuracy 0.76  >>> ALERT FIRES (below 0.8 for 3 days)
day  8  accuracy 0.79  (firing)
day  9  accuracy 0.86  >>> alert resolved
day 10  accuracy 0.87
day 11  accuracy 0.88

Rules for good alerts

A few rules keep alerts healthy. Alert on symptoms users feel, not on every internal cause. Require the problem to last, with a for clause. Add hysteresis, so alerts do not flap. Route by severity, pages for urgent problems, tickets for slow ones. Link every alert to a runbook. And review alerts monthly. If an alert fired and nobody did anything, delete it, or fix it. Every alert should either demand action, or not exist.

Summary