Incident Response

Things will break. A data feed will change, a model will drift, a deployment will go wrong at four in the morning. What separates good teams is not that they avoid incidents, but how calmly and quickly they respond, and how much they learn. In this lesson, we prepare, with runbooks, and blameless reviews.

• Incidents are certain
• Prepare before they happen
• Learn without blame

The life of an incident

An incident follows a pattern. First, detection, an alert fires, or a user complains. Second, triage, how serious is it, who is affected, who should lead. Third, mitigation, stop the bleeding first, with a rollback, or a fallback, before searching for the root cause. Fourth, resolution, a real fix. And fifth, learning, a blameless review, that records what happened, why, and what will change. Notice the order. Mitigate first, investigate later.

1. Detectalert or complaint
2. Triageseverity, owner
3. Mitigaterollback or fallback
4. Fix and learnresolution, blameless review

A runbook for a quality alert

A runbook turns panic into a checklist. Here is one, for our accuracy alert. First, confirm the alert, check the dashboard, and the drift report. Second, rule out data problems, because upstream changes are the most common cause. Third, mitigate, by moving the champion alias back to the previous version, or by switching to a rule based fallback. Fourth, fix, by retraining on fresh data, and promoting only if the gate passes. And fifth, learn. Write a short review, and add a test, so this failure can never surprise you again.

# Runbook: churn-api accuracy alert

1. Confirm the alert
   - Open the quality dashboard. Is accuracy below 0.80 for 2 weeks?
   - Check drift report: which features moved? (PSI > 0.25)

2. Rule out data problems
   - Did an upstream pipeline change? Check data validation logs.
   - Missing values or a new category? Fix the source first.

3. Mitigate
   - Roll back: move the "champion" alias to the previous version.
   - Or switch the API to the rule-based fallback.

4. Fix
   - Trigger the retraining pipeline on fresh data.
   - Promote only if the quality gate passes.

5. Learn
   - Write a short post-mortem. Add a test for this failure.

Fallbacks and safe failure

Every model service needs a plan for when the model cannot answer. A fallback might be the previous model version, a simple rule, a cached answer, or a clear message, and a human. Design it before the incident. Decide which is safer, failing open, allowing the action, or failing closed, blocking it. A fraud check may fail closed, and a recommendation box may fail open. Test your fallbacks regularly, because an untested fallback is a rumour.

Summary