Quality and Retraining

Drift in the inputs is a warning. But what matters in the end, is whether the predictions are still right. The trouble is, the true answers often arrive late. A customer who leaves, does so weeks after the prediction. In this lesson, you will measure quality with delayed labels, and decide when to retrain.

• True answers arrive late
• Measure quality over time
• Decide when to retrain

Closing the feedback loop

Here is the feedback loop. The model predicts today, and the prediction is stored, with an identifier. Weeks later, the real outcome arrives, the customer stayed, or left. A job joins the outcomes to the stored predictions, and computes the real accuracy, for each week. Those numbers go onto a dashboard, and into alerts. Until the labels arrive, the drift detector acts as the early warning. Together, they tell you both, something is changing, and, it is already hurting.

1. Predict and storewith an identifier
2. Outcome arrivesdays or weeks later
3. Join and scorereal accuracy per week
4. Alertearly warning plus real damage

Weekly quality report

Here is a weekly quality report, simulated. The world changes gradually, starting in week three. Each week, we score the model on that week's customers, and print the accuracy. If it falls below eighty percent, the report prints an alert. For the first weeks, all is well. Then the accuracy slips, and from week five, the alert fires, every week. This is the moment to retrain, ideally on fresh data, which reflects the new world.

import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from churn import make_data

train = make_data(3000, seed=1)
model = LogisticRegression(max_iter=500).fit(train.drop(columns="churn"), train["churn"])

print("week  accuracy  status")
for week in range(1, 9):
    mix = min(1.0, max(0.0, (week - 3) / 4))
    normal, shifted = make_data(800, seed=10 + week), make_data(800, seed=40 + week, shifted=True)
    batch = pd.concat([normal.sample(int(800 * (1 - mix)), random_state=week), shifted.sample(int(800 * mix), random_state=week)])
    score = accuracy_score(batch["churn"], model.predict(batch.drop(columns="churn")))
    print(f"{week:>4}  {score:.2f}      {'ALERT: retrain' if score < 0.80 else 'ok'}")

week  accuracy  status
   1  0.88      ok
   2  0.87      ok
   3  0.87      ok
   4  0.81      ok
   5  0.73      ALERT: retrain
   6  0.65      ALERT: retrain
   7  0.63      ALERT: retrain
   8  0.62      ALERT: retrain

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

When should you retrain?

There are three common policies. Retrain on a schedule, for example every week. It is simple, and predictable, but may waste effort, or react too slowly. Retrain on a trigger, when drift or quality alerts fire. It is efficient, but needs good monitoring. Or retrain continuously, as data streams in, which is powerful, and complex. Whatever you choose, a retrained model must pass the same gates as any other, and prove it is better than the champion, before it is promoted.

Summary