Why Models Decay

Six months after launch, your phone rings. HomeWorth's prices are too low, and customers are angry. Nothing has changed in your code. There are no errors in the logs. But the housing market has moved, and your model is still living in the past. This is called drift, and it happens to every model.

• Six months later, complaints
• No bugs, no errors
• The world has moved

Two kinds of drift

There are two main types. Data drift means that the inputs change. Incomes rise, and a new kind of customer shows up. Concept drift is more dangerous, because the relationship itself changes. The same house, with the same features, is now worth more. Your model isn't broken. It's simply out of date. So a deployed model needs regular monitoring, like a patient.

Detecting drift

We can catch data drift by comparing live data against the training data. The Kolmogorov Smirnov test measures how different two distributions are. Here, we simulate a market where incomes have risen by thirty percent, but everything else is unchanged. We test each column. Run it. It flags med inc as drift, and it leaves the others alone. In a real system, this check runs each day, and sends an alert.

import numpy as np
from scipy.stats import ks_2samp
from sklearn.datasets import fetch_california_housing

X = fetch_california_housing(as_frame=True).data
train = X.sample(10000, random_state=0)

live = X.sample(2000, random_state=1).copy()
live["MedInc"] = live["MedInc"] * 1.3

for col in ["MedInc", "HouseAge", "AveRooms"]:
    stat, p = ks_2samp(train[col], live[col])
    print(f"{col:<9} shift {stat:.3f}  {'DRIFT' if p < 0.01 else 'ok'}")

MedInc    shift 0.217  DRIFT
HouseAge  shift 0.011  ok
AveRooms  shift 0.018  ok

The retraining loop

Here's the loop that keeps a model alive. Monitor the inputs and the predictions. When you see drift, or when real outcomes show that errors are rising, retrain on fresh data. Test the new model against the current champion. And deploy it only if it wins. Our M L Ops course teaches you how to automate this entire loop.

1. Monitorinputs and errors
2. Detectdrift or falling accuracy
3. Retrainon fresh data
4. Promoteonly if it wins

Summary