Detecting Drift

Drift is the quiet enemy. The data, or the world, slowly changes, and a model that was good yesterday becomes bad. The good news is that you can detect many kinds of drift, even before you know the true answers. In this lesson, we meet the types of drift, and write a real detector.

• The world changes
• Detect it before users complain
• Compare today with training

Types of drift

There are three kinds to know. Data drift, also called covariate shift, is when the inputs change. For example, customers now pay higher prices. Concept drift is when the relationship between inputs and the right answer changes. The same customer, behaves differently, after a competitor launches an offer. And label drift is when the share of each outcome changes, for example, more people churn. Data drift can be seen immediately, from inputs alone. Concept drift is harder, and needs the true answers.

A drift detector

Here is a drift detector. We use two measures. The K S test asks, statistically, whether two samples come from the same distribution, and gives a p value. The population stability index, P S I, measures how much the distribution moved. Below point one is stable, above point two five is a big shift. We compare the training data with a normal week, and with a drifted week, for three features. A feature is flagged, only if both measures agree. The normal week is fine. In the drifted week, charges and calls are flagged, with a huge index, exactly what we expected.

import numpy as np
from scipy.stats import ks_2samp
from churn import make_data

def psi(expected, actual, bins=10):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None)
    return float(np.sum((a - e) * np.log(a / e)))

train = make_data(3000, seed=1)
same_week = make_data(1000, seed=2)
drifted = make_data(1000, seed=3, shifted=True)

for name, week in [("normal week", same_week), ("drifted week", drifted)]:
    print(name)
    for col in ["tenure", "charges", "calls"]:
        p = ks_2samp(train[col], week[col]).pvalue
        score = psi(train[col], week[col])
        flag = "DRIFT" if p < 0.01 and score > 0.1 else "ok"
        print(f"  {col:8s} KS p={p:.4f}  PSI={score:.2f}  {flag}")

normal week
  tenure   KS p=0.9997  PSI=0.01  ok
  charges  KS p=0.1080  PSI=0.01  ok
  calls    KS p=0.9127  PSI=0.00  ok
drifted week
  tenure   KS p=0.2094  PSI=0.01  ok
  charges  KS p=0.0000  PSI=2.33  DRIFT
  calls    KS p=0.0000  PSI=0.68  DRIFT

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Reading the PSI

A quick guide for the population stability index. A value below point one means no meaningful change. Between point one and point two five means a moderate change, worth watching. Above point two five means a major change, and the model should be reviewed. These thresholds are conventions, not laws, so tune them to your problem, and avoid alerting on every tiny wobble. Pair the numbers with a plot, because humans spot patterns that statistics miss.

Summary