Watching for Drift

In production, you don't get answers. Nobody labels each apple. So how would you know when accuracy falls? You can't see that directly. But you can watch the inputs. If the images have changed, then you should be worried.

• No labels in production
• You can't see accuracy
• Watch the inputs

A daily drift check

We keep a reference, from when the model was working well. Each day, we compare today's brightness with that reference, using a statistical test. Monday and Tuesday are normal, with small wobbles. On Wednesday, a lamp fails. Run it. Monday, fine. Tuesday, fine. Wednesday, the alert fires, with the average brightness down to a hundred and three. No labels, and we caught the problem the same day. Run this on a few simple numbers. Brightness, sharpness, colour balance, and the model's own predictions.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
reference = rng.normal(120, 12, 500)

for day, level in [("Monday", 120), ("Tuesday", 118), ("Wednesday", 104)]:
    today = rng.normal(level, 12, 100)
    p = stats.ks_2samp(reference, today).pvalue
    print(f"{day:<10} brightness {today.mean():5.1f}  p={p:.4f}  {'ALERT' if p < .001 else 'ok'}")

Monday     brightness 120.0  p=0.9139  ok
Tuesday    brightness 118.1  p=0.5499  ok
Wednesday  brightness 103.1  p=0.0000  ALERT

A monitoring plan

Here's a full plan. Log simple image statistics for every frame. Track the share of each predicted class, because a sudden change is a warning. Have a person check a small random sample each day, as that's your true accuracy. Save the images that the model finds unusual, for your next training round. And set clear rules for when to retrain.

Summary