The Leak in Your Split

A team reports ninety nine percent accuracy from a video dataset. Everyone's thrilled. In production, it's a disaster. What went wrong? Frames from the same video were in both training and test. The model had simply memorised them. That's the most common fatal error in vision projects.

• 99% on a video dataset
• A disaster in production
• Same video in both sets

Splitting by object

We simulate ten photos each of a hundred apples. Photos of one apple look alike, and in this data, the label is pure random noise. There's nothing to learn. So honest accuracy should be fifty percent. We score with a random split, then with a split that keeps each apple on one side. Run it. The random split says one hundred percent! Perfect. And that's on labels that mean nothing. The fair split says fifty two, which is the truth. A model can recognise the individual apple, and remember its label. If you only did a random split, you'd believe that you'd solved it.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, cross_val_score

rng = np.random.default_rng(0)
fruit = np.repeat(np.arange(100), 10)
looks = rng.normal(0, 1, (100, 20))[fruit] + rng.normal(0, 0.1, (1000, 20))
label = rng.integers(0, 2, 100)[fruit]

model = RandomForestClassifier(100, random_state=0, n_jobs=2)
print(f"Random split: {cross_val_score(model, looks, label, cv=KFold(5, shuffle=True, random_state=0)).mean():.2f}")
print(f"By fruit:     {cross_val_score(model, looks, label, groups=fruit, cv=GroupKFold(5)).mean():.2f}")

Random split: 1.00
By fruit:     0.52

Splitting vision data

Always split by the thing that you'll see new in production. Split medical scans by patient, not by scan. Split video by clip. Split factory data by batch, or better, by date, so that you test on the future. Check for near duplicates between sets. If the accuracy looks too good to be true, then it nearly always is.

Summary