Data Leakage

A hospital team once built a model with amazing accuracy for predicting a disease. Then it failed in the real world. Why? One feature showed that a patient had already been given the medicine for that disease. The model was reading the answer. That is data leakage. Information from the future, or from the test set, sneaks into training.

• An amazing model that failed
• It was reading the answer
• Information sneaks in

Finding patterns in pure noise

This next experiment should shock you. We generate five thousand columns of pure random noise, and random labels. There is nothing to learn here. A fair score would be around fifty percent, a coin flip. Now, in the first attempt, we pick the best twenty features using all the data, and then cross validate. That is the leak, because feature selection has seen the test folds. In the second attempt, the selection happens inside each fold. Run it. The leaky version reports a brilliant score on pure noise. The safe one tells the truth.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))   # pure random noise
y = rng.integers(0, 2, 100)        # random labels

leaky = SelectKBest(f_classif, k=20).fit_transform(X, y)
bad = cross_val_score(LogisticRegression(), leaky, y, cv=5).mean()

safe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
good = cross_val_score(safe, X, y, cv=5).mean()

print(f"With leakage:    {bad:.2f}")
print(f"Without leakage: {good:.2f}")

With leakage:    0.85
Without leakage: 0.48

Where leakage hides

Leakage hides in four main places. Target leakage, a feature that contains the answer, or only exists after the event. Preprocessing leakage, when you scale or fill in values using all the data before splitting. Time leakage, when you train on the future to predict the past. And group leakage, when the same customer appears in both train and test.

Pipelines stop leakage

The cure for most of these is a pipeline. A pipeline bundles every step together, the scaler and the model. Now, inside each cross validation fold, the scaler is fitted only on that fold's training rows. It cannot peek. We cross validate the pipeline, then fit it on the training set, and finally open the sealed test set. Two scores, and both are honest. From now on, every serious model in this course lives inside a pipeline.

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", Ridge(alpha=1.0)),
])

print("CV R2:  ", cross_val_score(pipe, X_train, y_train, cv=5).mean().round(3))
pipe.fit(X_train, y_train)
print("Test R2:", round(pipe.score(X_test, y_test), 3))

CV R2:   0.611
Test R2: 0.576

Summary