Nested Cross Validation

Imagine a hundred students all guessing on a test, and you hire the one who scored the highest. Are they the smartest, or just the luckiest? Tuning does the same thing. You try a hundred settings, and pick the best score. Part of that score is luck, so the reported number flatters you.

• A hundred students guess
• You hire the highest score
• Part of that score is luck

Two loops, one honest score

The fix has two loops. The inner loop does the tuning, using cross validation to choose the settings. The outer loop then holds out a fold that the tuning never saw, and scores the final choice on it. We repeat this for each outer fold. It's slow, but it tells you how well your whole process works, including the tuning. Alternatively, keep one final test set locked away, and that does the same job.

Measuring the flattery

Here are the two numbers side by side. The first is the best score reported by grid search. The second wraps the whole search inside another cross validation. Run it. The honest score is a little lower. On this small grid, the gap is small. But try thousands of settings on a small dataset, and the gap can become large. If you report only the first number, you may be promising your boss something that you cannot deliver.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)

pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.1, 1, 10], "svc__gamma": [0.001, 0.01, 0.1]}

inner = GridSearchCV(pipe, grid, cv=3).fit(X, y)
outer = cross_val_score(GridSearchCV(pipe, grid, cv=3), X, y, cv=5)

print("Score reported by the search:", round(inner.best_score_, 4))
print("Honest nested score:         ", round(outer.mean(), 4))

Score reported by the search: 0.9737
Honest nested score:          0.9789

Summary