Cross Validation

Suppose your test set happens, purely by luck, to be full of easy houses. Your score would look brilliant, and it would be false. A single split is a single exam, and anyone can have a lucky day. So let's give our model five exams.

• One split can be lucky
• A lucky score is a false score
• Give it five exams

Five folds

Here is how it works. Cut the training data into five equal parts, called folds. In round one, the first fold is the test, and the model trains on the other four. In round two, the second fold is the test. And so on. After five rounds, every row has been tested exactly once. We take the average of the five scores. And now we also get something a single split cannot give us. The spread between the scores tells us how stable the model is.

Cross validation in code

Cross val score does the whole thing in one call. We pass the model, the data, and cv equals five. Scikit learn likes bigger to mean better, so errors come back as negative numbers, and we flip the sign. Look at the five scores. They vary a lot! That is a warning sign. The data is stored in geographic order, so each fold is a different region of California. Without shuffling, our exams are not fair.

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

X, y = fetch_california_housing(return_X_y=True)

scores = cross_val_score(LinearRegression(), X, y, cv=5,
                         scoring="neg_mean_absolute_error")

maes = -scores * 100000
print("Five scores:", maes.round(0))
print(f"Mean: ${maes.mean():,.0f}")
print(f"Spread: ${maes.std():,.0f}")

Five scores: [54599. 56618. 57655. 53191. 51685.]
Mean: $54,750
Spread: $2,181

Shuffle the folds

Let's fix that. K fold, with shuffle set to true, mixes the rows before cutting them into folds. Same model, same data. Run it. Now the five scores are close together, and the spread is tiny. We have a score that we can trust, and a reason to trust it. This little detail catches out many beginners. Always ask how your data is ordered.

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score

X, y = fetch_california_housing(return_X_y=True)

folds = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(LinearRegression(), X, y, cv=folds,
                         scoring="neg_mean_absolute_error")

maes = -scores * 100000
print("Five scores:", maes.round(0))
print(f"Mean: ${maes.mean():,.0f}")
print(f"Spread: ${maes.std():,.0f}")

Five scores: [53320. 53668. 52918. 51714. 54219.]
Mean: $53,168
Spread: $843

Summary