AI Hints
Home›Statistics & Math for ML Tutorial›Cross-Validation and Interviews
Statistics & Math for ML · Lesson 115 of 138

Cross-Validation

Many splits.

⏱ 2 min read▶ Video at 2:16:44🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

One train test split can be lucky or unlucky. Maybe the easy examples all landed in the test set. Cross validation repeats the split several times, and averages the results, for a much more reliable score.

Cross-Validation
From the video: Cross-Validation
  • One split can be lucky
  • Repeat the split
  • Average the scores

K-fold

In five fold cross validation, we cut the data into five equal parts, called folds. We train on four folds, and test on the fifth. Then we rotate, so every fold gets a turn as the test set. That gives five scores, and we report their mean and standard deviation.

Splitinto k folds
Trainon k − 1 folds
Teston the one left out
Repeatk times, average

cross_val_score

Scikit learn does it in one line. We score a logistic regression with five fold cross validation on the breast cancer dataset that comes with scikit learn. Run it. Five scores, each a little different. The mean is our estimate, and the standard deviation shows how much it wobbles.

cv.pypython
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression())

scores = cross_val_score(model, X, y, cv=5)
print("scores:", scores.round(3))
print(f"mean {scores.mean():.3f} ± {scores.std():.3f}")
Output
scores: [0.982 0.982 0.974 0.974 0.991]
mean 0.981 ± 0.007

Summary

Cross validation trains and tests on several splits, and averages them. It gives a more reliable score, with a sense of its spread. Next, interview questions on statistics.

Key takeaways
  • k folds
  • cross_val_score
  • Report mean ± std
Next up: Interview Questions: Part 1 — statistics