AI Hints
Statistics & Math for ML · Lesson 112 of 138

Train and Test Splits

Honest scores.

⏱ 2 min read▶ Video at 2:13:04🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A student who memorizes last year's exam answers can score a hundred percent on last year's exam, and still fail this year. Models are the same. To know how well a model really works, test it on data it has never seen.

Train and Test Splits
From the video: Train and Test Splits
  • Memorized answers
  • Perfect on old questions
  • Test on unseen data

Training error lies

We split three hundred noisy points into training and test sets. Then we fit polynomials of growing degree, and measure the error on both. Run it. Training error keeps falling as the model gets more complex. But test error falls, then rises again. That rise is overfitting.

split.pypython
import numpy as np
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
x = rng.uniform(0, 1, 300)
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, 300)
xtr, xte, ytr, yte = train_test_split(x, y, test_size=0.5, random_state=0)

for deg in [1, 3, 9, 15]:
    c = np.polyfit(xtr, ytr, deg)
    mse = lambda xs, ys: ((np.polyval(c, xs) - ys) ** 2).mean()
    print(f"degree {deg:2d}: train {mse(xtr, ytr):.3f}  test {mse(xte, yte):.3f}")
Output
degree  1: train 0.273  test 0.292
degree  3: train 0.080  test 0.101
degree  9: train 0.070  test 0.099
degree 15: train 0.069  test 0.116

Three sets

In practice, we use three sets. The training set teaches the model. The validation set helps choose settings, like the degree, or the learning rate. And the test set is used once, at the very end, for an honest final score. Using the test set to make choices quietly turns it into training data.

keyvalue
Trainfit the weights
Validationchoose settings
Testfinal score, once
Typical split70 / 15 / 15

Summary

Always judge a model on data it hasn't seen. Training error alone always looks too good. Next, measuring classifiers beyond accuracy.

Key takeaways
  • train_test_split
  • Test error reveals overfitting
  • Touch the test set once
Next up: Precision and Recall — beyond accuracy