Train and Test Splits
Honest scores.
A student who memorizes last year's exam answers can score a hundred percent on last year's exam, and still fail this year. Models are the same. To know how well a model really works, test it on data it has never seen.

- Memorized answers
- Perfect on old questions
- Test on unseen data
Training error lies
We split three hundred noisy points into training and test sets. Then we fit polynomials of growing degree, and measure the error on both. Run it. Training error keeps falling as the model gets more complex. But test error falls, then rises again. That rise is overfitting.
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(0)
x = rng.uniform(0, 1, 300)
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.3, 300)
xtr, xte, ytr, yte = train_test_split(x, y, test_size=0.5, random_state=0)
for deg in [1, 3, 9, 15]:
c = np.polyfit(xtr, ytr, deg)
mse = lambda xs, ys: ((np.polyval(c, xs) - ys) ** 2).mean()
print(f"degree {deg:2d}: train {mse(xtr, ytr):.3f} test {mse(xte, yte):.3f}")degree 1: train 0.273 test 0.292 degree 3: train 0.080 test 0.101 degree 9: train 0.070 test 0.099 degree 15: train 0.069 test 0.116
Three sets
In practice, we use three sets. The training set teaches the model. The validation set helps choose settings, like the degree, or the learning rate. And the test set is used once, at the very end, for an honest final score. Using the test set to make choices quietly turns it into training data.
| key | value |
|---|---|
| Train | fit the weights |
| Validation | choose settings |
| Test | final score, once |
| Typical split | 70 / 15 / 15 |
Summary
Always judge a model on data it hasn't seen. Training error alone always looks too good. Next, measuring classifiers beyond accuracy.
Key takeaways- train_test_split
- Test error reveals overfitting
- Touch the test set once