The Danger of Overfitting

Imagine a student who memorises every answer in the practice exam, word for word. On the practice exam, they score one hundred percent. On the real exam, with new questions, they fail. They did not learn the subject. They learned the paper. Models do exactly the same thing, and it is the most common way that machine learning goes wrong.

• A student memorises the practice exam
• Perfect practice, failed real exam
• Models do the same thing

Too simple, too complex

Watch the left side, as the model gets more complex. On the right are two scores. The green line is the error on the data it trained on. It keeps falling, and falling, and it looks wonderful. The orange line is the error on new data, and that is the only one that matters. At first, it falls too. Then it turns and climbs. That point where they split is where the model stops learning the pattern, and starts memorising the noise. To the left is underfitting. To the right is overfitting. Our job is to stay in the middle.

Catching it in code

Let's catch this in real numbers. We make a noisy curve, and hold back a quarter of the points. Then we try four degrees. For each, we print the score on training data, and on test data. Run it, and read the table closely. As the degree grows, the train score keeps climbing. At degree twelve, it looks excellent. But look at the test score on that row. It has collapsed. That model is the student who memorised the paper.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
x = rng.uniform(0, 6, 40).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(0, 0.3, 40)

X_tr, X_te, y_tr, y_te = train_test_split(x, y, random_state=1)

for d in [1, 3, 5, 12]:
    m = make_pipeline(StandardScaler(), PolynomialFeatures(d),
                      LinearRegression())
    m.fit(X_tr, y_tr)
    print(f"degree {d:>2}  train {m.score(X_tr, y_tr):6.3f}"
          f"  test {m.score(X_te, y_te):6.3f}")

degree  1  train  0.493  test -0.189
degree  3  train  0.849  test  0.636
degree  5  train  0.855  test  0.554
degree 12  train  0.894  test -51.764

Summary