Regularisation

When a model overfits, its weights often become enormous, as it twists itself to touch every point. So here is a clever idea. What if we charged the model a fee for having large weights? Then it would only use a big weight if that really earned its place.

• Overfit models have huge weights
• Charge a fee for big weights
• Only useful weights survive

Two kinds of fee

This idea is called regularisation. We add a penalty to the loss. Ridge regression charges for the square of each weight, so all the weights shrink, but rarely reach zero. Lasso charges for the absolute size, and it has a wonderful side effect. It pushes useless weights all the way to zero, which removes those features completely. A setting called alpha controls how large the fee is.

Taming a wild model

Here is our wild degree twelve model, the one that collapsed. We compare three versions. Plain linear regression, with no fee. Ridge. And lasso. We scale the features first, which matters for regularisation. For each, we print the test score, the biggest weight, and how many weights are still in use. Run it. The plain model has an enormous weight, and a bad score. Ridge and lasso tame it, and their test scores recover. Lasso also switches several features off. Same data, same features, and a healthy model.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
x = rng.uniform(0, 6, 40).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(0, 0.3, 40)
X_tr, X_te, y_tr, y_te = train_test_split(x, y, random_state=1)

for name, reg in [("Plain", LinearRegression()),
                  ("Ridge", Ridge(alpha=1.0)),
                  ("Lasso", Lasso(alpha=0.01, max_iter=50000))]:
    m = make_pipeline(PolynomialFeatures(12), StandardScaler(), reg)
    m.fit(X_tr, y_tr)
    size = np.abs(m[-1].coef_).max()
    used = (m[-1].coef_ != 0).sum()
    print(f"{name:<6} test {m.score(X_te, y_te):7.3f}"
          f"  biggest weight {size:10.2f}  used {used}")

Plain  test  -8.965  biggest weight    1171.51  used 12
Ridge  test   0.250  biggest weight       0.50  used 12
Lasso  test   0.428  biggest weight       1.66  used 3

Summary