AI Hints
Home›Statistics & Math for ML Tutorial›Likelihood and Overfitting
Statistics & Math for ML · Lesson 111 of 138

Regularization

Keeping weights small.

⏱ 2 min read▶ Video at 2:11:43🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Overfit models often have huge weights, pulling in different directions to hit every training point. Regularization adds a penalty for big weights to the loss. The model has to balance fitting the data against keeping its weights small.

Regularization
From the video: Regularization
  • Overfit models have huge weights
  • Penalize big weights
  • Balance fit and size

Ridge regression

We fit the same degree twelve polynomial, with plain linear regression, and with ridge, which adds an L two penalty on the weights. Alpha sets how strong the penalty is. Run it. The plain model's largest weight is over eight thousand. Ridge keeps its weights small, so the curve is much smoother.

ridge.pypython
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 1, 15)).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel() + rng.normal(0, 0.25, 15)
X = PolynomialFeatures(12).fit_transform(x)

for model in [LinearRegression(), Ridge(alpha=0.001)]:
    w = model.fit(X, y).coef_
    print(f"{type(model).__name__:16s} largest weight: {np.abs(w).max():,.0f}")
Output
LinearRegression largest weight: 8,211
Ridge            largest weight: 9

L1 and L2

Ridge uses the L two penalty, the sum of squared weights, which shrinks every weight smoothly. Lasso uses the L one penalty, the sum of absolute weights, which can push some weights to exactly zero, so it also selects features. Elastic net mixes the two. In neural networks, L two is called weight decay.

MethodPenaltyEffect
RidgeL2: Σ w²shrinks all weights
LassoL1: Σ |w|sets some to zero
Elastic netbotha mix
Weight decayL2 in networkssame as ridge

Summary

Regularization penalizes big weights to prevent overfitting. Ridge shrinks weights, and lasso can remove features. Next, measuring performance honestly, with train and test splits.

Key takeaways
  • Loss + penalty
  • Ridge: L2
  • Lasso: L1
Next up: Train and Test Splits — honest scores