Regularization
Keeping weights small.
Overfit models often have huge weights, pulling in different directions to hit every training point. Regularization adds a penalty for big weights to the loss. The model has to balance fitting the data against keeping its weights small.

- Overfit models have huge weights
- Penalize big weights
- Balance fit and size
Ridge regression
We fit the same degree twelve polynomial, with plain linear regression, and with ridge, which adds an L two penalty on the weights. Alpha sets how strong the penalty is. Run it. The plain model's largest weight is over eight thousand. Ridge keeps its weights small, so the curve is much smoother.
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.preprocessing import PolynomialFeatures
rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 1, 15)).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel() + rng.normal(0, 0.25, 15)
X = PolynomialFeatures(12).fit_transform(x)
for model in [LinearRegression(), Ridge(alpha=0.001)]:
w = model.fit(X, y).coef_
print(f"{type(model).__name__:16s} largest weight: {np.abs(w).max():,.0f}")LinearRegression largest weight: 8,211 Ridge largest weight: 9
L1 and L2
Ridge uses the L two penalty, the sum of squared weights, which shrinks every weight smoothly. Lasso uses the L one penalty, the sum of absolute weights, which can push some weights to exactly zero, so it also selects features. Elastic net mixes the two. In neural networks, L two is called weight decay.
| Method | Penalty | Effect |
|---|---|---|
| Ridge | L2: Σ w² | shrinks all weights |
| Lasso | L1: Σ |w| | sets some to zero |
| Elastic net | both | a mix |
| Weight decay | L2 in networks | same as ridge |
Summary
Regularization penalizes big weights to prevent overfitting. Ridge shrinks weights, and lasso can remove features. Next, measuring performance honestly, with train and test splits.
Key takeaways- Loss + penalty
- Ridge: L2
- Lasso: L1