Bias and Variance
The trade-off.
A model can fail in two opposite ways. It can be too simple, and miss the real pattern. That's high bias, or underfitting. Or it can be too complex, and memorize the noise. That's high variance, or overfitting.

- Too simple: misses the pattern
- Too complex: memorizes noise
- Bias and variance
Three models
We take fifteen noisy points from a curve, and fit polynomials of degree one, three, and twelve. Run it. The straight line is too simple, and misses the curve. Degree three follows the pattern nicely. Degree twelve wiggles wildly to hit every point, and would predict new points badly.
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 1, 15))
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.25, 15)
grid = np.linspace(0, 1, 300)
plt.scatter(x, y, color="#ffffff", zorder=3)
for deg in [1, 3, 12]:
plt.plot(grid, np.polyval(np.polyfit(x, y, deg), grid), label=f"degree {deg}")
plt.ylim(-2, 2); plt.legend(); plt.title("Underfit, good fit, overfit")
plt.savefig("bias_variance.png")The trade-off
More complex models have lower bias, but higher variance. The best model is somewhere in the middle. To fix underfitting, use a more flexible model, or better features. To fix overfitting, get more data, use a simpler model, or add regularization.
High bias: underfit
- too simplebad on training dataadd flexibility
High variance: overfit
- too complexgreat on training, bad on new datamore data, regularize
Summary
Bias is error from being too simple, and variance is error from being too sensitive to the training data. Aim for the balance. Next, regularization.
Key takeaways- Underfit: high bias
- Overfit: high variance
- Find the middle