AI Hints
Home›Statistics & Math for ML Tutorial›Likelihood and Overfitting
Statistics & Math for ML · Lesson 110 of 138

Bias and Variance

The trade-off.

⏱ 2 min read▶ Video at 2:10:39🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A model can fail in two opposite ways. It can be too simple, and miss the real pattern. That's high bias, or underfitting. Or it can be too complex, and memorize the noise. That's high variance, or overfitting.

Bias and Variance
From the video: Bias and Variance
  • Too simple: misses the pattern
  • Too complex: memorizes noise
  • Bias and variance

Three models

We take fifteen noisy points from a curve, and fit polynomials of degree one, three, and twelve. Run it. The straight line is too simple, and misses the curve. Degree three follows the pattern nicely. Degree twelve wiggles wildly to hit every point, and would predict new points badly.

bias_variance.pypython
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(1)
x = np.sort(rng.uniform(0, 1, 15))
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.25, 15)
grid = np.linspace(0, 1, 300)

plt.scatter(x, y, color="#ffffff", zorder=3)
for deg in [1, 3, 12]:
    plt.plot(grid, np.polyval(np.polyfit(x, y, deg), grid), label=f"degree {deg}")
plt.ylim(-2, 2); plt.legend(); plt.title("Underfit, good fit, overfit")
plt.savefig("bias_variance.png")

The trade-off

More complex models have lower bias, but higher variance. The best model is somewhere in the middle. To fix underfitting, use a more flexible model, or better features. To fix overfitting, get more data, use a simpler model, or add regularization.

High bias: underfit

  • too simplebad on training dataadd flexibility

High variance: overfit

  • too complexgreat on training, bad on new datamore data, regularize

Summary

Bias is error from being too simple, and variance is error from being too sensitive to the training data. Aim for the balance. Next, regularization.

Key takeaways
  • Underfit: high bias
  • Overfit: high variance
  • Find the middle
Next up: Regularization — keeping weights small