AI Hints
Home›Statistics & Math for ML Tutorial›Residuals and Many Features
Statistics & Math for ML · Lesson 73 of 138

Residuals

What the line misses.

⏱ 2 min read▶ Video at 1:25:49🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A residual is what the model got wrong for one point. The real value, minus the prediction. Looking at residuals is the best way to check whether a straight line was the right choice.

Residuals
From the video: Residuals
  • Residual = actual − predicted
  • What the model missed
  • Was a line the right choice?

A residual plot

Here, the real relationship is curved, but we fit a straight line anyway. On the left, the line looks almost fine. On the right, we plot the residuals. Run it. They make a clear U shape. Residuals should look like random noise around zero. A pattern means the model is missing something.

residuals.pypython
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 200)
y = 0.5 * x ** 2 + rng.normal(0, 3, 200)
m, b = np.polyfit(x, y, 1)

fig, ax = plt.subplots(1, 2, figsize=(9, 4.5))
ax[0].scatter(x, y, s=10); ax[0].plot(x, m * x + b, color="#22d3ee")
ax[1].scatter(x, y - (m * x + b), s=10); ax[1].axhline(0, color="#22d3ee")
plt.tight_layout(); plt.savefig("resid.png")

What residuals tell you

AU shape means the relationship is curved, so try adding a squared feature. A funnel, where residuals grow wider, means the errors get bigger for bigger values, so try a log. And one huge residual is an outlier worth investigating. Random scatter around zero is what you want.

keyvalue
Random scattergood
U shapecurved: add x²
Funnelgrowing errors: try log
One huge pointan outlier

Summary

Residuals show what the model missed. They should look like random noise. Patterns tell you how to improve the model. Next, regression with many features.

Key takeaways
  • actual − predicted
  • Plot them
  • Patterns mean problems
Next up: Multiple Regression — many features