AI Hints
Statistics & Math for ML · Lesson 71 of 138

Least Squares

Minimizing squared errors.

⏱ 2 min read▶ Video at 1:23:30🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

What makes one line better than another? For each point, the line makes an error, the gap between the real score and the predicted one. The best line is the one that makes the total of the squared errors as small as possible. That's called least squares.

Least Squares
From the video: Least Squares
  • Every line makes errors
  • Square them, add them up
  • The smallest total wins

Comparing lines

We write a function that computes the sum of squared errors for any slope and intercept. Then we try a few guesses, and the least squares line. Run it. The polyfit line has the smallest error. No other line can beat it.

sse.pypython
import numpy as np
from study import hours, scores

def sse(m, b):
    return ((scores - (m * hours + b)) ** 2).sum()

print("guess 1:", sse(5, 40))
print("guess 2:", sse(6, 35))
m, b = np.polyfit(hours, scores, 1)
print("best:   ", sse(m, b).round(1))
Output
guess 1: 45
guess 2: 29
best:    22.7
Also uses study.py
import numpy as np

hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
scores = np.array([42, 50, 53, 61, 64, 72, 74, 85])

The formula

For one feature, there's a neat formula. The slope is the covariance of X and Y, divided by the variance of X. And the intercept makes the line pass through the point of the two means. Later in the course, we'll find the same line with gradient descent, the way neural networks learn.

keyvalue
Erroractual − predicted
Least squaresminimize Σ error²
Slopecov(x, y) ÷ var(x)
Interceptȳ − slope × x̄

Summary

Least squares picks the line with the smallest sum of squared errors. The slope is covariance over variance. Next, measuring how good the line is.

Key takeaways
  • Minimize Σ error²
  • slope = cov / var
  • Line passes through the means
Next up: R-Squared — how much is explained