AI Hints
Home›Statistics & Math for ML Tutorial›Solving and Eigenvectors
Statistics & Math for ML · Lesson 86 of 138

Regression with Matrices

The normal equation.

⏱ 2 min read▶ Video at 1:41:26🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

With five hundred houses and three features, we have five hundred equations, but only a few unknown weights. There's no exact solution. Linear algebra gives us the best one, in a single formula.

Regression with Matrices
From the video: Regression with Matrices
  • 500 equations
  • A few unknown weights
  • The best solution

The normal equation

First, we add a column of ones to X, for the intercept. The normal equation says the best weights solve X transpose X, times W, equals X transpose Y. We solve it. Run it. The same weights as scikit learn's linear regression gave us earlier.

normal_eq.pypython
import numpy as np
from houses import X, price

Xb = np.column_stack([np.ones(len(X)), X])

w = np.linalg.solve(Xb.T @ Xb, Xb.T @ price)
print("intercept:", w[0].round(1))
print("size, rooms, age:", w[1:].round(2))
Output
intercept: 47.0
size, rooms, age: [ 2.03  9.51 -1.52]
Also uses houses.py
import numpy as np

rng = np.random.default_rng(0)
n = 500
size = rng.uniform(50, 250, n)
rooms = np.round(size / 30 + rng.normal(0, 1, n)).clip(1)
age = rng.uniform(0, 60, n)
price = 2.0 * size + 10 * rooms - 1.5 * age + 50 + rng.normal(0, 25, n)
X = np.column_stack([size, rooms, age])

When to use it

The normal equation is exact and quick for a few thousand features. But it needs to solve a features by features system, which gets very slow with millions of features. That's when we switch to gradient descent, which takes small steps towards the answer. We'll learn it in the calculus section.

keyvalue
FormulaXᵀX w = Xᵀy
Ones columnthe intercept
Few featuresfast and exact
Millionsuse gradient descent

Summary

The normal equation finds the least squares weights in one step. Linear regression is linear algebra. Next, eigenvectors.

Key takeaways
  • Add a ones column
  • Solve XᵀX w = Xᵀy
  • Same as LinearRegression
Next up: Eigenvectors — directions that don't turn