Regression with Matrices
The normal equation.
With five hundred houses and three features, we have five hundred equations, but only a few unknown weights. There's no exact solution. Linear algebra gives us the best one, in a single formula.

- 500 equations
- A few unknown weights
- The best solution
The normal equation
First, we add a column of ones to X, for the intercept. The normal equation says the best weights solve X transpose X, times W, equals X transpose Y. We solve it. Run it. The same weights as scikit learn's linear regression gave us earlier.
import numpy as np
from houses import X, price
Xb = np.column_stack([np.ones(len(X)), X])
w = np.linalg.solve(Xb.T @ Xb, Xb.T @ price)
print("intercept:", w[0].round(1))
print("size, rooms, age:", w[1:].round(2))intercept: 47.0 size, rooms, age: [ 2.03 9.51 -1.52]
Also uses houses.py
import numpy as np
rng = np.random.default_rng(0)
n = 500
size = rng.uniform(50, 250, n)
rooms = np.round(size / 30 + rng.normal(0, 1, n)).clip(1)
age = rng.uniform(0, 60, n)
price = 2.0 * size + 10 * rooms - 1.5 * age + 50 + rng.normal(0, 25, n)
X = np.column_stack([size, rooms, age])When to use it
The normal equation is exact and quick for a few thousand features. But it needs to solve a features by features system, which gets very slow with millions of features. That's when we switch to gradient descent, which takes small steps towards the answer. We'll learn it in the calculus section.
| key | value |
|---|---|
| Formula | XᵀX w = Xᵀy |
| Ones column | the intercept |
| Few features | fast and exact |
| Millions | use gradient descent |
Summary
The normal equation finds the least squares weights in one step. Linear regression is linear algebra. Next, eigenvectors.
Key takeaways- Add a ones column
- Solve XᵀX w = Xᵀy
- Same as LinearRegression