Projections
Shadows of vectors.
Shine a light straight down onto a stick, and its shadow on the floor is a projection. In linear algebra, projecting one vector onto another finds how much of it points in that direction. It's how PCA and regression work underneath.

- A stick and its shadow
- How much points that way?
- Under PCA and regression
Projecting onto a direction
To project A onto B, take the dot product of A with B, divide by B dot B, and multiply B by that amount. What's left over, A minus the projection, is at a right angle to B. Run it. The leftover's dot product with B is zero.
import numpy as np
a = np.array([3.0, 4.0])
b = np.array([5.0, 0.0])
proj = (a @ b) / (b @ b) * b
rest = a - proj
print("projection:", proj)
print("leftover: ", rest)
print("leftover · b =", rest @ b)projection: [3. 0.] leftover: [0. 4.] leftover · b = 0.0
Where projections appear
Linear regression projects the target onto the space spanned by the features. The residuals are the leftover part, at right angles to every feature. And PCA projects data onto its main directions, keeping the long shadows, and dropping the short ones.
| key | value |
|---|---|
| proj | (a·b ÷ b·b) × b |
| Leftover | at a right angle |
| Regression | project y onto features |
| PCA | project onto components |
Summary
A projection is the shadow of one vector on another. The leftover is at a right angle. Regression and PCA are both projections. Next, matrix rank.
Key takeaways- (a·b / b·b) b
- Leftover ⟂ b
- Shadows of data