AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Geometry
Statistics & Math for ML · Lesson 128 of 138

Projections

Shadows of vectors.

⏱ 2 min read▶ Video at 2:33:12🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Shine a light straight down onto a stick, and its shadow on the floor is a projection. In linear algebra, projecting one vector onto another finds how much of it points in that direction. It's how PCA and regression work underneath.

Projections
From the video: Projections
  • A stick and its shadow
  • How much points that way?
  • Under PCA and regression

Projecting onto a direction

To project A onto B, take the dot product of A with B, divide by B dot B, and multiply B by that amount. What's left over, A minus the projection, is at a right angle to B. Run it. The leftover's dot product with B is zero.

projection.pypython
import numpy as np

a = np.array([3.0, 4.0])
b = np.array([5.0, 0.0])

proj = (a @ b) / (b @ b) * b
rest = a - proj
print("projection:", proj)
print("leftover:  ", rest)
print("leftover · b =", rest @ b)
Output
projection: [3. 0.]
leftover:   [0. 4.]
leftover · b = 0.0

Where projections appear

Linear regression projects the target onto the space spanned by the features. The residuals are the leftover part, at right angles to every feature. And PCA projects data onto its main directions, keeping the long shadows, and dropping the short ones.

keyvalue
proj(a·b ÷ b·b) × b
Leftoverat a right angle
Regressionproject y onto features
PCAproject onto components

Summary

A projection is the shadow of one vector on another. The leftover is at a right angle. Regression and PCA are both projections. Next, matrix rank.

Key takeaways
  • (a·b / b·b) b
  • Leftover ⟂ b
  • Shadows of data
Next up: Matrix Rank — how much information