AI Hints
Statistics & Math for ML · Lesson 88 of 138

PCA

The main directions.

⏱ 2 min read▶ Video at 1:43:39🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Data often has many features that overlap. Height and arm length, or a house's size and its number of rooms. Principal component analysis, or PCA, finds new directions that capture the most variation, so we can keep the important ones and drop the rest.

PCA
From the video: PCA
  • Features often overlap
  • Find the main directions
  • Keep the important ones

Directions of spread

Here's some two dimensional data, stretched along a diagonal. We centre it, compute its covariance matrix, and find the eigenvectors. Then we draw each eigenvector as an arrow, scaled by how much spread it captures. Run it. The long arrow points along the data's main direction. That's the first principal component.

pca_idea.pypython
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
x = rng.normal(0, 2, 400)
data = np.column_stack([x, 0.6 * x + rng.normal(0, 0.7, 400)])

values, vectors = np.linalg.eigh(np.cov((data - data.mean(0)).T))
plt.scatter(*data.T, s=8, alpha=0.6)
for val, vec in zip(values, vectors.T):
    plt.quiver(0, 0, *(vec * 2 * np.sqrt(val)), color="#22d3ee", angles="xy", scale_units="xy", scale=1)
plt.gca().set_aspect("equal"); plt.title("Principal components")
plt.savefig("pca.png")

What PCA gives you

Each principal component is a new feature, made from a mix of the old ones. They're sorted by how much variance they capture, and they're uncorrelated with each other. Often, a handful of components capture most of the information in dozens, or even hundreds, of features.

keyvalue
Componenta direction of spread
Sortedmost variance first
Uncorrelatedno overlap
Usecompress, visualize, denoise

Summary

PCA finds the directions where data varies most. They're the eigenvectors of the covariance matrix. Next, PCA in practice.

Key takeaways
  • Centre the data
  • Eigenvectors of the covariance
  • Most variance first
Next up: PCA in Practice — fewer features