PCA
The main directions.
Data often has many features that overlap. Height and arm length, or a house's size and its number of rooms. Principal component analysis, or PCA, finds new directions that capture the most variation, so we can keep the important ones and drop the rest.

- Features often overlap
- Find the main directions
- Keep the important ones
Directions of spread
Here's some two dimensional data, stretched along a diagonal. We centre it, compute its covariance matrix, and find the eigenvectors. Then we draw each eigenvector as an arrow, scaled by how much spread it captures. Run it. The long arrow points along the data's main direction. That's the first principal component.
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
x = rng.normal(0, 2, 400)
data = np.column_stack([x, 0.6 * x + rng.normal(0, 0.7, 400)])
values, vectors = np.linalg.eigh(np.cov((data - data.mean(0)).T))
plt.scatter(*data.T, s=8, alpha=0.6)
for val, vec in zip(values, vectors.T):
plt.quiver(0, 0, *(vec * 2 * np.sqrt(val)), color="#22d3ee", angles="xy", scale_units="xy", scale=1)
plt.gca().set_aspect("equal"); plt.title("Principal components")
plt.savefig("pca.png")What PCA gives you
Each principal component is a new feature, made from a mix of the old ones. They're sorted by how much variance they capture, and they're uncorrelated with each other. Often, a handful of components capture most of the information in dozens, or even hundreds, of features.
| key | value |
|---|---|
| Component | a direction of spread |
| Sorted | most variance first |
| Uncorrelated | no overlap |
| Use | compress, visualize, denoise |
Summary
PCA finds the directions where data varies most. They're the eigenvectors of the covariance matrix. Next, PCA in practice.
Key takeaways- Centre the data
- Eigenvectors of the covariance
- Most variance first