AI Hints
Statistics & Math for ML · Lesson 89 of 138

PCA in Practice

Fewer features.

⏱ 2 min read▶ Video at 1:44:56🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Scikit learn comes with a dataset of handwritten digits. Each image is eight by eight pixels, so sixty four features. Can we squash sixty four numbers down to just a few, and still keep most of the information?

PCA in Practice
From the video: PCA in Practice
  • Handwritten digits
  • 64 pixel features
  • Squash them down?

Explained variance

We fit PCA from scikit learn on the digits, keeping ten components. Explained variance ratio says what share of the total variance each component captures. Run it. Just ten components keep about three quarters of the information in all sixty four pixels.

pca_digits.pypython
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

X, y = load_digits(return_X_y=True)
pca = PCA(n_components=10).fit(X)

print("features:", X.shape[1])
print("first 3:", pca.explained_variance_ratio_[:3].round(3))
print("10 components keep:", pca.explained_variance_ratio_.sum().round(3))
Output
features: 64
first 3: [0.149 0.136 0.118]
10 components keep: 0.738

Seeing sixty four dimensions

With just two components, we can draw every image as a dot, coloured by which digit it is. Run it. Even in two dimensions, the digits form clear groups. PCA is a great first look at any high dimensional data.

pca_plot.pypython
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

X, y = load_digits(return_X_y=True)
Z = PCA(n_components=2).fit_transform(X)

plt.scatter(Z[:, 0], Z[:, 1], c=y, cmap="tab10", s=8)
plt.colorbar(label="digit"); plt.title("Digits in 2 dimensions")
plt.savefig("digits_pca.png")

Summary

PCA compresses many features into a few components that keep most of the variance. It's great for visualizing and speeding up models. Next, a linear algebra quiz.

Key takeaways
  • PCA(n_components=k)
  • explained_variance_ratio_
  • Plot in 2D
Next up: Linear Algebra Quiz — test yourself