PCA

Hold a teapot up to the light, and look at its shadow on the wall. One angle gives you a useless blob. Another angle clearly shows the spout and the handle. A shadow loses one whole dimension, yet the right angle keeps almost everything that matters. Principal component analysis finds that best angle for your data.

• A teapot and its shadow
• The right angle keeps the shape
• PCA finds that angle

Finding the best angle

Here is a cloud of points that uses two numbers each. Notice that it is long and thin. Most of the action is along one direction. P C A finds that direction, the one along which the data spreads the most. That is the first principal component. Now we project every point on to that line. Two numbers have become one, and we kept nearly all the information. The second component would be at a right angle to the first, and so on.

Keeping 90 percent

Each digit image has sixty four features. We ask P C A to keep ninety percent of the information, and let it choose how many components that needs. Run it. Sixty four features squeeze down to around thirty. We have thrown away over half of our columns, and kept ninety percent of the signal. The shares show how much each component carries, and they fall quickly. As usual, we scale first.

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)

pca = make_pipeline(StandardScaler(), PCA(n_components=0.90))
Z = pca.fit_transform(X)

print("Features before:", X.shape[1])
print("Features after: ", Z.shape[1])
print("First five shares:", pca[-1].explained_variance_ratio_[:5].round(3))

Features before: 64
Features after:  31
First five shares: [0.12  0.096 0.084 0.065 0.049]

Does the model suffer?

Does this hurt accuracy? Let's check. We train the same classifier, on a growing number of components. Run it. With only five, the model struggles. By thirty, it is close to the score with all sixty four. So we get a model that is smaller, and faster, with very little loss. One warning though. The new components are mixtures of the old features, so they are hard to explain to people.

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)

for n in [5, 10, 20, 30, 64]:
    m = make_pipeline(StandardScaler(), PCA(n_components=n),
                      LogisticRegression(max_iter=3000))
    print(f"{n:>2} components  accuracy {cross_val_score(m, X, y, cv=3).mean():.3f}")

 5 components  accuracy 0.772
10 components  accuracy 0.837
20 components  accuracy 0.902
30 components  accuracy 0.917
64 components  accuracy 0.930

Summary