AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Geometry
Statistics & Math for ML · Lesson 129 of 138

Matrix Rank

How much information.

⏱ 2 min read▶ Video at 2:34:20🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A data matrix might have ten columns, but if some columns are just combinations of others, it really carries less information. The rank of a matrix counts how many truly independent columns it has.

Matrix Rank
From the video: Matrix Rank
  • Ten columns
  • Some are copies or combinations
  • Rank counts the real ones

np.linalg.matrix_rank

Here, the third column is just the first plus the second, and the fourth is double the first. So only two columns are independent. Run it. The matrix has four columns, but its rank is two.

rank.pypython
import numpy as np

rng = np.random.default_rng(0)
a, b = rng.normal(size=100), rng.normal(size=100)
X = np.column_stack([a, b, a + b, 2 * a])

print("columns:", X.shape[1])
print("rank:   ", np.linalg.matrix_rank(X))
Output
columns: 4
rank:    2

Why rank matters

If the rank is lower than the number of columns, the matrix X transpose X is singular, so the normal equation can't be solved exactly. Linear regression can't tell the duplicate features apart. Dropping redundant features, or using ridge regression, fixes it. Low rank is also why PCA can compress data so well.

keyvalue
Rankindependent columns
Full rankno redundant columns
Low rankXᵀX is singular
Fixesdrop features, ridge

Summary

The rank counts how many independent columns a matrix has. Redundant features lower it, and break the normal equation. Next, second derivatives, and curvature.

Key takeaways
  • np.linalg.matrix_rank
  • Duplicates lower rank
  • Ridge helps
Next up: Second Derivatives — curvature and minimums