AI Hints
Home›Statistics & Math for ML Tutorial›Testing in Practice
Statistics & Math for ML · Lesson 60 of 138

Permutation Tests

Shuffle the labels.

⏱ 2 min read▶ Video at 1:09:47🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Here's a test you can build yourself, with no formulas at all. If the null is true, and the group labels don't matter, then shuffling the labels shouldn't change anything. So shuffle them thousands of times, and see how often chance alone beats the real difference.

Permutation Tests
From the video: Permutation Tests
  • If labels don't matter
  • Shuffle them
  • How often does chance win?

Shuffling

We use the study app scores again. The real difference in means is our statistic. Then we shuffle all the scores together, split them into two fake groups, and compute the difference ten thousand times. The p value is the share of shuffles at least as extreme as the real one. Run it. Close to the t test's answer.

perm_test.pypython
import numpy as np

rng = np.random.default_rng(14)
app, no_app = rng.normal(75, 10, 40), rng.normal(70, 10, 40)
real = app.mean() - no_app.mean()

both = np.concatenate([app, no_app])
diffs = []
for _ in range(10_000):
    rng.shuffle(both)
    diffs.append(both[:40].mean() - both[40:].mean())
print(f"real diff {real:.2f}, p = {np.mean(np.abs(diffs) >= abs(real)):.4f}")
Output
real diff 5.20, p = 0.0240

Why permutation tests are great

They make very few assumptions. The data doesn't need to be normal. And they work with any statistic, like a difference in medians. They also show exactly what a p value means. The chance that random shuffling would give a result this extreme. Sci Pi even has a permutation test function built in.

keyvalue
Assumeslabels are swappable
No normalityneeded
Any statisticmeans, medians, more
scipystats.permutation_test

Summary

A permutation test shuffles labels to see what chance alone produces. It's simple, flexible, and makes few assumptions. Next, the two kinds of mistakes a test can make.

Key takeaways
  • Shuffle the labels
  • Count extreme results
  • Few assumptions
Next up: Type I and II Errors — false alarms and misses