Permutation Tests
Shuffle the labels.
Here's a test you can build yourself, with no formulas at all. If the null is true, and the group labels don't matter, then shuffling the labels shouldn't change anything. So shuffle them thousands of times, and see how often chance alone beats the real difference.

- If labels don't matter
- Shuffle them
- How often does chance win?
Shuffling
We use the study app scores again. The real difference in means is our statistic. Then we shuffle all the scores together, split them into two fake groups, and compute the difference ten thousand times. The p value is the share of shuffles at least as extreme as the real one. Run it. Close to the t test's answer.
import numpy as np
rng = np.random.default_rng(14)
app, no_app = rng.normal(75, 10, 40), rng.normal(70, 10, 40)
real = app.mean() - no_app.mean()
both = np.concatenate([app, no_app])
diffs = []
for _ in range(10_000):
rng.shuffle(both)
diffs.append(both[:40].mean() - both[40:].mean())
print(f"real diff {real:.2f}, p = {np.mean(np.abs(diffs) >= abs(real)):.4f}")real diff 5.20, p = 0.0240
Why permutation tests are great
They make very few assumptions. The data doesn't need to be normal. And they work with any statistic, like a difference in medians. They also show exactly what a p value means. The chance that random shuffling would give a result this extreme. Sci Pi even has a permutation test function built in.
| key | value |
|---|---|
| Assumes | labels are swappable |
| No normality | needed |
| Any statistic | means, medians, more |
| scipy | stats.permutation_test |
Summary
A permutation test shuffles labels to see what chance alone produces. It's simple, flexible, and makes few assumptions. Next, the two kinds of mistakes a test can make.
Key takeaways- Shuffle the labels
- Count extreme results
- Few assumptions