Many Tests
The multiple testing trap.
A researcher tests whether twenty different foods cause headaches. One of them comes out significant. Jelly beans! But remember, five percent of tests are false alarms. With twenty tests, you'd expect about one, even if nothing works.

- Twenty foods tested
- One is significant
- Expected by chance
At least one false alarm
If each test has a five percent false alarm rate, the chance that at least one of twenty is a false alarm is one minus zero point nine five to the power of twenty. Run it. About sixty four percent. The Bonferroni correction divides alpha by the number of tests, which brings it back down to about five percent.
tests = 20
alpha = 0.05
print("P(at least one false alarm):", round(1 - (1 - alpha) ** tests, 3))
bonferroni = alpha / tests
print("Bonferroni alpha:", bonferroni)
print("P(any false alarm) now:", round(1 - (1 - bonferroni) ** tests, 3))P(at least one false alarm): 0.642 Bonferroni alpha: 0.0025 P(any false alarm) now: 0.049
In machine learning too
This trap is everywhere in machine learning. Try enough models, features, and settings on the same test set, and one will look great by luck. That's why we keep a separate validation set for choosing, and only touch the test set once, at the very end.
| key | value |
|---|---|
| Many tests | many false alarms |
| Bonferroni | α ÷ number of tests |
| ML version | tuning on the test set |
| Fix | validation set, test once |
Summary
Running many tests guarantees some false alarms. Correct for it, or test fewer things. Next, effect size, and whether a result really matters.
Key takeaways- 1 − 0.95²⁰ ≈ 64%
- Bonferroni: α / m
- Touch the test set once