AI Hints
Statistics & Math for ML · Lesson 63 of 138

Many Tests

The multiple testing trap.

⏱ 2 min read▶ Video at 1:13:30🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A researcher tests whether twenty different foods cause headaches. One of them comes out significant. Jelly beans! But remember, five percent of tests are false alarms. With twenty tests, you'd expect about one, even if nothing works.

Many Tests
From the video: Many Tests
  • Twenty foods tested
  • One is significant
  • Expected by chance

At least one false alarm

If each test has a five percent false alarm rate, the chance that at least one of twenty is a false alarm is one minus zero point nine five to the power of twenty. Run it. About sixty four percent. The Bonferroni correction divides alpha by the number of tests, which brings it back down to about five percent.

multiple.pypython
tests = 20
alpha = 0.05

print("P(at least one false alarm):", round(1 - (1 - alpha) ** tests, 3))

bonferroni = alpha / tests
print("Bonferroni alpha:", bonferroni)
print("P(any false alarm) now:", round(1 - (1 - bonferroni) ** tests, 3))
Output
P(at least one false alarm): 0.642
Bonferroni alpha: 0.0025
P(any false alarm) now: 0.049

In machine learning too

This trap is everywhere in machine learning. Try enough models, features, and settings on the same test set, and one will look great by luck. That's why we keep a separate validation set for choosing, and only touch the test set once, at the very end.

keyvalue
Many testsmany false alarms
Bonferroniα ÷ number of tests
ML versiontuning on the test set
Fixvalidation set, test once

Summary

Running many tests guarantees some false alarms. Correct for it, or test fewer things. Next, effect size, and whether a result really matters.

Key takeaways
  • 1 − 0.95²⁰ ≈ 64%
  • Bonferroni: α / m
  • Touch the test set once
Next up: Effect Size — big enough to matter?