Type I and II Errors
False alarms and misses.
A hypothesis test can be wrong in two ways. It can raise a false alarm, saying there's an effect when there isn't. Or it can miss a real effect. These two mistakes have names, type one and type two errors.

- Tests can be wrong
- False alarms
- Missed effects
The four outcomes
There are four possibilities. If there's no real effect, and we don't reject, good. If there's no effect, but we reject, that's a type one error, a false alarm. If there is an effect, and we reject, good. And if there's an effect, but we miss it, that's a type two error.
| No real effect | Real effect | |
|---|---|---|
| Reject the null | Type I error: false alarm | Correct: found it |
| Don't reject | Correct | Type II error: missed it |
Alpha is the false alarm rate
The significance level, alpha, is the type one error rate. Let's prove it. We run ten thousand t tests on two groups drawn from the same distribution, so there's no real effect. Run it. About five percent of the tests still come out significant. Those are all false alarms.
import numpy as np
from scipy.stats import ttest_ind
rng = np.random.default_rng(0)
a = rng.normal(70, 10, size=(10_000, 30))
b = rng.normal(70, 10, size=(10_000, 30))
p = ttest_ind(a, b, axis=1).pvalue
print("false alarm rate:", (p < 0.05).mean())false alarm rate: 0.0543
Summary
A type one error is a false alarm, and its rate is alpha. A type two error is a missed effect. Next, power, the chance of catching a real effect.
Key takeaways- Type I: false alarm
- Type II: a miss
- α = false alarm rate