AI Hints
Statistics & Math for ML · Lesson 61 of 138

Type I and II Errors

False alarms and misses.

⏱ 2 min read▶ Video at 1:11:07🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A hypothesis test can be wrong in two ways. It can raise a false alarm, saying there's an effect when there isn't. Or it can miss a real effect. These two mistakes have names, type one and type two errors.

Type I and II Errors
From the video: Type I and II Errors
  • Tests can be wrong
  • False alarms
  • Missed effects

The four outcomes

There are four possibilities. If there's no real effect, and we don't reject, good. If there's no effect, but we reject, that's a type one error, a false alarm. If there is an effect, and we reject, good. And if there's an effect, but we miss it, that's a type two error.

No real effectReal effect
Reject the nullType I error: false alarmCorrect: found it
Don't rejectCorrectType II error: missed it

Alpha is the false alarm rate

The significance level, alpha, is the type one error rate. Let's prove it. We run ten thousand t tests on two groups drawn from the same distribution, so there's no real effect. Run it. About five percent of the tests still come out significant. Those are all false alarms.

alpha.pypython
import numpy as np
from scipy.stats import ttest_ind

rng = np.random.default_rng(0)
a = rng.normal(70, 10, size=(10_000, 30))
b = rng.normal(70, 10, size=(10_000, 30))

p = ttest_ind(a, b, axis=1).pvalue
print("false alarm rate:", (p < 0.05).mean())
Output
false alarm rate: 0.0543

Summary

A type one error is a false alarm, and its rate is alpha. A type two error is a missed effect. Next, power, the chance of catching a real effect.

Key takeaways
  • Type I: false alarm
  • Type II: a miss
  • α = false alarm rate
Next up: Statistical Power — sample size