AI Hints
Statistics & Math for ML · Lesson 62 of 138

Statistical Power

Sample size.

⏱ 2 min read▶ Video at 1:12:22🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

If an effect is real, how likely is your test to find it? That's called power. A study with low power will often miss real effects, and that's a waste of everyone's time.

Statistical Power
From the video: Statistical Power
  • If the effect is real
  • Will we find it?
  • That's power

Power grows with sample size

Now there's a real effect. The app really adds three points. For different group sizes, we simulate two thousand studies, and count how often the test finds the effect. Run it. With twenty people per group, we usually miss it. With two hundred, we find it about eighty four percent of the time.

power.pypython
import numpy as np
from scipy.stats import ttest_ind

rng = np.random.default_rng(1)
for n in [20, 50, 100, 200]:
    a = rng.normal(73, 10, size=(2_000, n))
    b = rng.normal(70, 10, size=(2_000, n))
    power = (ttest_ind(a, b, axis=1).pvalue < 0.05).mean()
    print(f"n = {n:3d} per group: power {power:.0%}")
Output
n =  20 per group: power 15%
n =  50 per group: power 34%
n = 100 per group: power 56%
n = 200 per group: power 84%

Planning a study

Before collecting data, decide how small an effect you care about, and calculate the sample size that gives you about eighty percent power. Smaller effects need much bigger samples. Libraries like stats models have functions that do this calculation for you.

keyvalue
PowerP(detect a real effect)
Targetabout 80%
Small effectsneed big samples
Planbefore collecting data

Summary

Power is the chance of detecting a real effect. It grows with sample size and effect size. Aim for about eighty percent. Next, the danger of running many tests.

Key takeaways
  • Power = 1 − P(type II)
  • More data, more power
  • Plan for 80%
Next up: Many Tests — the multiple testing trap