Statistical Power
Sample size.
If an effect is real, how likely is your test to find it? That's called power. A study with low power will often miss real effects, and that's a waste of everyone's time.

- If the effect is real
- Will we find it?
- That's power
Power grows with sample size
Now there's a real effect. The app really adds three points. For different group sizes, we simulate two thousand studies, and count how often the test finds the effect. Run it. With twenty people per group, we usually miss it. With two hundred, we find it about eighty four percent of the time.
import numpy as np
from scipy.stats import ttest_ind
rng = np.random.default_rng(1)
for n in [20, 50, 100, 200]:
a = rng.normal(73, 10, size=(2_000, n))
b = rng.normal(70, 10, size=(2_000, n))
power = (ttest_ind(a, b, axis=1).pvalue < 0.05).mean()
print(f"n = {n:3d} per group: power {power:.0%}")n = 20 per group: power 15% n = 50 per group: power 34% n = 100 per group: power 56% n = 200 per group: power 84%
Planning a study
Before collecting data, decide how small an effect you care about, and calculate the sample size that gives you about eighty percent power. Smaller effects need much bigger samples. Libraries like stats models have functions that do this calculation for you.
| key | value |
|---|---|
| Power | P(detect a real effect) |
| Target | about 80% |
| Small effects | need big samples |
| Plan | before collecting data |
Summary
Power is the chance of detecting a real effect. It grows with sample size and effect size. Aim for about eighty percent. Next, the danger of running many tests.
Key takeaways- Power = 1 − P(type II)
- More data, more power
- Plan for 80%