Effect Size
Big enough to matter?.
With a million users, an A/B test can find that a new button increases sign ups by zero point zero one percent, with a tiny p value. It's statistically significant. But is it worth anything? The p value can't tell you. You need the effect size.

- A million users
- A 0.01% improvement
- Significant, but does it matter?
Cohen's d
A popular effect size is Cohen's d. The difference in means, divided by the pooled standard deviation. It measures the difference in standard deviations, so it doesn't depend on sample size. Here, with a huge sample, the p value is tiny, but d is only about zero point zero two. Run it. Significant, but tiny.
import numpy as np
from scipy.stats import ttest_ind
rng = np.random.default_rng(0)
a = rng.normal(70.2, 10, 200_000)
b = rng.normal(70.0, 10, 200_000)
pooled = np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2)
d = (a.mean() - b.mean()) / pooled
print(f"p = {ttest_ind(a, b).pvalue:.1e}, d = {d:.3f}")p = 3.0e-10, d = 0.020
Reading d
As a rough guide, a d of about zero point two is small, zero point five is medium, and zero point eight or more is large. Always report the effect size, and a confidence interval, alongside the p value. Together, they tell you whether the effect is real, and whether it's big enough to care about.
| Cohen's d | Effect |
|---|---|
| 0.2 | small |
| 0.5 | medium |
| 0.8+ | large |
Summary
Effect size tells you how big a difference is, separately from whether it's significant. Report both. Next, common pitfalls in inference.
Key takeaways- d = diff ÷ pooled std
- Significant ≠ important
- Report effect sizes