AI Hints
Home›Statistics & Math for ML Tutorial›Effect Size and Pitfalls
Statistics & Math for ML · Lesson 64 of 138

Effect Size

Big enough to matter?.

⏱ 2 min read▶ Video at 1:14:45🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

With a million users, an A/B test can find that a new button increases sign ups by zero point zero one percent, with a tiny p value. It's statistically significant. But is it worth anything? The p value can't tell you. You need the effect size.

Effect Size
From the video: Effect Size
  • A million users
  • A 0.01% improvement
  • Significant, but does it matter?

Cohen's d

A popular effect size is Cohen's d. The difference in means, divided by the pooled standard deviation. It measures the difference in standard deviations, so it doesn't depend on sample size. Here, with a huge sample, the p value is tiny, but d is only about zero point zero two. Run it. Significant, but tiny.

cohen.pypython
import numpy as np
from scipy.stats import ttest_ind

rng = np.random.default_rng(0)
a = rng.normal(70.2, 10, 200_000)
b = rng.normal(70.0, 10, 200_000)

pooled = np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2)
d = (a.mean() - b.mean()) / pooled
print(f"p = {ttest_ind(a, b).pvalue:.1e}, d = {d:.3f}")
Output
p = 3.0e-10, d = 0.020

Reading d

As a rough guide, a d of about zero point two is small, zero point five is medium, and zero point eight or more is large. Always report the effect size, and a confidence interval, alongside the p value. Together, they tell you whether the effect is real, and whether it's big enough to care about.

Cohen's dEffect
0.2small
0.5medium
0.8+large

Summary

Effect size tells you how big a difference is, separately from whether it's significant. Report both. Next, common pitfalls in inference.

Key takeaways
  • d = diff ÷ pooled std
  • Significant ≠ important
  • Report effect sizes
Next up: Inference Pitfalls — p-hacking and more