Broken Experiments
Checks before you trust a test.
Your test shows a great win for the new version. But something's slightly off. The new version got a little less traffic than the old one, though the split was meant to be fifty fifty. It's only a one percent gap. Should you care? You absolutely should. That small gap may mean that the entire experiment is broken.
- A great win
- Slightly fewer users in B
- A small gap, a big warning
Sample ratio mismatch
We expected an even split. We got about fifty thousand against forty nine thousand. A chi square test asks whether that could be chance. Run it. The p value is tiny. So this isn't chance. Something's removing users from one group. Perhaps the new version crashes on old phones, so those users never get recorded. That means the remaining users in that group are not comparable, and all of your results are invalid. Big tech companies run this check automatically on every test, and they find problems in a surprising share of them.
from scipy import stats
a, b = 50210, 49120
expected = (a + b) / 2
p = stats.chisquare([a, b], [expected, expected]).pvalue
print(f"Group sizes: {a:,} and {b:,}")
print(f"p-value: {p:.2e}")
print("Stop! The split is broken." if p < 0.001 else "Split looks fine.")Group sizes: 50,210 and 49,120 p-value: 5.43e-04 Stop! The split is broken.
Summary
Check the split, before you look at the result. Next, how to test your testing system itself.
Key takeaways- Group sizes should match the plan
- A chi-square test checks this
- A mismatch invalidates the test