Testing Your Tests

Here's a strange experiment. Show both groups exactly the same thing, and run the analysis. There can't be a real difference. So if your system reports a significant result more than five percent of the time, there's a bug in how you're measuring. It's called an AA test, and it can reveal mistakes that have been hiding for years.

• Show both groups the same thing
• There can't be a real difference
• It reveals hidden bugs

The classic counting mistake

First, a correct AA test, run two thousand times. Then a very common mistake. We randomise users, but we analyse page views. Each user views eight pages, and heavy users are very different from light ones. The views aren't independent. Run it. The correct test gets five percent false alarms, as promised. The mistaken test gets nearly forty eight percent! Almost half of your experiments would declare a winner, when nothing changed at all. That one error has wasted huge amounts of company time. Always analyse at the same level at which you randomised.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
ps = []
for _ in range(2000):
    a, b = rng.normal(10, 3, 500), rng.normal(10, 3, 500)
    ps.append(stats.ttest_ind(a, b).pvalue)
ps = np.array(ps)
print(f"False alarms at 5%: {(ps < .05).mean():.1%}")

heavy = []
for _ in range(2000):
    u = rng.lognormal(0, 1, 500)
    a = np.repeat(u[:250], 8) + rng.normal(0, .3, 2000)
    b = np.repeat(u[250:], 8) + rng.normal(0, .3, 2000)
    heavy.append(stats.ttest_ind(a, b).pvalue)
print(f"False alarms, counting views not users: {(np.array(heavy) < .05).mean():.1%}")

False alarms at 5%: 5.2%
False alarms, counting views not users: 47.9%

An experiment checklist

Before you trust a result, check these. Was the split the expected ratio? Does an AA test give five percent false alarms? Are you analysing at the level where you randomised? Are the groups similar on past behaviour? Did you run for the planned length? And is the effect big enough to matter? If you can't tick all of those, don't ship.

Summary