Trustworthy Experiments

The new model scored three points better in the A B test. Time to celebrate? Not yet. Small groups produce random differences, and luck can look like progress. Statistics tells you how likely it is, that you are being fooled. In this lesson, you will run the numbers, in code, and see how much data you really need.

• Luck can look like progress
• Statistics measures the risk
• Enough data is not optional

The vocabulary of a test

A few words are worth knowing. The control group sees the current model. The treatment group sees the new one. The p value is the probability of seeing a difference at least this big, if there were really no difference at all. A small p value, usually below five percent, means luck is an unlikely explanation. Power is the chance of detecting a real improvement, when one exists. And sample size is how many users you need, to reach that power. Decide all of these before the test starts, not after.

The same result, three sample sizes

Here is a classic lesson, in code. A function compares two conversion rates, with a standard test, and returns a p value. We feed it the same result, fifty percent against fifty three percent, at three sample sizes. With two hundred users per group, the p value is huge, nothing can be concluded. With two thousand, it is close, but not enough. With twenty thousand, the same three point gap is clearly real. Then a second function estimates how many users are needed, to detect a given improvement. Detecting three points needs about four thousand per group, and detecting one point needs nearly forty thousand.

import numpy as np
from scipy.stats import norm

def ab_test(success_a, n_a, success_b, n_b):
    p_a, p_b = success_a / n_a, success_b / n_b
    pooled = (success_a + success_b) / (n_a + n_b)
    se = np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    return p_a, p_b, 2 * (1 - norm.cdf(abs(z)))

for n in (200, 2000, 20000):
    a, b, p = ab_test(int(n * 0.50), n, int(n * 0.53), n)
    verdict = "significant" if p < 0.05 else "not significant"
    print(f"{n:6d} users per group: {a:.0%} vs {b:.0%}   p-value {p:.3f}   {verdict}")

def users_needed(base, lift, alpha=0.05, power=0.8):
    p1, p2 = base, base + lift
    z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
    return int(np.ceil(z ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / lift ** 2))

print("users per group needed to detect +3 points from 50%:", users_needed(0.50, 0.03))
print("users per group needed to detect +1 point  from 50%:", users_needed(0.50, 0.01))

   200 users per group: 50% vs 53%   p-value 0.548   not significant
  2000 users per group: 50% vs 53%   p-value 0.058   not significant
 20000 users per group: 50% vs 53%   p-value 0.000   significant
users per group needed to detect +3 points from 50%: 4353
users per group needed to detect +1 point  from 50%: 39237

Mistakes that fool teams

A few classic mistakes fool teams. Peeking, checking the result every day, and stopping as soon as it looks good, which inflates false wins. Testing many metrics, and celebrating the one that happened to move. Stopping too early, with too little data. Ignoring novelty effects, where users react to anything new. And leaking users, between groups, so the comparison is unfair. The defences are simple. Fix the sample size and duration in advance, choose one main metric, and randomise properly.

Summary