A/B Testing
Which version wins?.
Tech companies test everything. Half the visitors see version A of a page, and half see version B. Then they check which one gets more sign ups. That's an A/B test, and it's a hypothesis test in disguise.

- Half see A, half see B
- Which gets more sign-ups?
- A hypothesis test
Comparing two rates
Version A got two hundred sign ups from two thousand visitors. Version B got two hundred and fifty from two thousand. We compare the two rates with a two proportion test. The pooled rate gives the standard error under the null. Run it. B's rate is two and a half points higher, and the p value is about one percent.
import numpy as np
from scipy.stats import norm
a, n_a = 200, 2000
b, n_b = 250, 2000
p_a, p_b = a / n_a, b / n_b
pool = (a + b) / (n_a + n_b)
se = np.sqrt(pool * (1 - pool) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se
print(f"A: {p_a:.1%} B: {p_b:.1%}")
print(f"z = {z:.2f}, p = {2 * norm.sf(abs(z)):.4f}")A: 10.0% B: 12.5% z = 2.50, p = 0.0124
Good A/B testing habits
Decide your sample size before you start, and don't stop early just because the result looks good. Randomly assign visitors, so the groups are fair. Test one change at a time. And remember that significant isn't the same as important. A tiny improvement can be significant with enough visitors.
| key | value |
|---|---|
| Random | assign visitors fairly |
| Plan | sample size in advance |
| Don't peek | and stop early |
| Significant | isn't always important |
Summary
An A/B test compares two versions with a hypothesis test. Plan it in advance, and don't stop early. Next, testing categories with chi square.
Key takeaways- Two proportions
- Pooled standard error
- Plan, then run