AI Hints
Home›Statistics & Math for ML Tutorial›Testing in Practice
Statistics & Math for ML · Lesson 58 of 138

A/B Testing

Which version wins?.

⏱ 2 min read▶ Video at 1:07:17🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Tech companies test everything. Half the visitors see version A of a page, and half see version B. Then they check which one gets more sign ups. That's an A/B test, and it's a hypothesis test in disguise.

A/B Testing
From the video: A/B Testing
  • Half see A, half see B
  • Which gets more sign-ups?
  • A hypothesis test

Comparing two rates

Version A got two hundred sign ups from two thousand visitors. Version B got two hundred and fifty from two thousand. We compare the two rates with a two proportion test. The pooled rate gives the standard error under the null. Run it. B's rate is two and a half points higher, and the p value is about one percent.

ab_test.pypython
import numpy as np
from scipy.stats import norm

a, n_a = 200, 2000
b, n_b = 250, 2000

p_a, p_b = a / n_a, b / n_b
pool = (a + b) / (n_a + n_b)
se = np.sqrt(pool * (1 - pool) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se

print(f"A: {p_a:.1%}  B: {p_b:.1%}")
print(f"z = {z:.2f}, p = {2 * norm.sf(abs(z)):.4f}")
Output
A: 10.0%  B: 12.5%
z = 2.50, p = 0.0124

Good A/B testing habits

Decide your sample size before you start, and don't stop early just because the result looks good. Randomly assign visitors, so the groups are fair. Test one change at a time. And remember that significant isn't the same as important. A tiny improvement can be significant with enough visitors.

keyvalue
Randomassign visitors fairly
Plansample size in advance
Don't peekand stop early
Significantisn't always important

Summary

An A/B test compares two versions with a hypothesis test. Plan it in advance, and don't stop early. Next, testing categories with chi square.

Key takeaways
  • Two proportions
  • Pooled standard error
  • Plan, then run
Next up: Chi-Square Test — categories