Hypothesis Tests

A new menu raised spending from five dollars to five forty, in a test. Is that a real effect, or just luck? A hypothesis test answers a precise question. If the menu really did nothing, how often would we see a gap this big?

• Spending rose by forty cents
• Real or luck?
• How often would luck do this?

The p-value

We compare the two groups with a t test, which gives a p value. That's the chance of seeing a gap this big, if there's truly no effect. Small means it's hard to explain by luck. Now look at the last part. We run two thousand tests where nothing's really different. Run it. Around one in twenty come out as significant, just by luck. That's what a five percent threshold means, and it's why testing many things will always produce some false wins.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
old = rng.normal(5.00, 2.0, 200)
new = rng.normal(5.40, 2.0, 200)

t, p = stats.ttest_ind(new, old)
print(f"New average ${new.mean():.2f}, old ${old.mean():.2f}")
print(f"p-value: {p:.3f}")

same = [stats.ttest_ind(rng.normal(5, 2, 200), rng.normal(5, 2, 200)).pvalue for _ in range(2000)]
print(f"With no real effect, p < 0.05 happens {np.mean(np.array(same) < .05):.0%} of the time")

New average $5.22, old $5.03
p-value: 0.335
With no real effect, p < 0.05 happens 5% of the time

A test with no formula

Here's a beautiful way to see what a p value means, using real A/B test data. Shuffle the group labels at random, thousands of times, and see how often a fake gap beats the real gap. If it's rarely beaten, then the real gap is unlikely to be chance. Run it. Our real gap beats nearly all of the shuffles. No formula, and no assumptions, just shuffling. That's a permutation test.

import numpy as np
import pandas as pd

df = pd.read_csv("ab_test.csv")
a = df[df["group"] == "A"]["converted"].values
b = df[df["group"] == "B"]["converted"].values
gap = b.mean() - a.mean()

rng = np.random.default_rng(0)
both = np.concatenate([a, b])
fake = []
for _ in range(3000):
    rng.shuffle(both)
    fake.append(both[:len(b)].mean() - both[len(b):].mean())

print(f"Real gap: {gap:.4f}")
print(f"Shuffles beating it: {(np.abs(fake) >= abs(gap)).mean():.3f}")

Real gap: 0.0143
Shuffles beating it: 0.044

What a p-value is not

Three common errors. A p value isn't the chance that the result is a fluke. It isn't the size of the effect. And significant doesn't mean important. With huge data, a tiny useless effect becomes significant. So always report the effect size, and a range. Ask, is it big enough to matter?

Summary