Why Statistics Matters

The founder storms in. The Oslo cafe had a brilliant month, and Cairo had a terrible one. Should we give the Oslo manager a bonus, and fire the one in Cairo? Before you say anything, let's run an experiment. What if nothing was really different at all?

• Reward Oslo? Fire Cairo?
• Maybe nothing is different
• Let's test that

Twelve identical cafes

Let's simulate twelve cafes, which are absolutely identical. Same customers, same skill, and the same average of one hundred sales per day. We average thirty days for each. Run it. They're not equal! Some look clearly better than others, and the gap between best and worst is large. And that's pure luck. Without statistics, you'd reward the lucky, and punish the unlucky. That's a mistake that's made every day in business.

import numpy as np

rng = np.random.default_rng(1)
cafes = rng.normal(100, 20, (12, 30)).mean(axis=1)

print("Twelve identical cafes, thirty days each:")
print(cafes.round(1))
print("Best minus worst:", round(cafes.max() - cafes.min(), 1))

Twelve identical cafes, thirty days each:
[ 98.7  99.4  98.1  99.7  96.   99.3 100.2  93.   95.4  99.6  96.1  99.4]
Best minus worst: 7.1

The big ideas of statistics

Statistics has a few big ideas. Your data is a sample of a bigger world. Any number you compute will vary from sample to sample. Bigger samples vary less. And statistics measures how much of what you see could just be chance. It doesn't remove uncertainty. It tells you how much you've got.

How sample size helps

Here's that idea in numbers. We take many samples of customers, and look at how far off the average is, for each size. Run it. With ten customers, your estimate can be far off. With a hundred, it's a lot closer. With a thousand, it's very tight. But look carefully. Ten times the data only makes the error about three times smaller. It follows a square root rule, and that decides how much data you really need.

import numpy as np
import pandas as pd

df = pd.read_csv("customers.csv").drop_duplicates()
true = df["avg_spend"].mean()
rng = np.random.default_rng(0)

for n in [10, 100, 1000]:
    means = [df["avg_spend"].sample(n, random_state=int(s)).mean() for s in rng.integers(0, 1e6, 300)]
    print(f"sample {n:>4}  average {np.mean(means):.2f}  typical miss {np.std(means):.2f}")
print("True mean:", round(true, 2))

sample   10  average 11.05  typical miss 1.13
sample  100  average 11.06  typical miss 0.33
sample 1000  average 11.04  typical miss 0.11
True mean: 11.03

Summary