Probability and Statistics

Machine learning doesn't give certainties, it gives good guesses. So you have to be comfortable with uncertainty. A few ideas from probability will protect you from mistakes that fool even smart, experienced people. Here's a famous one.

• Models give good guesses
• Be comfortable with uncertainty
• Don't get fooled

Why averages are stable

One die roll is wild, anything from one to six. But the average of many dice is remarkably stable. Here, we average one, two, ten, and one hundred dice, and we measure the spread. Run it. The spread keeps shrinking. This is why more data gives better estimates, and why a random forest's average is more stable than a single tree. Averages of many things settle down, and they form a bell curve. That's the central limit theorem.

import numpy as np

rng = np.random.default_rng(0)

rolls = rng.integers(1, 7, 100000)
print("Mean of one die:", rolls.mean().round(3))

for n in [1, 2, 10, 100]:
    means = rng.integers(1, 7, (20000, n)).mean(axis=1)
    print(f"Average of {n:>3} dice: spread {means.std():.3f}")

Mean of one die: 3.498
Average of   1 dice: spread 1.709
Average of   2 dice: spread 1.203
Average of  10 dice: spread 0.539
Average of 100 dice: spread 0.171

The test that fools doctors

Now a puzzle that fools most people. A disease affects one percent of people. A test is ninety nine percent accurate at spotting it, with a five percent false alarm rate. You test positive. What's the chance that you have it? Most people say around ninety nine. Let's compute it. Run it. It's only about seventeen percent! Why? Because healthy people vastly outnumber sick people, so most positive tests are false alarms. This is Bayes' rule, and it's why rare event models need such care.

disease = 0.01
true_positive = 0.99
false_positive = 0.05

both = disease * true_positive
false_alarm = (1 - disease) * false_positive

chance = both / (both + false_alarm)

print(f"Test is positive. Real chance of disease: {chance:.1%}")

Test is positive. Real chance of disease: 16.7%

Did the change really help?

Suppose you change the HomeWorth site, and valuations seem to rise. Is it real, or just luck? This is an A B test. We compare the old and new versions. A confidence interval gives a likely range for the true lift. And a p value says how often you would see a gap this large by pure chance, if there were no real effect. Run it. Look at the range. If it includes zero, you can't be sure there is a real gain. Patience in testing has saved companies from many expensive mistakes.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
old = rng.normal(100, 30, 200)
new = rng.normal(106, 30, 200)

result = stats.ttest_ind(new, old)
se = np.sqrt(old.var(ddof=1) / 200 + new.var(ddof=1) / 200)
gap = new.mean() - old.mean()

print(f"Measured lift: {gap:.1f}")
print(f"95% range: {gap - 1.96 * se:.1f} to {gap + 1.96 * se:.1f}")
print(f"p-value: {result.pvalue:.3f}")

Measured lift: 2.9
95% range: -3.0 to 8.7
p-value: 0.335

Summary