AI Hints
Statistics & Math for ML · Lesson 56 of 138

The p-value

How surprising?.

⏱ 2 min read▶ Video at 1:04:39🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A friend flips a coin a hundred times, and gets sixty heads. Is the coin unfair? A fair coin would usually give about fifty. But sixty could still happen by chance. How surprised should we be?

The p-value
From the video: The p-value
  • 100 flips, 60 heads
  • Is the coin unfair?
  • How surprised should we be?

Simulating the null

Let's assume the null. The coin is fair. We simulate a hundred thousand experiments of a hundred fair flips, and count how often we see a result at least as extreme as sixty. That means sixty or more, or forty or fewer. Run it. About five and a half percent of the time.

p_sim.pypython
import numpy as np

rng = np.random.default_rng(0)
heads = rng.binomial(100, 0.5, 100_000)

extreme = (heads >= 60) | (heads <= 40)
print("p-value ~", extreme.mean().round(4))
Output
p-value ~ 0.0566

The exact test

Sci Pi's binom test does it exactly. Run it. A p value of about zero point zero five seven. That's just above the usual five percent line, so we don't reject the null. Sixty heads is a bit unusual, but not unusual enough to call the coin unfair.

p_exact.pypython
from scipy.stats import binomtest

result = binomtest(60, n=100, p=0.5)
print("p-value:", round(result.pvalue, 4))
Output
p-value: 0.0569

What a p-value is, and is not

The p value is the chance of seeing data at least this extreme, if the null were true. It is not the chance that the null is true. A small p value means the data would be surprising under the null. And not significant doesn't prove there's no effect. It just means we don't have strong enough evidence.

keyvalue
p-valueP(data this extreme | null)
NotP(null is true)
p < 0.05significant
p ≥ 0.05not enough evidence

Summary

The p value is how surprising the data would be if nothing were going on. Below five percent, we usually reject the null. Next, the t test.

Key takeaways
  • Simulate the null
  • binomtest
  • p is not P(null)
Next up: The t-Test — comparing two groups