The p-value
How surprising?.
A friend flips a coin a hundred times, and gets sixty heads. Is the coin unfair? A fair coin would usually give about fifty. But sixty could still happen by chance. How surprised should we be?

- 100 flips, 60 heads
- Is the coin unfair?
- How surprised should we be?
Simulating the null
Let's assume the null. The coin is fair. We simulate a hundred thousand experiments of a hundred fair flips, and count how often we see a result at least as extreme as sixty. That means sixty or more, or forty or fewer. Run it. About five and a half percent of the time.
import numpy as np
rng = np.random.default_rng(0)
heads = rng.binomial(100, 0.5, 100_000)
extreme = (heads >= 60) | (heads <= 40)
print("p-value ~", extreme.mean().round(4))p-value ~ 0.0566
The exact test
Sci Pi's binom test does it exactly. Run it. A p value of about zero point zero five seven. That's just above the usual five percent line, so we don't reject the null. Sixty heads is a bit unusual, but not unusual enough to call the coin unfair.
from scipy.stats import binomtest
result = binomtest(60, n=100, p=0.5)
print("p-value:", round(result.pvalue, 4))p-value: 0.0569
What a p-value is, and is not
The p value is the chance of seeing data at least this extreme, if the null were true. It is not the chance that the null is true. A small p value means the data would be surprising under the null. And not significant doesn't prove there's no effect. It just means we don't have strong enough evidence.
| key | value |
|---|---|
| p-value | P(data this extreme | null) |
| Not | P(null is true) |
| p < 0.05 | significant |
| p ≥ 0.05 | not enough evidence |
Summary
The p value is how surprising the data would be if nothing were going on. Below five percent, we usually reject the null. Next, the t test.
Key takeaways- Simulate the null
- binomtest
- p is not P(null)