Entropy
Measuring surprise.
If a coin always lands heads, flipping it tells you nothing new. If it's fair, every flip is a surprise. Entropy measures the average surprise of a random event, or how uncertain it is. It's the idea behind cross entropy, and behind decision trees.

- A coin that's always heads
- A fair coin
- Entropy: average surprise
Entropy in bits
Entropy adds up minus P times log two of P, over all outcomes. We measure it in bits. Run it. A fair coin has exactly one bit of entropy. A ninety ten coin has less, because it's more predictable. And a coin that always lands heads has zero. A fair eight sided die has three bits.
import numpy as np
def entropy(probs):
p = np.array([x for x in probs if x > 0])
return -(p * np.log2(p)).sum()
print("fair coin: ", entropy([0.5, 0.5]))
print("90/10 coin: ", round(entropy([0.9, 0.1]), 3))
print("always heads: ", entropy([1.0]) + 0)
print("8-sided die: ", entropy([1 / 8] * 8))fair coin: 1.0 90/10 coin: 0.469 always heads: 0.0 8-sided die: 3.0
Entropy in ML
Decision trees pick splits that reduce entropy the most. That's called information gain. Cross entropy measures the surprise when you use the model's probabilities to predict the truth. And KL divergence, the difference between them, measures how far one distribution is from another.
| key | value |
|---|---|
| Entropy | −Σ p log p |
| Max | when all outcomes are equal |
| Trees | split to reduce entropy |
| KL divergence | distance between distributions |
Summary
Entropy measures uncertainty, in bits. It's highest when every outcome is equally likely. Next, finding the parameters that make the data most likely.
Key takeaways- −Σ p log₂ p
- Fair coin = 1 bit
- Trees use information gain