AI Hints
Home›Statistics & Math for ML Tutorial›Softmax and Entropy
Statistics & Math for ML · Lesson 108 of 138

Entropy

Measuring surprise.

⏱ 2 min read▶ Video at 2:08:15🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

If a coin always lands heads, flipping it tells you nothing new. If it's fair, every flip is a surprise. Entropy measures the average surprise of a random event, or how uncertain it is. It's the idea behind cross entropy, and behind decision trees.

Entropy
From the video: Entropy
  • A coin that's always heads
  • A fair coin
  • Entropy: average surprise

Entropy in bits

Entropy adds up minus P times log two of P, over all outcomes. We measure it in bits. Run it. A fair coin has exactly one bit of entropy. A ninety ten coin has less, because it's more predictable. And a coin that always lands heads has zero. A fair eight sided die has three bits.

entropy.pypython
import numpy as np

def entropy(probs):
    p = np.array([x for x in probs if x > 0])
    return -(p * np.log2(p)).sum()

print("fair coin:     ", entropy([0.5, 0.5]))
print("90/10 coin:    ", round(entropy([0.9, 0.1]), 3))
print("always heads:  ", entropy([1.0]) + 0)
print("8-sided die:   ", entropy([1 / 8] * 8))
Output
fair coin:      1.0
90/10 coin:     0.469
always heads:   0.0
8-sided die:    3.0

Entropy in ML

Decision trees pick splits that reduce entropy the most. That's called information gain. Cross entropy measures the surprise when you use the model's probabilities to predict the truth. And KL divergence, the difference between them, measures how far one distribution is from another.

keyvalue
Entropy−Σ p log p
Maxwhen all outcomes are equal
Treessplit to reduce entropy
KL divergencedistance between distributions

Summary

Entropy measures uncertainty, in bits. It's highest when every outcome is equally likely. Next, finding the parameters that make the data most likely.

Key takeaways
  • −Σ p log₂ p
  • Fair coin = 1 bit
  • Trees use information gain
Next up: Maximum Likelihood — the most likely parameters