AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Probability in Models
Statistics & Math for ML · Lesson 134 of 138

KL Divergence

Distance between distributions.

⏱ 2 min read▶ Video at 2:40:20🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

How different are two probability distributions? For example, the true distribution of classes, and the one a model predicts. KL divergence measures that. It shows up in training language models, image generators, and much more.

KL Divergence
From the video: KL Divergence
  • How different are two distributions?
  • True vs predicted
  • KL divergence

Measuring the gap

KL divergence from Q to P adds up P times the log of P over Q. It's zero when the distributions are identical, and grows as they differ. Sci Pi's entropy function computes it when you give it two distributions. Run it. A close guess has a small divergence, and a bad guess has a large one.

kl.pypython
import numpy as np
from scipy.stats import entropy

p = np.array([0.7, 0.2, 0.1])
close = np.array([0.6, 0.25, 0.15])
bad = np.array([0.1, 0.2, 0.7])

print("by hand:", round((p * np.log(p / close)).sum(), 4))
print("close:  ", round(entropy(p, close), 4))
print("bad:    ", round(entropy(p, bad), 4))
Output
by hand: 0.0227
close:   0.0227
bad:     1.1675

Not quite a distance

KL divergence isn't symmetric. The divergence from P to Q usually differs from Q to P. It's also closely linked to cross entropy. Cross entropy equals the entropy of the true distribution, plus the KL divergence. So minimizing cross entropy is the same as minimizing KL divergence from the truth.

keyvalue
KL(P‖Q)Σ p log(p ÷ q)
Zeroidentical distributions
Not symmetricKL(P‖Q) ≠ KL(Q‖P)
Linkcross-entropy = H(P) + KL

Summary

KL divergence measures how far one distribution is from another. Minimizing cross entropy minimizes it. Next, checking whether a model's probabilities can be trusted.

Key takeaways
  • Σ p log(p/q)
  • scipy.stats.entropy(p, q)
  • Not symmetric
Next up: Calibration — can you trust the probability?