KL Divergence
Distance between distributions.
How different are two probability distributions? For example, the true distribution of classes, and the one a model predicts. KL divergence measures that. It shows up in training language models, image generators, and much more.

- How different are two distributions?
- True vs predicted
- KL divergence
Measuring the gap
KL divergence from Q to P adds up P times the log of P over Q. It's zero when the distributions are identical, and grows as they differ. Sci Pi's entropy function computes it when you give it two distributions. Run it. A close guess has a small divergence, and a bad guess has a large one.
import numpy as np
from scipy.stats import entropy
p = np.array([0.7, 0.2, 0.1])
close = np.array([0.6, 0.25, 0.15])
bad = np.array([0.1, 0.2, 0.7])
print("by hand:", round((p * np.log(p / close)).sum(), 4))
print("close: ", round(entropy(p, close), 4))
print("bad: ", round(entropy(p, bad), 4))by hand: 0.0227 close: 0.0227 bad: 1.1675
Not quite a distance
KL divergence isn't symmetric. The divergence from P to Q usually differs from Q to P. It's also closely linked to cross entropy. Cross entropy equals the entropy of the true distribution, plus the KL divergence. So minimizing cross entropy is the same as minimizing KL divergence from the truth.
| key | value |
|---|---|
| KL(P‖Q) | Σ p log(p ÷ q) |
| Zero | identical distributions |
| Not symmetric | KL(P‖Q) ≠ KL(Q‖P) |
| Link | cross-entropy = H(P) + KL |
Summary
KL divergence measures how far one distribution is from another. Minimizing cross entropy minimizes it. Next, checking whether a model's probabilities can be trusted.
Key takeaways- Σ p log(p/q)
- scipy.stats.entropy(p, q)
- Not symmetric