Cross-Entropy
The loss for classifiers.
A classifier says the picture is a cat with ninety percent confidence, and it is a cat. Good. Another says one percent, and it's a cat. Terrible. Cross entropy measures how much probability the model gave to the right answer.

- 90% cat, and it's a cat
- 1% cat, and it's a cat
- How much went to the truth?
Minus log of the right answer
For one example, the cross entropy loss is minus the log of the probability given to the correct class. Run it. When the model is confident and right, the loss is tiny. At a half, it's about zero point seven. And when the model is confident but wrong, the loss is huge.
import numpy as np
for p_correct in [0.9, 0.5, 0.1, 0.01]:
loss = -np.log(p_correct)
print(f"P(correct) = {p_correct:<5} loss = {loss:.2f}")P(correct) = 0.9 loss = 0.11 P(correct) = 0.5 loss = 0.69 P(correct) = 0.1 loss = 2.30 P(correct) = 0.01 loss = 4.61
Why not squared error?
Cross entropy punishes confident mistakes very strongly, which pushes the model to fix them fast. It also gives nice, simple gradients when paired with a sigmoid or softmax. For a whole dataset, we take the average loss over all examples. In PyTorch, it's called cross entropy loss. In scikit learn, it's log loss.
| key | value |
|---|---|
| Loss | −log P(correct class) |
| Confident and right | near 0 |
| Confident and wrong | very large |
| Also called | log loss |
Summary
Cross entropy is minus the log of the probability given to the right answer. It strongly punishes confident mistakes. Next, entropy itself, measuring surprise.
Key takeaways- −log(p_correct)
- Punishes confident errors
- Average over the data