AI Hints
Home›Statistics & Math for ML Tutorial›Softmax and Entropy
Statistics & Math for ML · Lesson 107 of 138

Cross-Entropy

The loss for classifiers.

⏱ 2 min read▶ Video at 2:07:02🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A classifier says the picture is a cat with ninety percent confidence, and it is a cat. Good. Another says one percent, and it's a cat. Terrible. Cross entropy measures how much probability the model gave to the right answer.

Cross-Entropy
From the video: Cross-Entropy
  • 90% cat, and it's a cat
  • 1% cat, and it's a cat
  • How much went to the truth?

Minus log of the right answer

For one example, the cross entropy loss is minus the log of the probability given to the correct class. Run it. When the model is confident and right, the loss is tiny. At a half, it's about zero point seven. And when the model is confident but wrong, the loss is huge.

cross_entropy.pypython
import numpy as np

for p_correct in [0.9, 0.5, 0.1, 0.01]:
    loss = -np.log(p_correct)
    print(f"P(correct) = {p_correct:<5} loss = {loss:.2f}")
Output
P(correct) = 0.9   loss = 0.11
P(correct) = 0.5   loss = 0.69
P(correct) = 0.1   loss = 2.30
P(correct) = 0.01  loss = 4.61

Why not squared error?

Cross entropy punishes confident mistakes very strongly, which pushes the model to fix them fast. It also gives nice, simple gradients when paired with a sigmoid or softmax. For a whole dataset, we take the average loss over all examples. In PyTorch, it's called cross entropy loss. In scikit learn, it's log loss.

keyvalue
Loss−log P(correct class)
Confident and rightnear 0
Confident and wrongvery large
Also calledlog loss

Summary

Cross entropy is minus the log of the probability given to the right answer. It strongly punishes confident mistakes. Next, entropy itself, measuring surprise.

Key takeaways
  • −log(p_correct)
  • Punishes confident errors
  • Average over the data
Next up: Entropy — measuring surprise