Softmax and Cross Entropy

For regression, mean squared error made sense. But when a classifier says, I'm ninety percent sure that it's a cat, and it's right, how do we score that? And what if it was ninety percent sure, and wrong? Good scoring should punish confident mistakes much harder than timid ones.

• How do we score a classifier?
• Confident and wrong should hurt
• Enter cross entropy

From logits to loss

The network gives three raw logits. Softmax turns them into chances that add up to one. Then cross entropy takes the chance given to the correct answer, and computes minus the log of it. If the correct class had a high chance, the loss is small. If it had a tiny chance, the loss is huge. Run it. Being right costs very little. Being wrong costs far more. The last line shows PyTorch's cross entropy giving the same number. It takes logits directly, which is more stable.

import torch

logits = torch.tensor([3.2, 1.3, 0.2])
chances = torch.softmax(logits, dim=0)
print("Logits: ", logits.tolist())
print("Chances:", chances.round(decimals=3).tolist())

true_class = 0
print("Loss if right:", (-torch.log(chances[true_class])).item())
print("Loss if wrong:", (-torch.log(chances[2])).item())

loss = torch.nn.functional.cross_entropy(logits.unsqueeze(0), torch.tensor([0]))
print("PyTorch cross entropy:", loss.item())

Logits:  [3.200000047683716, 1.2999999523162842, 0.20000000298023224]
Chances: [0.8339999914169312, 0.125, 0.041999999433755875]
Loss if right: 0.18178442120552063
Loss if wrong: 3.1817843914031982
PyTorch cross entropy: 0.18178442120552063

Quick check

A question. A ten class model has random weights. What loss would you expect, at the very start of training? A, zero. B, about two point three. C, about ten. D, about one hundred. The answer is B. Random guessing gives each class a one in ten chance, and minus the log of point one is about two point three. This makes a great sanity check. If your first loss is far from this, then something's broken.

Summary