Label Smoothing

We tell a network that an image is one hundred percent a cat. But nothing's really one hundred percent. Pushing a model to be perfectly certain makes its weights grow, without limit, and it becomes overconfident. What if we taught it a bit of humility?

• One hundred percent a cat
• Certainty makes weights grow
• Teach some humility

Smoothing the targets

Label smoothing changes the target slightly. Instead of one for the right class and zero for everything else, it uses about point nine, and shares out the rest. We compare a very confident model and a moderately confident one. Run it. With a plain loss, the very sure model gets a loss of zero. It's being rewarded for its total certainty. With smoothing, that same model gets a much higher loss, of point eight, and the moderate one scores far better. So smoothing punishes overconfidence. It's a single argument in PyTorch, and it nearly always helps a bit on image and language tasks.

import torch
from torch.nn.functional import cross_entropy

sure = torch.tensor([[12.0, 0.0, 0.0]])
ok = torch.tensor([[3.0, 0.0, 0.0]])
label = torch.tensor([0])

for name, logits in [("Very sure", sure), ("Moderately sure", ok)]:
    plain = cross_entropy(logits, label).item()
    soft = cross_entropy(logits, label, label_smoothing=0.1).item()
    print(f"{name:<16} plain {plain:.4f}, smoothed {soft:.4f}")

Very sure        plain 0.0000, smoothed 0.8000
Moderately sure  plain 0.0949, smoothed 0.2949

Other ways to regularise labels

Label smoothing has some relatives. Mixup blends two images and their labels together, so the network sees a picture that's sixty percent dog, and forty percent cat. CutMix pastes a patch from one image onto another. Both make models more robust, and better calibrated. They're standard in nearly all modern image training recipes.

Summary