AI Hints
Home›Statistics & Math for ML Tutorial›Softmax and Entropy
Statistics & Math for ML · Lesson 106 of 138

Softmax

Many classes.

⏱ 2 min read▶ Video at 2:05:53🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

What if there are more than two classes, like cat, dog, and bird? A model gives each class a score. Softmax turns those scores into probabilities that are all positive, and add up to one.

Softmax
From the video: Softmax
  • Cat, dog or bird?
  • A score for each
  • Softmax makes probabilities

Scores to probabilities

Softmax raises E to the power of each score, then divides by the total. Bigger scores get much bigger shares. Subtracting the largest score first keeps the numbers safe from overflowing. Run it. Cat has the highest score, so it gets most of the probability, and the three add up to one.

softmax.pypython
import numpy as np

def softmax(scores):
    e = np.exp(scores - scores.max())
    return e / e.sum()

scores = np.array([2.0, 1.0, 0.1])
probs = softmax(scores)
for name, p in zip(["cat", "dog", "bird"], probs):
    print(f"{name}: {p:.3f}")
print("total:", probs.sum())
Output
cat: 0.659
dog: 0.242
bird: 0.099
total: 1.0

Where you see softmax

Softmax is the last layer of almost every classifier network, from digit recognizers to image models. Language models use it too. At each step, they score every word in their vocabulary, and softmax turns those scores into the probability of each possible next word.

keyvalue
Inputany scores, called logits
Outputprobabilities summing to 1
Safe tricksubtract the max
Used inclassifiers, language models

Summary

Softmax turns a list of scores into probabilities that add to one. Next, the loss we use to train classifiers, cross entropy.

Key takeaways
  • e^score ÷ total
  • Adds to 1
  • Last layer of classifiers
Next up: Cross-Entropy — the loss for classifiers