Softmax
Many classes.
What if there are more than two classes, like cat, dog, and bird? A model gives each class a score. Softmax turns those scores into probabilities that are all positive, and add up to one.

- Cat, dog or bird?
- A score for each
- Softmax makes probabilities
Scores to probabilities
Softmax raises E to the power of each score, then divides by the total. Bigger scores get much bigger shares. Subtracting the largest score first keeps the numbers safe from overflowing. Run it. Cat has the highest score, so it gets most of the probability, and the three add up to one.
import numpy as np
def softmax(scores):
e = np.exp(scores - scores.max())
return e / e.sum()
scores = np.array([2.0, 1.0, 0.1])
probs = softmax(scores)
for name, p in zip(["cat", "dog", "bird"], probs):
print(f"{name}: {p:.3f}")
print("total:", probs.sum())cat: 0.659 dog: 0.242 bird: 0.099 total: 1.0
Where you see softmax
Softmax is the last layer of almost every classifier network, from digit recognizers to image models. Language models use it too. At each step, they score every word in their vocabulary, and softmax turns those scores into the probability of each possible next word.
| key | value |
|---|---|
| Input | any scores, called logits |
| Output | probabilities summing to 1 |
| Safe trick | subtract the max |
| Used in | classifiers, language models |
Summary
Softmax turns a list of scores into probabilities that add to one. Next, the loss we use to train classifiers, cross entropy.
Key takeaways- e^score ÷ total
- Adds to 1
- Last layer of classifiers