Knowledge Distillation

A great teacher doesn't just tell you the answer. They tell you how they see the problem. Show a big model a picture of a two, and it might say, ninety percent a two, nine percent a seven, and one percent a three. That small nine percent is real knowledge. It's saying that twos can look like sevens. That's what a student model can learn from.

• A great teacher shows their thinking
• Ninety percent a two, nine a seven
• That nine percent is knowledge

How distillation works

Train a big teacher model. Then soften its outputs, using a high temperature, so that the small probabilities become visible. Train a small student, to match those soft outputs as well as the true labels. The student receives far more information from each example than a single right answer would give.

1. Train a teacherbig and accurate
2. Soften outputshigh temperature
3. Train a studentmatch the teacher
4. Mix in true labelsa small share
5. Ship the studentsmall and fast

Where it pays off

Distillation has shaped modern A I. It's how many small language models are made, trained on the outputs of larger ones. On phones, it delivers much of a big model's accuracy, in a tiny fraction of its size. It needs care, though. The temperature and the mix need tuning, and a poor setup can do worse than plain training, so always compare with a student trained alone.

Summary