Training and Gradient Descent

A modern model can have billions of weights. Nobody sets them by hand. They are found automatically, by an elegant idea, called gradient descent. In this lesson, you will see it as a ball rolling down a hill, and you will understand why training takes so much computing power.

• Billions of weights, set automatically
• The idea: roll downhill
• Why training is so expensive

Measuring the mistake

First, we need a way to measure how wrong the model is. That number is called the loss. If the model guesses well, the loss is small. If it guesses badly, the loss is big. For a language model, the loss measures how surprised the model was by the real next word. Training has a single goal, to make the loss as small as possible.

Rolling downhill

Now picture the loss as a landscape. Each point on the landscape is one setting of the weights, and the height is the loss. We want to reach the lowest valley. But we cannot see the whole landscape. So we do something clever. We feel the slope under our feet, and take a small step downhill. Then again, and again. Watch the ball. Every step lowers the error, until the ball settles in the valley. This is gradient descent.

The learning rate

How big should each step be? That is the learning rate. If the steps are too small, learning is painfully slow. If the steps are too big, the ball overshoots the valley, and bounces around. A good learning rate takes confident steps, that settle in smoothly. Choosing it well is one of the most important decisions in training.

Backpropagation in one minute

But a network has billions of weights. How do we know which way to nudge each one? The answer is backpropagation. First, the data flows forward, and the model makes a guess. Then the error flows backward, through every layer. Each weight learns how much it contributed to the mistake, and which way to change. Then all weights nudge a tiny bit together. Forward, backward, nudge. Repeat trillions of times.

1. Forwarddata flows through, a guess is made
2. Measurecompute the loss
3. Backwardblame flows to each weight
4. Nudgeevery weight improves a little

Why training is expensive

Here is why training large models is so costly. Each model has more weights than the last. A model from twenty eighteen had about a hundred and ten million weights. GPT two had one and a half billion. GPT three, in twenty twenty, had one hundred and seventy five billion. Every single weight is nudged, again and again, over trillions of words. This needs thousands of special chips, running for weeks. That is why only a few companies train the biggest models from scratch.

Summary