The Big Idea: Predict the Next Word
The surprisingly simple trick behind modern language models.
Here is the secret that surprises almost everyone. Behind all the magic of a language model, there is one very simple job. Guess the next word. That is it. In this lesson, we will see it with our own eyes.
- One simple job: guess the next word
- Repeat it, again and again
- Watch it happen, step by step
A real conversation
First, a conversation. Someone types, write a one line story about a dragon who is afraid of fire. And then the assistant answers, word by word. Watch how the reply appears one piece at a time. It is not copying a story from the internet. It is building a new sentence, piece by piece.
Guessing the next word
Now let's look inside, with real numbers from a small open model called GPT two. After the words, once upon a time, there was a little, the most likely next word is girl, with about seven percent. Notice that even the best guess is not very sure. Boy, and man, are also possible. The model picks one, adds it to the sentence, and asks the same question again. After, girl, the word who is most likely. Then it asks again. Next word, next word, next word. Do this hundreds of times, and a whole story appears.
Where do the chances come from?
But where do these chances come from? From a neural network, a huge web of tiny connected units, loosely inspired by the brain. Words go in on the left. Signals flow through many layers, and each layer transforms the information a little. And on the right, the network outputs a chance for every possible next word. In this picture there are only a handful of units. A modern model has billions of connections, and that scale is a big part of the magic.
Why does this simple trick work?
Why does such a simple trick produce such smart behaviour? Because to predict the next word well, the model must learn grammar, facts, style, and even a little reasoning. To finish the sentence, the capital of France is, it has to know geography. To continue a story, it has to understand the characters. Predicting the next word, at enormous scale, forces the model to learn how the world is written about.
Summary
What a first clip. A language model has one simple job, guess the next word. It gives every possible word a chance, picks one, and repeats. A neural network produces those chances, and learning to predict well forces it to learn a lot about the world. Next, we open the neural network, and see how machines actually learn.
Key takeaways- A language model predicts the next word
- It gives every word a chance, then picks
- Neural networks produce those chances