Pre-training at Scale

How does a transformer actually learn language? Not with a teacher. With a very simple game, played billions of times. In this lesson, you will learn what pre-training is, how much data and computing it needs, and why bigger has usually meant better.

• Learning without a teacher
• A simple game, played billions of times
• Bigger has usually meant better

The pre-training game

Pre-training is a fill in the blank game. Take a sentence from a book or a web page. Hide the next word. Ask the model to guess it. Compare the guess with the real word, measure the loss, and nudge the weights. The text itself provides the answers, so no human labelling is needed. Do this over trillions of words, and the model learns grammar, facts, and style. This is called self supervised learning.

1. Take real textfrom books, web, code
2. Hide the next wordthe text is its own answer
3. Guess and comparemeasure the loss
4. Nudge the weightsthen repeat trillions of times

How much data?

The amounts are staggering. Early models trained on a few billion words. GPT three saw hundreds of billions of tokens. Modern models train on trillions. That is more text than a person could read, in many thousands of lifetimes. And the training runs on thousands of specialised chips, called GPUs, for weeks, or months, costing millions of dollars.

Scaling laws

Researchers made a surprising discovery. Performance improves smoothly and predictably, when you scale up three things together. The size of the model. The amount of data. And the amount of computing. These are called scaling laws. They gave companies the confidence to invest in ever larger models, because they could predict the improvement before spending the money.

Summary