From Tiny to Giant
How big models really differ.
Your tiny model has about a hundred thousand weights, and it trained for minutes. The largest models have hundreds of billions of weights, and they cost hundreds of millions of dollars to train. Is the difference just size? Mostly, yes. But a few extra ideas are needed to make scale work.
- A hundred thousand weights
- Versus hundreds of billions
- Mostly size, plus a few ideas
Tiny versus giant
Compare the two. Our model reads characters, and giants read word pieces, from a vocabulary of over a hundred thousand tokens. Ours has three layers, and they have dozens. We trained on a few lines, while they train on large parts of the internet. We used one laptop, and they use huge clusters of tens of thousands of G P Us, for months.
| Your TinyGPT | Giant model | |
|---|---|---|
| Tokens | Characters | Word pieces |
| Layers | 3 | Dozens |
| Weights | 100 thousand | Hundreds of billions |
| Data | A few lines | Much of the internet |
| Hardware | A laptop | 100,000 GPUs |
The extra ideas
Scale needs a few extra tricks. Better tokenisers split text into word pieces. Mixed precision saves memory. Training across thousands of G P Us needs careful engineering. After that first training, a model only predicts text, and doesn't yet act helpful. So fine tuning teaches it to follow instructions, and feedback from humans teaches it what good answers look like. That last stage turned GPT into ChatGPT.
Summary
The same architecture, scaled up, with careful engineering, and human feedback on top. That's the secret of the giants. Aria can see, read, and write. But so far, she only predicts things. Next, a totally different kind of network, one that can imagine. Generative models.
Key takeaways- Mostly the same design
- Data and compute make the giants
- Fine-tuning makes them helpful