Under the Hood
Chips, memory and speed.
For the curious, this lesson looks inside the machine. What hardware runs these models? Why does a reply appear word by word? And what tricks make big models fast enough to use? You do not need to remember every detail, but understanding the shape of the answer, will make you a sharper user, and builder.
- What hardware runs these models?
- Why replies appear word by word
- Tricks that make big models fast
Why GPUs?
A model is mostly one operation, repeated billions of times, multiplying large grids of numbers, called matrices. A normal processor does a few things at once, very quickly. A graphics processor, a G P U, does thousands of simple things at the same time. That is exactly what matrix multiplication needs. So G P Us, and similar chips, are the engines of modern A I, and training a large model uses thousands of them, working together.
- A CPUa few fast workers
- A GPUthousands of simple workers
- Matrix mathsthe same step, many times
- AI chipsbuilt for exactly this job
Where the memory goes
Memory is the real limit. To run a model, all its weights must sit in the fast memory of the chip. A model with seventy billion weights, stored in sixteen bits, needs about one hundred and forty gigabytes. That does not fit on one chip, so it is split across several. And on top of the weights, the model needs room for its working notes, called the cache, which grows with the length of the conversation.
- Model weightsbillions of numbers, always loaded
- The cacheworking notes that grow with the conversation
- Your prompt and replytokens being processed
- Spare roomneeded to avoid running out
Why replies come word by word
Now we can explain the typing effect. To produce each new token, the model must run through all its layers, using everything before. Then it picks one token, and the process repeats. The tokens cannot be made in parallel, because each depends on the one before it. So the reply is naturally sequential, and streaming simply shows each token, as soon as it exists.
- Read everything so fartokens and cache
- Run through all layersa lot of maths
- Pick one tokenappend it
- Repeatone token at a time
Tricks that make it fast
Engineers use many tricks. The cache stores earlier calculations, so they are not repeated. Batching, serves many users at once, sharing the same hardware. Quantization, shrinks weights, so they load faster. Speculative decoding, uses a small model to guess several tokens ahead, and the big model checks them all at once. And mixture of experts, activates only part of the model, for each token, which saves work.
Summary
Good. G P Us do thousands of simple calculations at once, which suits matrix maths. Memory is the real limit, with weights, and cache. Replies appear word by word, because each token depends on the last, and clever tricks, like caching, batching, quantization, and experts, make it fast. Next, we look more closely at one of those clever designs, the mixture of experts.
Key takeaways- GPUs: thousands of simple workers
- Memory holds weights and cache
- Tricks: cache, batching, quantization, experts