Mixture of Experts and Efficiency

Imagine a hospital, where every patient is examined by every single doctor. It would be slow, and wasteful. A mixture of experts model is built more sensibly. In this lesson, you will see how it works, and why many of the largest modern models use it.

• Every doctor for every patient?
• Send each token to the right experts
• Bigger knowledge, lower cost

How a mixture of experts works

Inside the model, the feed forward part of a block is replaced by many smaller parts, called experts. A tiny router looks at each token, and chooses only a few experts, perhaps two out of many, to handle it. The other experts rest. So the model can hold a huge amount of knowledge, in total, while using only a small part of it, for each token.

1. Token arrivesone piece of text
2. Router decidespicks the best two experts
3. Chosen experts workthe others rest
4. Combine and continuethe answer moves up a layer

Why this matters

The benefit is efficiency. A model with hundreds of billions of weights in total, may use only a fraction, for each token. That makes it faster and cheaper to run, than a dense model of the same total size. The price is complexity. All the experts must still be stored in memory, and training them to cooperate is harder. Still, it is one of the most important ideas, for scaling.

Other efficiency ideas

Researchers are working on many other ways, to do more with less. Distillation, trains a small model, to imitate a big one. Pruning, removes weights that barely matter. Better data, so that smaller models learn more. And new designs, that handle very long text, more cheaply. The goal is always the same, more capability, per dollar, and per watt.

Summary