Sampling and Temperature

When a model has a chance for every next word, how does it choose one? It could always pick the most likely word. But then every answer would be boring, and identical. The choice rule is called sampling, and one setting, temperature, controls how adventurous it is.

• How does the model choose a word?
• Always the safest word is boring
• Temperature: the creativity dial

The temperature dial

Here are real chances from GPT two, for the next word after, once upon a time, there was a little. At a low temperature, the model sharpens the differences. The most likely words become even more likely, and the answers become focused, and predictable. At a medium temperature, the chances stay as the model believes them. At a high temperature, the chances flatten out. Unlikely words get a real opportunity, and answers become surprising, and sometimes strange.

Choosing the temperature

So which temperature should you use? For facts, maths, and code, use a low temperature, near zero, so answers stay accurate and consistent. For emails and everyday writing, a medium temperature is a good balance. For brainstorming, poems, and fresh ideas, use a higher temperature, to get surprising variety. Most tools set a sensible default, but now you know what the dial does.

Other sampling controls

Two more controls are common. Top K says, only consider the K most likely words. Top P, also called nucleus sampling, says, only consider the smallest group of words, whose chances add up to P percent. Both cut off the long tail of silly options, while keeping some variety. You will often see them next to temperature, in the settings of a model.

Summary