Evaluating Language Models

How do researchers measure whether one language model is better than another? There is no single number that captures everything. In this lesson, you will meet the main methods, perplexity, benchmarks, human ratings, and models as judges, and learn what each one can, and cannot, tell you.

• No single number tells the whole story
• Four main methods
• What each can and cannot tell you

Perplexity: how surprised is the model

The oldest measure is perplexity. It tells you how surprised a model is, by real text it has not seen. A lower perplexity means the model predicted the text better. It is useful during training, and for comparing models of the same kind. But it does not tell you whether answers are helpful, honest, or safe, so it is only one piece of the picture.

Benchmarks

Benchmarks are collections of questions, with known answers. Some test general knowledge, some test maths, some test coding, some test common sense. They give a quick, comparable score. But take care. Models may have seen the questions in training. Many benchmarks become too easy, as models improve. And a high score, may not match your own task. Use them as a first filter, and not as a final verdict.

Human ratings and model judges

For open ended tasks, like writing and chatting, people compare answers side by side, and pick the better one. This is the most trustworthy method, but it is slow, and expensive. So teams often use a strong model, as a judge, to score answers, using a clear rubric. It is fast, and scales well, but the judge has its own biases, such as preferring longer answers. A good approach combines both, with spot checks by humans.

Summary