Why Evals Matter
You can't improve what you can't measure.
A team changes one line in Juno's prompt, to make it friendlier. It seems fine in three tests. They ship it. A week later, they find that refund mistakes have tripled. Nobody noticed, as nobody was measuring. In agents, a small change can break something far away.
- One line changed in a prompt
- Refund mistakes tripled
- Nobody was measuring
Why testing agents is different
Normal software gives the same output each time, so a test either passes or fails. Agents don't work like that. The same input can give different words, and take different paths. So we don't check for exact text. We score behaviour, across many cases. That's an eval, and it's the most important practice in professional agent development.
Normal tests
- Same output every time
- Exact matches
- Pass or fail
Evals
- Output varies
- Score behaviour
- Measure over many cases
Three levels of evals
Evals come in three levels. Unit evals check one small step, such as the router. Trajectory evals check the path, meaning which tools were used. And end to end evals check the result. Do lots of the fast ones, and fewer of the slow ones. It's just like the test pyramid in normal software.
Start small, start early
Don't wait for the perfect test set. Twenty real examples are enough to begin with, and that's what the best teams recommend. Take them from real support tickets. Add every bug that you find. Over time, this set of examples becomes one of the most valuable things that your company owns.
Summary
Measure behaviour over many cases, and start with twenty examples. Next, we'll build your first eval.
Key takeaways- Agents need evals, not exact tests
- Three levels
- Start with twenty real cases