Evaluating Outputs

How do you know if an answer is good? Or if one model is better than another? Gut feeling is not enough. In this lesson, you will learn a simple scorecard for judging answers, and how professionals test A I systems.

• Good answer, or just fluent?
• A simple scorecard
• How professionals test

A simple answer scorecard

Here is a scorecard you can use for any answer. Is it accurate, are the facts right? Is it complete, does it cover what I asked? Is it relevant, does it stay on topic? Is it clear, easy to understand? And is it safe, free from harmful or private content? Score each from one to five, and you will quickly see where an answer falls short.

Testing an A I system properly

If you build something with A I, test it properly. First, collect a set of real example questions, with good answers. This is your test set. Second, run every question through your system. Third, score the answers, with the scorecard, or with another model acting as a judge. Fourth, change something, a prompt, or a model, and run the test again. Now you can see, with numbers, whether you improved things.

1. Collect test questionswith known good answers
2. Run them allthrough your system
3. Score the answersscorecard or a judge model
4. Change and compareimprove with evidence

About benchmarks

You will often see benchmarks, public tests, where models compete. They are useful, but take them with care. Models may be trained on the test questions. A benchmark may measure something different from your task. And small differences in scores often do not matter in real life. The best test is always your own examples, from your own work.

Summary