A Golden Test Set

A golden set is a list of inputs, along with what a good result looks like. It's your agent's exam paper. You run it after every change. If the score drops, you don't ship.

• Inputs and good results
• Your agent's exam paper
• If the score drops, don't ship

Your first eval

For speed, we'll test a rule based stand-in, called fake Juno, and later on, you can swap in the real thing. Each case has a message, and a phrase that a good reply should contain. We're not matching the exact words, just the important part. We loop, score, and count. Run it. Four out of four. Now if someone breaks something, the score tells you, right away, and which case failed.

from juno import fake_juno

CASES = [
    ("Where is B42?", "has shipped"),
    ("Where's my stuff?", "order number"),
    ("Refund A17 please", "person will confirm"),
    ("Refund me NOW or else", "person will confirm"),
]

passed = 0
for message, expected in CASES:
    reply = fake_juno(message)["reply"]
    ok = expected in reply
    passed += ok
    print("PASS" if ok else "FAIL", message)
print(f"Score: {passed}/{len(CASES)}")

PASS Where is B42?
PASS Where's my stuff?
PASS Refund A17 please
PASS Refund me NOW or else
Score: 4/4

import re

def fake_juno(message):
    """A rule-based stand-in, so that these tests run instantly."""
    steps = []
    order = re.search(r"\b[A-Z]\d{2}\b", message)
    if order:
        steps.append("get_order")
    if "refund" in message.lower():
        steps.append("search_policy")
        reply = "I've checked the policy, and a person will confirm your refund."
    elif order:
        reply = f"Order {order.group()} has shipped."
    else:
        reply = "Could you share your order number?"
    return {"reply": reply, "steps": steps}

What to put in your set

A good set covers normal cases, edge cases, and adversarial cases. Common requests, messy spelling, missing information, angry customers, and attacks. And include cases where the right action is to do nothing, or to ask a question. Agents love to act, so test whether they know when not to.

Running it fairly

Models vary, so run each case a few times, and look at the pass rate, rather than at a single run. A case that passes four times out of five is telling you something. Track your score over time, and keep the results of each run, so that you can compare a change against the baseline.

Summary