Testing Prompts
Prompts are code.
Someone on the team improves a prompt, adding one friendly sentence. It looks harmless. Two weeks later, you notice that legal threats are no longer being sent to a person. A prompt is code. It deserves the same care as code.
- One friendly sentence added
- Legal threats stop being escalated
- A prompt is code
A prompt test set
We keep our prompts in a dictionary, with versions. We've got four test cases, including the tricky legal and injury ones. Each version gets scored on the same cases. Run it. Both versions score four out of four. So should you ship version two? It adds nothing that we can measure here. So either add harder test cases, or keep the shorter prompt. Without this test, you'd only be guessing. With it, any change has to prove itself. Run that on each change, and a broken prompt can never reach customers.
from typing import Literal
from pydantic import BaseModel
from llm import ask
class R(BaseModel):
team: Literal["refunds", "shipping", "human"]
PROMPTS = {"v1": "Choose a team for this message.",
"v2": "Choose a team. Legal threats and injuries always go to human."}
CASES = [("my refund is late", "refunds"), ("where is the parcel", "shipping"),
("my lawyer will contact you", "human"), ("the book injured my child", "human")]
for name, p in PROMPTS.items():
score = sum(ask(f"{p}\nMessage: {m}", schema=R).team == want for m, want in CASES)
print(f"{name}: {score}/{len(CASES)}")v1: 4/4 v2: 4/4
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Also uses llm.py
from google import genai
from google.genai import types
client = genai.Client()
def ask(prompt, system=None, schema=None, temperature=0.1):
config = types.GenerateContentConfig(temperature=temperature, system_instruction=system)
if schema:
config.response_mime_type = "application/json"
config.response_schema = schema
r = client.models.generate_content(model="gemini-flash-latest", config=config, contents=prompt)
return r.parsed if schema else r.textManaging prompts
Keep your prompts in version control, rather than pasted in a dashboard. Give each one a version number, and log which version made each reply. Add every real failure to your test set. Test the same prompt on a new model before you switch, as a model upgrade is effectively a code change. Treat all of this just as you'd treat software.
Summary
If you can't test it, you can't safely change it. Next, how to keep your costs under control.
Key takeaways- Prompts are code
- Score every version
- Failures become test cases