Prompts as Code
Version, test, review.
In many L L M apps, the prompt is the most important piece of logic. A single changed sentence can improve answers, or quietly break them. Yet teams often edit prompts live, in a dashboard, with no history. The cure is to treat prompts exactly like code. Versioned, tested, and reviewed.
- A prompt is logic
- One sentence can break things
- Treat it like code
Prompt discipline
A few rules make prompts manageable. Keep them in files, in Git, not scattered in code, or hidden in a dashboard. Give each version a name, and record which version produced which answer. Use templates, with named placeholders, so that the structure stays fixed, and only the data changes. Review prompt changes in pull requests, like code. And run your evaluation set on every change, so you know whether the new prompt is really better.
A prompt regression test
Here is a tiny but real test harness. We have two versions of a prompt. Version one just says, answer the question. Version two is more careful. It sets a role, asks for one short sentence, and tells the model to say, I do not know, when the answer is not in the policies. Next, a stand in for the model, so that this demo runs offline. And a test set, five questions, with what each answer must contain, and a maximum length. Then, for each prompt version, we run every question, and count the tests that pass. Version one fails the careful cases, and rambles. Version two passes them all. That is a regression test, for a prompt.
PROMPTS = {
"v1": "Answer the question. Question: {question}",
"v2": "You are a support agent. Reply in one short sentence. "
"If the answer is not in our policies, say 'I do not know'. Question: {question}",
}
KB = {
"refund window": "Refunds are possible within 30 days.",
"shipping time": "Shipping takes 3 to 5 days.",
"support hours": "Support is open 9 to 5, Monday to Friday.",
}
def fake_llm(prompt):
"""A stand-in for a real model call, so this demo runs offline."""
question = prompt.split("Question:")[-1].lower()
known = [key for key in KB if key in question]
if known:
answer = KB[known[0]]
if "one short sentence" in prompt:
return answer
return "Great question! Let me explain in detail. " + answer + " Please tell us if anything else is unclear, and thank you for being a customer of ours."
if "I do not know" in prompt:
return "I do not know."
return "Our policy is probably similar to other stores, around 14 days."
CASES = [
("What is the refund window?", "30 days", 40),
("What is the shipping time?", "3 to 5", 40),
("What are the support hours?", "9 to 5", 40),
("Do you sell gift cards?", "I do not know", 40),
("What is the warranty period?", "I do not know", 40),
]
for name, template in PROMPTS.items():
passed = 0
for question, must_contain, max_words in CASES:
answer = fake_llm(template.format(question=question))
ok = must_contain in answer and len(answer.split()) <= max_words
passed += ok
print(f"prompt {name}: {passed}/{len(CASES)} tests passed")prompt v1: 3/5 tests passed prompt v2: 5/5 tests passed
Summary
Excellent. Keep prompts in Git, version them, and run a test set on every change. A tiny harness like this one can catch a worse prompt before users do. Next, how to evaluate answers that are free text, which is the hardest part of L L M operations.
Key takeaways- Prompts in files, versioned
- Templates keep structure fixed
- Regression-test every change