Answers with Citations
Trust through evidence.
In twenty twenty three, a New York lawyer filed a legal brief, with cases written by a chatbot. The cases didn't exist. The judge wasn't impressed. Language models can invent things, in a confident voice. So the most important feature that you can add is evidence.
- 2023: a lawyer's legal brief
- The cases didn't exist
- Add evidence
Citing and refusing
Each source gets a letter. We tell the model to answer only from the sources, to cite them, and to reply not found if they don't say. Our code then checks that every citation is real. Run it. Can I return a sale book? No, and it cites source B. Do you ship to Mars? Not found. It refused to guess. And that's the real test of any RAG system. Then the code check makes sure that no invented source can slip by. People can now click and verify each claim, and that's how trust gets built.
import re
from llm import ask
docs = {"A": "Returns are free within thirty days if unused.",
"B": "Sale items cannot be returned.",
"C": "Gift cards never expire."}
context = "\n".join(f"[{k}] {v}" for k, v in docs.items())
for q in ["Can I return a sale book?", "Do you ship to Mars?"]:
reply = ask(f"Answer from these sources only, citing like [A]. If they don't say, reply exactly NOT FOUND.\n{context}\nQuestion: {q}")
tags = set(re.findall(r"\[([A-Z])\]", reply))
print(reply.strip(), "| valid sources:", tags <= set(docs))No, sale items cannot be returned [B]. | valid sources: True NOT FOUND | valid sources: True
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Also uses llm.py
from google import genai
from google.genai import types
client = genai.Client()
def ask(prompt, system=None, schema=None, temperature=0.1):
config = types.GenerateContentConfig(temperature=temperature, system_instruction=system)
if schema:
config.response_mime_type = "application/json"
config.response_schema = schema
r = client.models.generate_content(model="gemini-flash-latest", config=config, contents=prompt)
return r.parsed if schema else r.textChecking for yourself
Go one step further, and test faithfulness. Does the source actually say what the answer claims? Have a second model check each claim against its source. Track how often the bot says not found, as a rate of zero means that it's probably guessing. And test it with questions that have no answer. Those are the most revealing tests of all.
Summary
Cite your sources, refuse when you can't, and verify with code. Next, how to change a prompt without breaking everything.
Key takeaways- Answer only from sources
- Validate citations in code
- Test unanswerable questions