Advanced RAG

Your demo works perfectly. Then real customers arrive. They ask, how much is it? What's it? Their questions are short, vague, and depend on the conversation. Real RAG needs a few more tricks.

• The demo works
• Real users ask 'how much is it?'
• Questions depend on context

Rewriting the question

A follow-up question like, how much is it, has no meaning on its own. So before searching, we ask the model to rewrite it, using the conversation so far. Run it. It becomes a full query about shipping costs and times to Canada. Now the search can work. This small step, called query rewriting, is one of the highest value fixes in any chat based RAG system.

from google import genai

client = genai.Client()

history = "Customer: Do you ship to Canada?\nAgent: Yes, we do."
follow_up = "how much is it and how long?"

prompt = (f"Rewrite the last message as one standalone search query.\n\n"
          f"{history}\nCustomer: {follow_up}\n\nQuery:")

print(client.models.generate_content(model="gemini-flash-latest",
                                     contents=prompt).text.strip())

shipping cost and delivery time to Canada

More upgrades

Other useful upgrades. Filter by metadata, such as date or product, before searching. Multi query asks the question several ways. Parent documents retrieve small chunks, but give the model the wider section. And handle tables and images carefully, as they often get mangled by simple text extraction. Try these only after you've measured, and only where your evaluation shows a failure.

A golden test set

Every serious project needs a golden set. A small list of real questions, with what the answer must and must not contain. Run it after each change, like a unit test. Here, we test three questions, on some canned replies. Run it. Two pass, and one fails, as it wrongly says yes to Australia. Now if a change breaks something, you'll know right away, before your customers do.

tests = [
    {"q": "Refund time?", "must": ["5"], "must_not": ["instant"]},
    {"q": "Ship to Australia?", "must": ["not"], "must_not": ["yes"]},
    {"q": "Sunday hours?", "must": ["closed"], "must_not": []},
]

def scribe(q):
    canned = {"Refund time?": "Refunds take 5 working days.",
              "Ship to Australia?": "Yes, we ship everywhere!",
              "Sunday hours?": "We are closed on Sundays."}
    return canned[q]

passed = 0
for t in tests:
    a = scribe(t["q"]).lower()
    ok = all(w in a for w in t["must"]) and not any(w in a for w in t["must_not"])
    passed += ok
    print("PASS" if ok else "FAIL", t["q"], "->", a)
print(f"{passed}/{len(tests)} passed")

PASS Refund time? -> refunds take 5 working days.
FAIL Ship to Australia? -> yes, we ship everywhere!
PASS Sunday hours? -> we are closed on sundays.
2/3 passed

Summary