Evaluating RAG

ARAG system can fail in two very different ways. It might retrieve the wrong documents. Or it might retrieve the right ones, and still write something false. If you only look at the final answer, you can't tell which part to fix. So we test each part separately.

• Two different ways to fail
• Wrong documents or wrong writing
• Test each part separately

The RAG triad

Three questions cover almost everything. Context relevance, did we find useful text? Faithfulness, is the answer fully supported by that text? And answer relevance, does it actually answer the question? If the first one fails, fix retrieval. If the second fails, fix the prompt, or the model. Measure all three, on a fixed set of real questions.

A model as judge

How do you mark thousands of answers? A common trick is to use a language model as the judge. We ask for a structured verdict. Is the answer supported by the source? Does it answer the question? We test one good answer, and one that invents things. Run it. The first passes. The second gets caught, with a reason. A judge isn't perfect, so spot check its decisions by hand. But it lets you score at a scale that no team could do by reading.

import json
from google import genai
from google.genai import types
from pydantic import BaseModel

class Verdict(BaseModel):
    faithful: bool
    answers_question: bool
    reason: str

client = genai.Client()
source = "Refunds reach your card in 5 working days. Gift cards cannot be refunded."

def judge(question, answer):
    prompt = (f"Source:\n{source}\n\nQuestion: {question}\nAnswer: {answer}\n\n"
              "Is the answer fully supported by the source? Does it answer the question?")
    r = client.models.generate_content(
        model="gemini-flash-latest", contents=prompt,
        config=types.GenerateContentConfig(response_mime_type="application/json",
                                           response_schema=Verdict))
    return json.loads(r.text)

print(judge("How long do refunds take?", "About five working days."))
print(judge("How long do refunds take?", "Refunds are instant, plus a free voucher."))

{'faithful': False, 'answers_question': True, 'reason': "The source states that refunds reach your card 'in 5 working days', which is a specific timeframe, whereas the answer qualifies this as 'About five working days', introducing an unsupported approximation."}
{'faithful': False, 'answers_question': True, 'reason': 'The answer attempts to answer how long refunds take by claiming they are instant, but this directly contradicts the source text, which states refunds take 5 working days. The mention of a free voucher is also fabricated and not supported by the source.'}

Summary