RAG in Production

Most business L L M apps do not rely on what the model remembers. They look up your documents first, and then answer from them. This is retrieval augmented generation. Its quality depends on two things, finding the right passages, and using them well. In this lesson, we measure the first part, retrieval.

• Look up first, then answer
• Retrieval quality limits everything
• Measure it separately

The pipeline to operate

A retrieval system has parts that need operating. Documents are collected, cleaned, and split into chunks. Each chunk is turned into a vector, and stored in an index. When a question arrives, the system finds the closest chunks, passes them to the model, with the question, and returns the answer, with sources. Every part can fail. Documents go stale. Chunks cut sentences in half. The index falls behind. So you need to monitor freshness, as well as quality, just like a data pipeline.

1. Collect and cleandocuments and sources
2. Chunk and embedvectors in an index
3. Retrieveclosest chunks for a question
4. Generateanswer, with citations

Measuring retrieval

Here is a retrieval test, using a simple keyword method. We have six policy documents, and six questions, each with the document that should be found. We turn the documents into vectors, with T F I D F. Then, for each question, we find the most similar document, and check whether it is the right one. We print a line for every question, and the final hit rate. The method finds four out of six. It misses the questions that use different words, like, delivery, instead of shipping, and, tomorrow, instead of express. That is the weakness of keyword search, and the reason that modern systems use embeddings, which match meaning.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "Refunds are possible within 30 days of delivery with the original receipt.",
    "Standard shipping takes 3 to 5 working days inside the country.",
    "Our support team is available from 9 to 5, Monday to Friday.",
    "Gift cards never expire and can be used on any product.",
    "To reset your password, open Settings and choose Security.",
    "Express shipping delivers within 24 hours for an extra fee.",
]
questions = [
    ("how long do I have to get my money back", 0),
    ("how many days does delivery take", 1),
    ("when can I talk to support", 2),
    ("do gift cards expire", 3),
    ("I forgot my password", 4),
    ("can I get it tomorrow", 5),
]

vectorizer = TfidfVectorizer(stop_words="english").fit(docs)
doc_vectors = vectorizer.transform(docs)

hits = 0
for question, right in questions:
    scores = cosine_similarity(vectorizer.transform([question]), doc_vectors)[0]
    top = int(scores.argmax())
    hits += top == right
    print("OK  " if top == right else "MISS", question, "->", top)
print("hit rate @1:", hits, "of", len(questions))

OK   how long do I have to get my money back -> 0
MISS how many days does delivery take -> 0
OK   when can I talk to support -> 2
OK   do gift cards expire -> 3
OK   I forgot my password -> 4
MISS can I get it tomorrow -> 0
hit rate @1: 4 of 6

What to measure in RAG

Measure retrieval and generation separately. For retrieval, track the hit rate, is the right passage in the top few results, and how high is it ranked. For generation, check groundedness, does the answer stay within the passages, and correctness, does it match the expected answer. Track freshness, how old is the newest document in the index. Track the share of questions that the system admits it cannot answer. And keep the test set of real questions, so that every change, to chunking, to the embedding model, or to the prompt, is measured, before release.

Summary