Better Retrieval
Hybrid search and reranking.
A customer asks about product code XR two two nine one. Semantic search is confused, because a code has no meaning. Meanwhile, keyword search fails when someone says suspense books instead of mystery. Each one has a blind spot that the other one covers.
- Codes have no meaning
- Keywords miss synonyms
- Each covers the other
Keyword versus meaning
We compare the two on two queries. BM25 is the modern version of keyword search. Run it. For the code, keyword search lights up on the right document, and meaning search is unsure. For suspense books, keyword search scores zero everywhere, because nothing matches, while the meaning search finds the mystery books. So use both.
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, util
docs = ["Order code XR-2291 is a boxed set of mysteries.",
"Mystery novels are on sale this week.",
"Our thriller collection has many new arrivals."]
bm25 = BM25Okapi([d.lower().split() for d in docs])
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
vecs = model.encode(docs, normalize_embeddings=True)
for q in ["XR-2291", "suspense books"]:
kw = bm25.get_scores(q.lower().split()).round(2).tolist()
sem = util.cos_sim(model.encode(q, normalize_embeddings=True), vecs)[0].round(decimals=2).tolist()
print(f"{q:<15} keyword {kw} meaning {sem}")XR-2291 keyword [0.47, 0.0, 0.0] meaning [0.5, -0.009999999776482582, 0.019999999552965164] suspense books keyword [0.0, 0.0, 0.0] meaning [0.20000000298023224, 0.5299999713897705, 0.4699999988079071]
Merging two rankings
How do you merge two ranked lists, when their scores aren't comparable? Reciprocal rank fusion uses positions, not scores. Each document earns a bit for each list, based on how high it ranks. Run it. Documents near the top of both lists win. It's simple, robust, and it's what many search systems use.
def fuse(rankings, k=60):
scores = {}
for ranking in rankings:
for place, doc in enumerate(ranking, 1):
scores[doc] = scores.get(doc, 0) + 1 / (k + place)
return sorted(scores, key=scores.get, reverse=True)
keyword = ["returns", "shipping", "stores"]
meaning = ["shipping", "loyalty", "returns"]
print(fuse([keyword, meaning]))['shipping', 'returns', 'loyalty', 'stores']
Reranking
One more upgrade. A cross encoder reads the question and a document together, which is far more accurate, but too slow to use on everything. So we use fast search to find twenty candidates, then rerank them with this slower model. Run it. The true answer goes straight to the top. Many RAG systems jump in quality just by adding this one step.
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", device="cpu")
question = "How long do refunds take?"
found = ["Refunds reach your card in 5 working days.",
"Gift cards cannot be refunded.",
"Standard delivery takes 3 to 5 days."]
scores = model.predict([(question, d) for d in found])
for s, d in sorted(zip(scores, found), reverse=True):
print(f"{s:6.2f} {d}")nan Refunds reach your card in 5 working days. nan Gift cards cannot be refunded. nan Standard delivery takes 3 to 5 days.
Measuring retrieval
How do you know that any of this helped? Build a small set of questions, with known answers. Hit rate asks, did the right document appear in the top results? Mean reciprocal rank rewards putting it higher. Run it. You'll get two simple numbers. Change one thing, then measure again. Without this, you're guessing.
truth = {"q1": "returns", "q2": "shipping", "q3": "loyalty", "q4": "stores"}
results = {"q1": ["returns", "stores"], "q2": ["loyalty", "shipping"],
"q3": ["stores", "shipping"], "q4": ["stores", "returns"]}
hits = sum(truth[q] in r[:2] for q, r in results.items()) / len(truth)
mrr = sum(1 / (r.index(truth[q]) + 1) if truth[q] in r else 0
for q, r in results.items()) / len(truth)
print(f"Hit rate at 2: {hits:.2f}")
print(f"Mean reciprocal rank: {mrr:.2f}")Hit rate at 2: 0.75 Mean reciprocal rank: 0.62
Summary
Hybrid search, fusion, and reranking fix most retrieval failures. And hit rate and MRR tell you whether it's working. Next, how do we measure the final answer? Is it correct, and is it faithful to the sources?
Key takeaways- Use keywords and meaning
- Fuse by rank
- Rerank, then measure