Reranking
The best result goes first.
Vector search is quick, but it's rough. It returns things about the right topic, but not always the best answer. A customer asks where a gift card refund goes. The search returns four related passages, and the one that really matters is in third place. So add a second, more careful step.
- Vector search is quick but rough
- The best answer is in third place
- Add a careful second step
A second opinion
We've got four documents about refunds and gift cards. We ask a model to rank them for this exact question. Run it. It puts document three first, refunds for gift cards go to your account balance. That's exactly right, though two others share many of the same words. A model reading the question and the documents together can see that difference. Embeddings are compared separately, so they can't. In production, you'd use a dedicated cross encoder, or a reranking service, which is faster and cheaper than a large model.
from pydantic import BaseModel
from llm import ask
class Order(BaseModel):
best_first: list[int]
docs = ["Gift cards never expire.", "Refunds reach your card within five working days.",
"Returns are free for thirty days.", "Refunds for gift cards go to your account balance."]
q = "I paid with a gift card. Where does my refund go?"
listing = "\n".join(f"{i}: {d}" for i, d in enumerate(docs))
order = ask(f"Question: {q}\nDocuments:\n{listing}\nRank document numbers, most useful first.", schema=Order)
print(order.best_first)
print("Top:", docs[order.best_first[0]])[3, 1, 0, 2] Top: Refunds for gift cards go to your account balance.
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Also uses llm.py
from google import genai
from google.genai import types
client = genai.Client()
def ask(prompt, system=None, schema=None, temperature=0.1):
config = types.GenerateContentConfig(temperature=temperature, system_instruction=system)
if schema:
config.response_mime_type = "application/json"
config.response_schema = schema
r = client.models.generate_content(model="gemini-flash-latest", config=config, contents=prompt)
return r.parsed if schema else r.textA two-stage pipeline
This shows the standard design for search. The first stage is fast, and it casts a wide net, returning perhaps fifty results from millions. The second stage is slow and careful, ordering those fifty. Then the top few go to the language model. Each stage gets narrower, and more precise.
- Fast retrievalfifty from millions
- Rerankbest five
- Generateanswer from those
- Citeshow the sources
Summary
Retrieve widely, then rank carefully. Next, how to make answers that people can check.
Key takeaways- Vectors compare separately
- Rerankers read both together
- Two stages beat one