An FAQ Bot

Half of the support tickets at Bright Books ask the same ten questions. You don't need a giant language model for those. You just need to match each question to a known answer. And most importantly, you need to know when you don't have one.

• The same ten questions
• No need for a giant model
• Know when you can't answer

Matching with a threshold

We embed the known questions once. For a new question, we find the closest one. If the similarity is below a limit, we pass it to a person, instead of guessing. Run it. When does my order get here finds the delivery answer, even though it shares few words with it. I want to send this back finds the returns answer. And do you sell gift cards scores only point two two, so it goes to a person. That's the most important line in this program. A bot which says, I'm not sure, is far better than one that confidently gives the wrong answer.

from sentence_transformers import SentenceTransformer

FAQ = {"How long does delivery take?": "Three to five working days.",
       "How do I return a book?": "Print the label from your account page.",
       "When will I get my refund?": "Within five working days."}

model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
Q = model.encode(list(FAQ), normalize_embeddings=True)

def answer(text, limit=0.3):
    scores = Q @ model.encode([text], normalize_embeddings=True)[0]
    best = scores.argmax()
    if scores[best] < limit:
        return f"({scores[best]:.2f}) Passing you to a person."
    return f"({scores[best]:.2f}) {list(FAQ.values())[best]}"

for q in ["when does my order get here", "i want to send this back", "do you sell gift cards?"]:
    print(f"{q:<30} {answer(q)}")

when does my order get here    (0.51) Three to five working days.
i want to send this back       (0.35) Print the label from your account page.
do you sell gift cards?        (0.22) Passing you to a person.

Choosing the threshold

How do you set that limit? Collect a hundred real questions, and mark which ones should match. Then try different limits. Too low, and you give wrong answers. Too high, and you pass too much to people. So it's a business choice. What costs more, a wrong answer, or a delay? And log all of the unanswered questions, because those show you which new answers to write.

Summary