Capstone: Designing Scribe

This is what the whole course was for. The owner of Bright Books wants Scribe live by next month. It has to read every message, route it, answer from the shop's real documents, check orders, and stay safe. You'll design it and build it, and each lesson you've learned has a part to play.

• Scribe goes live next month
• Read, route, answer, check, stay safe
• Every lesson has a part to play

The Scribe pipeline

Here's our design. Clean the message, removing private details. Triage it, finding the topic and any order number. Retrieve the right policies. Generate a grounded reply, with tools for order lookups. And check the output, before it reaches anyone. If Scribe isn't sure, it hands over to a person. Notice how much of this isn't a large language model at all. We use the simplest tool at each step.

1. Cleanredact private data
2. Triagetopic and order number
3. Retrievefind policies
4. Generategrounded reply
5. Checkthen send or hand off

Step one: triage

We build the first half. Regular expressions for email redaction and order numbers, because they're exact. Sentence embeddings with logistic regression, for the topic, because they're cheap and strong. Triage ties them together, and returns clean data. Run it. Everything is extracted correctly, with a confidence score. No large model is needed yet, so this runs in milliseconds, at almost no cost, on every single message.

import re
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

encoder = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")

train = [("my parcel never arrived", "delivery"), ("where is my order", "delivery"),
         ("tracking says delivered but nothing came", "delivery"),
         ("i want my money back", "refund"), ("please refund this", "refund"),
         ("you charged me twice", "refund"),
         ("do you sell audiobooks", "question"), ("what time do you open", "question"),
         ("can you recommend a thriller", "question")]

texts, labels = zip(*train)
router = LogisticRegression(max_iter=1000).fit(encoder.encode(texts), labels)

def clean(text):
    text = re.sub(r"[\w.]+@[\w.]+\w", "[EMAIL]", text)
    return re.sub(r"\s+", " ", text).strip()

def order_number(text):
    found = re.findall(r"#(\d{5})", text)
    return found[0] if found else None

def triage(message):
    text = clean(message)
    v = encoder.encode([text])
    return dict(topic=router.predict(v)[0], order=order_number(text),
                confidence=round(float(router.predict_proba(v).max()), 2), text=text)

print(triage("Hi, order #48213   never came. I'm sam@mail.com"))
print(triage("Are you open on Sundays?"))

{'topic': np.str_('delivery'), 'order': '48213', 'confidence': 0.48, 'text': "Hi, order #48213 never came. I'm [EMAIL]"}
{'topic': np.str_('question'), 'order': None, 'confidence': 0.5, 'text': 'Are you open on Sundays?'}

Summary