NLP · Lesson 88 of 106
Layered Classification
Cheap first, costly when needed.
Sending each ticket to a language model works very well, but it's costly and slow. Yet most tickets are easy. So what if a cheap model handled what it's sure about, and passed only the hard ones up the chain?
- Every ticket to an LLM is costly
- Most tickets are easy
- Pass only the hard ones up
A confidence gate
We train a tiny classifier. For each new ticket, we look at its confidence. Above point six, we trust it. Below that, it goes to the language model. Run it. All six of these were easy, so all six were handled by the tiny model, for free, in microseconds. In a real system, maybe eighty percent go down the fast route, and your large model bill falls by eighty percent. It's worth checking how good the fast route's answers are, as confidence doesn't always mean correct.
layered.pypython
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from tickets import QUERIES, TICKETS
texts, y = zip(*TICKETS)
model = make_pipeline(TfidfVectorizer(), LogisticRegression(C=50, max_iter=1000)).fit(texts, y)
cheap = 0
for text, answer in QUERIES:
p = model.predict_proba([text])[0]
route = "fast model" if p.max() > .6 else "send to LLM"
cheap += route == "fast model"
print(f"{p.max():.2f} {route:<12} {text[:34]}")
print(f"Handled cheaply: {cheap}/{len(QUERIES)}")Output
0.93 fast model where is my parcel, it has not arr 0.97 fast model i need my money back please 0.98 fast model the pages are torn and the cover i 0.91 fast model courier lost my package 0.94 fast model refund the payment to my card 0.89 fast model print is smudged and hard to read Handled cheaply: 6/6
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Also uses llm.py
from google import genai
from google.genai import types
client = genai.Client()
def ask(prompt, system=None, schema=None, temperature=0.1):
config = types.GenerateContentConfig(temperature=temperature, system_instruction=system)
if schema:
config.response_mime_type = "application/json"
config.response_schema = schema
r = client.models.generate_content(model="gemini-flash-latest", config=config, contents=prompt)
return r.parsed if schema else r.textSummary
Cheap first, and costly when needed. Next, the human side of conversation.
Key takeaways- Trust confident predictions
- Escalate the rest
- Check the fast route's accuracy
Next up: Designing Conversations — the words around the model