Error Analysis
The most valuable hour.
When a model is wrong, most people try a bigger model. The best engineers do something more boring, and far more effective. They read the mistakes, one by one. Andrew Ng calls this the most useful hour you can spend.
- People try a bigger model
- The best read the mistakes
- The most useful hour
Reading the mistakes
Cross val predict gives an honest prediction for every example, from a model that hasn't seen it. Then we collect the mistakes. Run it. Seventy nine percent, with five mistakes. Now read them. Tracking hasn't moved for a week, was labelled delivery, but it was predicted as refund. Why? Our tiny dataset has no other example with those words. Shipping took three weeks, is similar. So the fix isn't a better model. It's more examples of tracking problems. Reading five lines has told us exactly what to do next.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
from sklearn.pipeline import make_pipeline
from tickets import TICKETS
texts, y = map(list, zip(*TICKETS))
model = make_pipeline(TfidfVectorizer(), LogisticRegression(C=20, max_iter=1000))
pred = cross_val_predict(model, texts, y, cv=4)
wrong = [(t, a, p) for t, a, p in zip(texts, y, pred) if a != p]
print(f"Accuracy {1 - len(wrong) / len(y):.2f}, {len(wrong)} mistakes")
for t, a, p in wrong[:4]:
print(f"{a} -> {p}: {t[:42]}")Accuracy 0.79, 5 mistakes delivery -> refund: my order arrived late and the box was crus delivery -> refund: tracking has not moved for a week delivery -> refund: shipping took three weeks, far too slow quality -> delivery: the binding fell apart after one read
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Group your mistakes
Make a simple table. Read fifty mistakes, and put each into a group. Too little data of that type. A wrong label. Sarcasm. A message which really has two topics. Then work on the largest group first. If half of your errors are wrong labels, then no model can help. This tells you where an extra week of work will really pay off.
| Error type | Fix |
|---|---|
| Too few examples | Collect more |
| Wrong labels | Fix the data |
| Two topics at once | Allow several labels |
| Sarcasm or context | Stronger model |
| Vague message | Ask a question |
Summary
Read the errors, group them, and fix the largest group. Next, a classic algorithm that finds hidden structure in sentences.
Key takeaways- Read every mistake
- Group them by cause
- Fix the biggest group first