Reading a Classifier
What did it really learn?.
A hospital once built a model to flag dangerous chest X rays. It worked brilliantly. Then researchers looked into it. The model had learned to read a small marker in the corner, showing which hospital took the scan. Text models do the same kind of thing. So you've got to look inside.
- A model that worked brilliantly
- It was reading a corner marker
- Always look inside
Top words per class
Logistic regression is easy to read, as it's just one weight for each word. We sort the weights, and print the strongest five for each label. Run it. Delivery cares about arrived, package, parcel, and courier. Quality cares about pages, cover, and ink. Refund cares about money and charged. Those all make perfect sense. If you'd seen something strange here, such as a customer's name, then that would be a warning that the model's memorised your data. This takes a minute, and it's the first thing that you should do with any text classifier.
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from tickets import TICKETS
texts, y = zip(*TICKETS)
vec = TfidfVectorizer(stop_words="english")
model = LogisticRegression(C=20, max_iter=1000).fit(vec.fit_transform(texts), y)
names = np.array(vec.get_feature_names_out())
for label, row in zip(model.classes_, model.coef_):
print(f"{label:<9}", ", ".join(names[np.argsort(-row)[:5]]))delivery arrived, package, delivery, parcel, courier quality pages, cover, ink, page, smudged refund refund, money, want, charged, cancel
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Explaining one prediction
What about a single prediction? Here's a simple way that works on any model, even a black box. Remove one word at a time, and see how much the answer changes. This message mentions two problems, a lost parcel and a refund. Run it. It chooses refund, though it's only forty nine percent sure. And the word refund alone adds twenty five points. So now you know what drove that decision, and you can see why it's unsure. Tools such as LIME and SHAP are more careful versions of this idea.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from tickets import TICKETS
texts, y = zip(*TICKETS)
model = make_pipeline(TfidfVectorizer(), LogisticRegression(C=20, max_iter=1000)).fit(texts, y)
text = "my parcel was lost and i want a refund"
base = model.predict_proba([text])[0]
top = base.argmax()
print("Prediction:", model.classes_[top], round(base[top], 2))
words = text.split()
effects = []
for i, w in enumerate(words):
without = " ".join(words[:i] + words[i + 1:])
effects.append((base[top] - model.predict_proba([without])[0][top], w))
for e, w in sorted(effects, reverse=True)[:3]:
print(f"{w:<8} {e:+.3f}")Prediction: refund 0.49 refund +0.251 want +0.171 my +0.132
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Summary
Look at weights, and remove words to see what matters. Next, the most valuable hour that you can spend on any model.
Key takeaways- Print the top words
- Remove words to explain
- Strange features reveal problems