Text Classification
Sorting the inbox.
It's time for Scribe's first real job. Each morning, two thousand messages need to be sorted into three piles. Delivery problems, refund requests, and questions. Done by hand, that's four hours. Let's see if we can do it in four milliseconds.
- Three piles of messages
- Four hours by hand
- Four milliseconds by machine
A ticket router
Here's a tiny set of labelled tickets. Our recipe is simple. TF-IDF, with word pairs, followed by logistic regression, all in one pipeline. Just eighteen examples. Now we test it on messages that it hasn't seen. Run it. The first three are sorted correctly, even with new wording. The last one, the book is damaged, isn't quite like anything it's seen, and see how the confidence is lower. With a few thousand real tickets, this simple recipe would be remarkably strong. And it runs in less than a millisecond.
tickets = [
("my parcel never arrived", "delivery"), ("where is my order", "delivery"),
("tracking says delivered but nothing came", "delivery"),
("courier left it at the wrong house", "delivery"),
("still waiting after two weeks", "delivery"), ("delivery is very late", "delivery"),
("i want my money back", "refund"), ("please refund this order", "refund"),
("the charge was wrong, return my payment", "refund"),
("how do i get a refund", "refund"), ("cancel and refund me", "refund"),
("you charged me twice", "refund"),
("do you have this in hardback", "question"), ("what time do you open", "question"),
("is the sequel available", "question"), ("do you sell gift cards", "question"),
("can you recommend a thriller", "question"), ("are you open on sunday", "question"),
]
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
texts, labels = zip(*tickets)
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())
model.fit(texts, labels)
for m in ["the courier lost my parcel", "give me my money back",
"do you stock audiobooks", "the book is damaged"]:
print(f"{model.predict([m])[0]:<9} {model.predict_proba([m]).max():.2f} {m}")delivery 0.45 the courier lost my parcel refund 0.51 give me my money back question 0.54 do you stock audiobooks delivery 0.39 the book is damaged
Why did it decide that?
With a linear model, we can look inside. Each word has a weight for each class. We print the top words for every class. Run it. Look at these lists. For refunds, it's money, refund, and charged. For delivery, words like delivered, and parcel. This is a huge advantage of simple models. You can read what they learned, you can spot problems, and you can explain it to the shop owner.
tickets = [
("my parcel never arrived", "delivery"), ("where is my order", "delivery"),
("tracking says delivered but nothing came", "delivery"),
("courier left it at the wrong house", "delivery"),
("still waiting after two weeks", "delivery"), ("delivery is very late", "delivery"),
("i want my money back", "refund"), ("please refund this order", "refund"),
("the charge was wrong, return my payment", "refund"),
("how do i get a refund", "refund"), ("cancel and refund me", "refund"),
("you charged me twice", "refund"),
("do you have this in hardback", "question"), ("what time do you open", "question"),
("is the sequel available", "question"), ("do you sell gift cards", "question"),
("can you recommend a thriller", "question"), ("are you open on sunday", "question"),
]
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
texts, labels = zip(*tickets)
vec = TfidfVectorizer()
clf = LogisticRegression().fit(vec.fit_transform(texts), labels)
words = vec.get_feature_names_out()
for i, name in enumerate(clf.classes_):
top = np.argsort(clf.coef_[i])[-5:][::-1]
print(f"{name:<9}", ", ".join(words[j] for j in top))delivery where, is, arrived, never, parcel question you, open, do, available, sequel refund refund, me, charged, twice, please
A real comparison
Let's try this at scale, with almost four thousand real posts. We compare naive Bayes, the classic, with logistic regression on TF-IDF. Run it. Both reach high accuracy, in seconds, and on a laptop. Remember these scores. Later, we'll compare them with transformers. Never skip this step. In many business problems, this baseline is hard to beat, and it costs almost nothing to run.
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
cats = ["rec.autos", "sci.med", "sci.space", "comp.graphics"]
data = fetch_20newsgroups(subset="all", categories=cats,
remove=("headers", "footers", "quotes"))
tr_x, te_x, tr_y, te_y = train_test_split(data.data, data.target,
stratify=data.target, random_state=0)
for name, m in [("Naive Bayes", make_pipeline(CountVectorizer(stop_words="english"), MultinomialNB())),
("Logistic", make_pipeline(TfidfVectorizer(stop_words="english", sublinear_tf=True),
LogisticRegression(max_iter=1000)))]:
print(f"{name:<12} {m.fit(tr_x, tr_y).score(te_x, te_y):.3f}")Naive Bayes 0.893 Logistic 0.908
Summary
TF-IDF and logistic regression are fast, cheap, and readable. That's your baseline for every text task. Scribe can now sort the inbox. But how do customers feel? Next, reading emotions.
Key takeaways- TF-IDF + logistic is a strong baseline
- You can read what it learned
- Always start here