Text as Data

How could a model that only understands numbers possibly read? The first answer is almost embarrassingly simple, and it still works remarkably well. Forget grammar, and forget word order. Just count the words.

• Models only understand numbers
• Forget grammar
• Just count the words

From words to numbers

The bag of words approach makes one column for each word, and counts how often each one appears. But common words like the and is would dominate. So we use T F I D F. It scores a word highly if it's frequent in this document, but rare across all documents. The word the gets a low score, and a word like orbit gets a high one. Those rare, telling words are what carry the meaning.

A news classifier

Here are thousands of real discussion posts, on four topics. T F I D F vectorizer turns each post into numbers, and a logistic regression learns from them. A pipeline holds both together. Note that we removed headers and footers, because they could leak the answer. Run it. Roughly ninety percent accuracy, on real, messy writing. And it correctly places our two new sentences about a rocket, and a doctor. This small recipe is still used in real products, because it's fast, cheap, and hard to beat as a baseline.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

kinds = ["rec.autos", "sci.space", "sci.med", "talk.politics.guns"]
data = fetch_20newsgroups(subset="all", categories=kinds,
                          remove=("headers", "footers", "quotes"))
X_tr, X_te, y_tr, y_te = train_test_split(
    data.data, data.target, stratify=data.target, random_state=0)

model = make_pipeline(TfidfVectorizer(stop_words="english", sublinear_tf=True),
                      LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)

print("Articles:", len(data.data))
print("Accuracy:", round(model.score(X_te, y_te), 3))
for text in ["the rocket reached orbit", "the doctor prescribed a new drug"]:
    print(f"{text!r}: {data.target_names[model.predict([text])[0]]}")

Articles: 3877
Accuracy: 0.891
'the rocket reached orbit': sci.space
'the doctor prescribed a new drug': sci.med

Summary