Naive Bayes

In the early days of email, spam nearly drowned the internet. Then people tried an idea from a clergyman who died in the seventeen hundreds, Thomas Bayes. His rule tells you how to update a belief when you see new evidence. It worked astonishingly well, and it is still used today.

• Spam was drowning email
• A rule from the 1700s
• Still in use today

Updating beliefs

Start with a belief. Suppose forty percent of all email is spam. Now you see the word prize. Prize appears far more often in spam, so your belief rises. Then you see the word meeting, and it falls a little. Each word nudges the chance up or down. The model is called naive, because it pretends that words are independent of each other. That is obviously untrue, yet it works very well, and it is incredibly fast.

A tiny spam filter

Here are six tiny emails, labelled as spam or ham. Count vectorizer turns each one into word counts, and multinomial naive Bayes learns how likely each word is, in each class. We test three new messages. Run it. Each message gets a label, and a confidence. Notice the mixed one, which contains both spam-like and normal words. Six emails of training data, and it already behaves sensibly. Real filters follow the same recipe, with millions of messages.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

emails = ["win money now", "cheap prize win", "free money prize",
          "meeting at noon", "lunch with team", "project meeting notes"]
labels = ["spam", "spam", "spam", "ham", "ham", "ham"]

model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(emails, labels)

for text in ["free prize meeting", "team lunch at noon", "win a cheap lunch"]:
    chance = model.predict_proba([text])[0].max()
    print(f"{text!r:<24} {model.predict([text])[0]}  ({chance:.2f})")

'free prize meeting'     spam  (0.67)
'team lunch at noon'     ham  (0.94)
'win a cheap lunch'      spam  (0.75)

Can you trust the chance?

One warning. Naive Bayes is often badly over-confident. When it says ninety nine percent, the truth might be seventy. A model is well calibrated if its chances match reality. Here we compare what the model said with what was true, in bins. Run it. In the raw model, the said row and the true row are far apart. After calibrating, they nearly match. When your business uses the chance itself, such as for setting a price, this matters a great deal.

from sklearn.calibration import calibration_curve, CalibratedClassifierCV
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB

X, y = make_classification(n_samples=20000, n_informative=8, n_redundant=6, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

raw = GaussianNB().fit(X_tr, y_tr)
fix = CalibratedClassifierCV(GaussianNB(), method="isotonic", cv=5).fit(X_tr, y_tr)

for name, m in [("Raw", raw), ("Calibrated", fix)]:
    truth, said = calibration_curve(y_te, m.predict_proba(X_te)[:, 1], n_bins=5)
    print(f"{name:<11} said {said.round(2)}")
    print(f"{'':<11} true {truth.round(2)}")

Raw         said [0.01 0.29 0.51 0.71 0.98]
            true [0.06 0.38 0.6  0.6  0.93]
Calibrated  said [0.05 0.28 0.51 0.7  0.95]
            true [0.05 0.23 0.49 0.63 0.95]

Summary