AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Probability in Models
Statistics & Math for ML · Lesson 133 of 138

Naive Bayes in Code

MultinomialNB.

⏱ 2 min read▶ Video at 2:39:05🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Earlier, we updated the chance of spam from one word, by hand. Now let's train a real naive Bayes spam filter on a few messages, with scikit learn, and see it classify new ones.

Naive Bayes in Code
From the video: Naive Bayes in Code
  • Bayes by hand, earlier
  • Now a real filter
  • Train and classify

A tiny spam filter

Count vectorizer turns each message into word counts. Multinomial NB learns how often each word appears in spam and in normal messages. Then predict proba gives the probability of spam for new messages. Run it. The prize message is very likely spam, and the lunch message is not.

naive_bayes.pypython
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

texts = ["win a free prize now", "free money click now", "claim your free prize",
         "lunch at noon tomorrow", "see you at the meeting", "notes from the meeting"]
labels = [1, 1, 1, 0, 0, 0]

vec = CountVectorizer()
model = MultinomialNB().fit(vec.fit_transform(texts), labels)

for msg in ["free prize inside", "meeting moved to noon"]:
    p = model.predict_proba(vec.transform([msg]))[0, 1]
    print(f"{msg!r}: P(spam) = {p:.2f}")
Output
'free prize inside': P(spam) = 0.93
'meeting moved to noon': P(spam) = 0.15

Smoothing

What if a word never appeared in spam during training? Its probability would be zero, and one unseen word would wipe out all the other evidence. Naive Bayes adds a small count to every word, called Laplace smoothing, so nothing is ever exactly impossible. In scikit learn, that's the alpha setting.

keyvalue
CountVectorizertext to word counts
MultinomialNBlearns word rates
Smoothingadd a small count
alphadefault 1.0

Summary

A naive Bayes filter turns words into counts, and combines their evidence with Bayes' theorem. Smoothing handles unseen words. Next, measuring how different two distributions are.

Key takeaways
  • CountVectorizer
  • MultinomialNB
  • Laplace smoothing
Next up: KL Divergence — distance between distributions