AI Hints
Statistics & Math for ML · Lesson 33 of 138

Bayes for Spam

The naive Bayes idea.

⏱ 2 min read▶ Video at 37:11🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Early spam filters used Bayes' theorem, and it's still a great first model for text. The idea is simple. Some words, like free, appear much more often in spam. So seeing them should raise our belief that an email is spam.

Bayes for Spam
From the video: Bayes for Spam
  • Early spam filters used Bayes
  • Some words appear in spam more
  • Evidence raises belief

One word of evidence

Say forty percent of emails are spam. The word free appears in sixty percent of spam, but only five percent of normal emails. Run it. After seeing free, the chance of spam jumps from forty percent to about eighty nine percent.

spam.pypython
prior_spam = 0.40
p_free_if_spam = 0.60
p_free_if_ham = 0.05

p_free = p_free_if_spam * prior_spam + p_free_if_ham * (1 - prior_spam)
posterior = p_free_if_spam * prior_spam / p_free

print("before:", prior_spam)
print("after seeing 'free':", round(posterior, 3))
Output
before: 0.4
after seeing 'free': 0.889

Why naive?

A real email has many words. Naive Bayes multiplies the evidence from every word, as if each word were independent of the others. That's not really true, which is why it's called naive. But it works surprisingly well, it's very fast, and scikit learn has it built in, as multinomial NB.

keyvalue
Each wordadds evidence
Naiveassumes words are independent
Fastgreat first text model
sklearnMultinomialNB

Summary

Naive Bayes updates the chance of spam word by word. It's simple, fast, and a classic first model for text. Next, counting, and permutations.

Key takeaways
  • Prior × word evidence
  • Naive: independent words
  • MultinomialNB
Next up: Counting: Permutations — when order matters