Naive Bayes from Scratch
Probability by counting.
Spam filters were among the first great successes of machine learning, and the method behind them is just counting. How often does this word appear in spam? How often in normal mail? Multiply those together. It's called Naive Bayes, and it's naive because it pretends that words have nothing to do with each other. And yet it works remarkably well.
- Spam filters: an early success
- Just counting
- Naive, yet remarkably good
The idea
Bayes' rule flips a question around. We want the chance of a label, given some words. That's hard. But the chance of seeing some words, given a label, is easy to count. So for each label, start with how common that label is, multiply by the chance of each word, and choose the highest.
- How common is the label?prior
- Chance of each wordcount it
- Multiply themnaive step
- Pick the highestthe answer
Sorting tickets by counting
Here are twenty four labelled support tickets. We count how many times each word appears under each label. To predict, we add up logs, instead of multiplying, because multiplying many tiny numbers runs out of precision. And plus one in the count is called smoothing. Without it, one unseen word would make the entire probability zero. Run it. Six out of six on new messages. No gradients. No training loop. And it took only a moment.
import math
from collections import Counter, defaultdict
from tickets import QUERIES, TICKETS
counts, totals, labels = defaultdict(Counter), Counter(), Counter()
for text, label in TICKETS:
labels[label] += 1
for w in text.split():
counts[label][w] += 1
totals[label] += 1
vocab = {w for c in counts.values() for w in c}
def predict(text):
def score(label):
s = math.log(labels[label] / len(TICKETS))
for w in text.split():
s += math.log((counts[label][w] + 1) / (totals[label] + len(vocab)))
return s
return max(labels, key=score)
right = sum(predict(q) == a for q, a in QUERIES)
print(f"Correct: {right}/{len(QUERIES)}")
print(predict("my refund never arrived"))Correct: 6/6 refund
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Why it still matters
Naive Bayes has real strengths. It trains instantly, even on huge datasets. It works with very few examples. It's simple to update with new data. So it's a great baseline, and it's often built into mail and log tools. If a fancy model can't beat it, then your problem may be your data.
Summary
Counting words and multiplying gets you surprisingly far. Next, how can you look inside a classifier, and see why it decided?
Key takeaways- Multiply word probabilities
- Add logs, smooth counts
- A strong baseline