Finding Rare Messages

Out of every thousand support messages, about twenty report real fraud. Those twenty are the ones that really matter. Your team builds a classifier, and proudly reports almost ninety nine percent accuracy. Before you celebrate, count how many frauds it found.

• Twenty in a thousand
• The ones that matter
• Count what it found

The accuracy illusion

This is a small simulated data set, with one thousand nine hundred and sixty normal messages, and only forty urgent ones. Stratify keeps the same share in both the training and test sets. We train the same model twice. The second time, class weight balanced makes mistakes on the rare class far more costly. Run it. The plain model has an accuracy near ninety nine percent, but its recall is point four. It missed six in every ten frauds, and nobody would notice from the accuracy. With balanced weights, recall jumps to one. Our toy data is easy, so a perfect score won't happen in real life. But the lesson does hold.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
normal = ["my order arrived late", "how do i change my address", "thanks for the help",
          "where is my parcel", "can i get a new invoice", "the app is slow today"]
urgent = ["someone stole my card", "my account was hacked", "i see a fraud charge",
          "unknown payment on my card", "my account was stolen"]
extra = ["please", "today", "hello", "again", "now", "sir", "team"]
def make(base, n):
    return [str(rng.choice(base)) + " " + " ".join(rng.choice(extra, 2)) for _ in range(n)]
X = make(normal, 1960) + make(urgent, 40)
y = [0] * 1960 + [1] * 40
tr, te, ytr, yte = train_test_split(X, y, stratify=y, test_size=0.5, random_state=0)
tf = TfidfVectorizer().fit(tr)

for name, w in [("Plain", None), ("Balanced", "balanced")]:
    m = LogisticRegression(class_weight=w).fit(tf.transform(tr), ytr)
    p = m.predict(tf.transform(te))
    print(f"{name:9} accuracy {np.mean(np.array(yte) == p):.3f}"
          f"  recall {recall_score(yte, p):.2f}  precision {precision_score(yte, p, zero_division=0):.2f}")

Plain     accuracy 0.988  recall 0.40  precision 1.00
Balanced  accuracy 1.000  recall 1.00  precision 1.00

Ways to handle rare classes

You've got several tools. Use class weights first, as they're cheap. Move the decision threshold, so that you trade precision for recall. Collect more rare examples, perhaps by using keywords, or the active learning idea. Use a language model to rewrite and expand them, then check the results. Above all, choose your metric from the cost of a mistake.

Precision or recall?

These two pull against each other, so ask what a mistake costs. With fraud, a missed case is expensive, while a false alarm only costs a quick check. So you favour recall. With a filter which automatically deletes email, a false alarm loses something important. So you favour precision. Decide that with the people who own the problem.

Reporting rare classes honestly

With so few rare examples, your numbers can jump around. Twenty test cases means that one extra miss moves recall by five points. So report the actual counts, along with the percentages, and put an interval around them. Use a precision and recall curve, rather than one single threshold. And when you've moved a threshold on a test set, you've used that set up, so check the final choice on fresh data.

Keeping humans in the loop

For serious cases, a classifier shouldn't decide alone. Send high scores straight to a person, and let them act quickly. Send the middle band to a slower review queue. Let the low scores pass, but sample some of them at random, as that's the only way to learn what you're missing. Those reviewed cases then become new training data for rare examples, which were the hardest thing to find.

Summary