AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Samples and Classes
Statistics & Math for ML · Lesson 126 of 138

Imbalanced Classes

When one class is rare.

⏱ 2 min read▶ Video at 2:30:35🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Fraud, disease, and machine failures are rare. A model trained on that data can learn to always say no, and still look accurate. There are a few standard ways to make it pay attention to the rare class.

Imbalanced Classes
From the video: Imbalanced Classes
  • Rare events
  • Always saying no looks good
  • Make the model pay attention

Class weights

We make data with only five percent positives, and train logistic regression twice. Once normally, and once with class weight balanced, which makes mistakes on the rare class cost more. Run it. Accuracy drops, from about ninety four to seventy nine percent. But recall on the rare class jumps from almost nothing to about seventy percent. Which matters more depends on your problem.

imbalanced.pypython
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=4000, weights=[0.95], class_sep=0.8, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, stratify=y, random_state=0)

for cw in [None, "balanced"]:
    m = LogisticRegression(class_weight=cw).fit(Xtr, ytr)
    print(f"{str(cw):9s} accuracy {m.score(Xte, yte):.3f}  recall {recall_score(yte, m.predict(Xte)):.3f}")
Output
None      accuracy 0.938  recall 0.036
balanced  accuracy 0.791  recall 0.709

Other tools

Besides class weights, you can oversample the rare class, undersample the common one, or move the decision threshold below a half. Whatever you do, judge the model with precision, recall, or AUC, never accuracy alone. And decide which mistake is worse for your problem, a false alarm, or a miss.

keyvalue
Class weightsrare mistakes cost more
Resamplingover or under sample
Thresholdlower than 0.5
Metricsrecall, precision, AUC

Summary

For imbalanced classes, use class weights, resampling, or a lower threshold, and judge with recall and precision. Next, distances, and the nearest neighbour model.

Key takeaways
  • class_weight='balanced'
  • Adjust the threshold
  • Don't trust accuracy
Next up: Distances for ML — k nearest neighbours