Rare Classes

In the real world, the thing you most want to find is usually the rarest. One fraud in a thousand payments. One failing machine in a factory. If you train normally, the model learns that it can be right most of the time by ignoring those cases. We need to make the rare class count.

• The important cases are the rare ones
• Models learn to ignore them
• Make them count

Class weights

The simplest fix is one setting. Class weight balanced tells the model that a mistake on a rare case costs far more than a mistake on a common one. Here, just three percent of our cases are positive. We train the plain model, and the balanced one. Run it. Look at recall. The plain model misses many of the rare cases, while the balanced model catches far more of them. F one stays low, because we now accept many more false alarms. Once again, there is a trade.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, weights=[0.97], n_informative=4,
                           class_sep=0.7, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)

for name, model in [("Plain", LogisticRegression()),
                    ("Balanced", LogisticRegression(class_weight="balanced"))]:
    pred = model.fit(X_tr, y_tr).predict(X_te)
    print(f"{name:<9} recall {recall_score(y_te, pred):.3f}"
          f"  F1 {f1_score(y_te, pred):.3f}")

Plain     recall 0.000  F1 0.000
Balanced  recall 0.605  F1 0.103

Making synthetic examples

Another approach is to create more rare examples. SMOTE builds new ones, by drawing lines between real rare cases, and placing new points along them. It comes from a library called imbalanced learn. There is one critical rule. It must only touch training data, so we use the pipeline from that library, which applies it only during fit. Faking data in your test set would be cheating. Run it, and we get a similar gain in recall.

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score, f1_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=5000, weights=[0.97], n_informative=4,
                           class_sep=0.7, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)

model = make_pipeline(SMOTE(random_state=0), LogisticRegression())
pred = model.fit(X_tr, y_tr).predict(X_te)

print(f"Rare cases in training: {y_tr.sum()}")
print(f"Recall {recall_score(y_te, pred):.3f}  F1 {f1_score(y_te, pred):.3f}")

Rare cases in training: 130
Recall 0.512  F1 0.089

A practical order

So what should you try, and in what order? First, use a metric that respects the rare class, like F one, or the area under the precision recall curve. Next, move the threshold. It is free, and it often helps the most. Then try class weights. Only then, try resampling. Recent research has found that with strong models, plain threshold tuning often does just as well as synthetic data.

Summary