Case Study: Fraud

A payment system must decide, in a fraction of a second, whether a transaction is fraud. Fraud is rare, perhaps one in a hundred, or far less. Fraudsters adapt, and the cost of a mistake is high in both directions. Blocking a good customer is bad, and missing fraud is worse. Let's see why rare events change everything.

• Fraud is rare
• Fraudsters adapt
• Both mistakes are costly

Rare events and accuracy

Here is the problem, in code. We create twenty thousand transactions, of which one percent are fraud. A model that always says, not fraud, is ninety nine percent accurate, and completely useless. Its precision and recall are zero. A plain model improves things, with good precision, but it catches only about a third of fraud. A class weighted model, which pays more attention to the rare cases, catches ninety five percent, but with many false alarms. And raising the cutoff, gives a middle path. The right balance depends on what a missed fraud, and a false alarm, cost the business.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=12, n_informative=6, weights=[0.99], flip_y=0, class_sep=1.2, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
print("fraud rate:", f"{y.mean():.1%}")

def report(name, model, threshold=0.5):
    flagged = model.predict_proba(Xte)[:, 1] >= threshold
    print(f"{name:26s} accuracy {accuracy_score(yte, flagged):.3f}  precision {precision_score(yte, flagged, zero_division=0):.2f}  recall {recall_score(yte, flagged):.2f}")

plain = LogisticRegression(max_iter=500).fit(Xtr, ytr)
weighted = LogisticRegression(max_iter=500, class_weight="balanced").fit(Xtr, ytr)
report("always say 'not fraud'", LogisticRegression(C=1e-9).fit(Xtr, ytr), threshold=0.99)
report("plain model", plain)
report("class-weighted model", weighted, threshold=0.5)
report("class-weighted, cutoff 0.9", weighted, threshold=0.9)

fraud rate: 1.0%
always say 'not fraud'     accuracy 0.990  precision 0.00  recall 0.00
plain model                accuracy 0.992  precision 0.63  recall 0.36
class-weighted model       accuracy 0.919  precision 0.11  recall 0.95
class-weighted, cutoff 0.9 accuracy 0.969  precision 0.21  recall 0.74

Operating a fraud model

Fraud systems have special operating needs. Latency, decisions in tens of milliseconds, so features must be precomputed, or streamed. Adversaries, attackers change tactics, so models decay fast, and need frequent retraining, and monitoring of new patterns. Delayed labels, a chargeback can arrive weeks later, so quality is measured late. Layers, rules catch known patterns, models catch subtle ones, and humans review the uncertain cases. And explanations, because analysts, and regulators, need to know why a transaction was blocked.

Rules or machine learning?

Should fraud detection use rules, or models? Both. Rules are transparent, instant to change, and ideal for known patterns, like a blocked country, or an impossible speed of travel. But they are brittle, and attackers learn them. Models find subtle combinations, and adapt as they retrain, but they are less transparent, and need data and monitoring. A strong system runs rules first, for the obvious cases, then a model for the rest, and routes uncertain cases to human reviewers, whose decisions become new labels.

Summary