Anomaly Detection

A payment of nine thousand dollars, at three in the morning, from a country you have never visited. Your bank texts you within seconds. Nobody wrote a rule for that exact case. A model learned what normal looks like for you, and this payment did not fit.

• A strange payment at 3 a.m.
• No rule for that exact case
• It learned what normal looks like

The idea of isolation

Isolation forest has a lovely insight. Imagine splitting data with random cuts. A normal point sits in a crowd, so it takes many cuts to isolate it. An odd point sits alone, far from everyone else, so it is cut off quickly. Therefore, the fewer cuts it takes, the stranger the point is. It needs no labels, and it is fast.

Catching strange payments

Here are five hundred normal payments, plus three strange ones that we planted. Contamination is our guess at the share of oddities. Predict returns minus one for an anomaly. Run it. It caught all three planted cases, and flagged a few extra normal ones as well. Look at the scores. Odd points score clearly lower than a normal one. In real systems, the flagged cases go to a human to review, and those decisions become labels for a supervised model.

import numpy as np
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(0)
normal = rng.normal(50, 10, size=(500, 2))
odd = np.array([[120, 5], [2, 130], [140, 140]])
payments = np.vstack([normal, odd])

model = IsolationForest(contamination=0.01, random_state=0).fit(payments)
flags = model.predict(payments)

print("Flagged as odd:", (flags == -1).sum())
print("Were the three planted cases caught?", (flags[-3:] == -1).tolist())
print("Scores of the odd ones:", model.score_samples(odd).round(2))
print("Score of a normal one: ", model.score_samples(normal[:1]).round(2))

Flagged as odd: 6
Were the three planted cases caught? [True, True, True]
Scores of the odd ones: [-0.8  -0.82 -0.83]
Score of a normal one:  [-0.39]

Soft clusters

Before we move on, one more tool. Gaussian mixtures are like a softer k means. Instead of saying that a point belongs to one group, they give a chance for each group. They also handle stretched shapes, and groups of different sizes. A score called B I C helps choose the number of groups, and it found three here. And each point gets a probability, for each group. When a point sits on a border, the model can say so honestly.

from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture

X, _ = make_blobs(n_samples=600, centers=3, cluster_std=[1.0, 2.2, 0.6],
                  random_state=3)

best = min(range(1, 7), key=lambda k: GaussianMixture(k, random_state=0).fit(X).bic(X))
gm = GaussianMixture(best, random_state=0).fit(X)

print("Best number of groups by BIC:", best)
print("Soft membership of three points:")
print(gm.predict_proba(X[:3]).round(3))

Best number of groups by BIC: 3
Soft membership of three points:
[[0.    0.    1.   ]
 [0.978 0.    0.022]
 [0.    0.    1.   ]]

Summary