Support Vector Machines

Imagine two groups of houses, and you must build a road between them. There are many possible routes. But the best road is the widest one, with as much space as possible on either side. If you build it there, new houses are less likely to end up on the wrong side. That is the idea behind the support vector machine.

• Build a road between two groups
• The widest road is the safest
• That is the SVM

The widest street

Here are two classes. Many lines could separate them, but look at how close some pass to the points. A support vector machine looks for the line with the widest possible margin, the empty street on either side. Only the points on the edge of the street matter. They hold it in place, and they are called support vectors. Delete any other point, and nothing changes. That is why these models were so loved, because the answer depends on only a few critical examples.

The C setting

Real data is messy, so we allow some points to break the rules. The setting C controls that. A small C says, keep the street wide, and accept a few violations. A big C says, punish every mistake, even if the street gets narrow. As always, it is the same trade-off between a simple and a complex model. Run it. Here, values in the middle do well, and we always scale first, because S V Ms are based on distance.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)

for C in [0.01, 1, 100]:
    m = make_pipeline(StandardScaler(), SVC(kernel="linear", C=C))
    print(f"C = {C:<6} accuracy {cross_val_score(m, X, y, cv=5).mean():.3f}")

C = 0.01   accuracy 0.968
C = 1      accuracy 0.972
C = 100    accuracy 0.960

Summary