Choosing a Model

You now know many algorithms. So on Monday morning, when a new problem lands on your desk, which one do you pick? There's a famous theorem, called no free lunch. It says that no single model is the best for every problem. But in practice, there is a sensible order to try them.

• Many models, one Monday morning
• No model always wins
• There is a sensible order

A quick bake-off

Here is a habit worth keeping. Try several models, with cross validation, in a few lines. We set up six very different models, each one properly scaled when it needs to be. A simple loop gives each one the same fair test. Run it. Several models land close together, and simple logistic regression actually comes first. That happens more often than people expect. When a simple model does as well as a complex one, choose the simple one.

import warnings
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

warnings.filterwarnings("ignore")
X, y = load_breast_cancer(return_X_y=True)
models = {
    "Logistic": make_pipeline(StandardScaler(), LogisticRegression()),
    "KNN": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "SVM": make_pipeline(StandardScaler(), SVC()),
    "Naive Bayes": GaussianNB(),
    "Forest": RandomForestClassifier(random_state=0),
    "Boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, m in models.items():
    print(f"{name:<12} {cross_val_score(m, X, y, cv=5).mean():.3f}")

Logistic     0.981
KNN          0.965
SVM          0.974
Naive Bayes  0.939
Forest       0.963
Boosting     0.965

Your starting guide

Here is a practical guide. Tables of data? Start with gradient boosting. Need to explain every decision? Use linear models, or a small tree. Very little data? Try a regularised linear model, or an S V M. Images, audio, or long text? That is the world of deep learning. And always, always start with a simple baseline, so that you know whether the complex model is actually adding anything.

Summary