Hyperparameter Tuning

Grid search and random search.

⏱ 2 min read▶ Video at 1:49:34🎓 Free

Watch this lesson in the Machine Learning courseWatch on YouTube ↗Full Machine Learning course

A model's weights are learned from data. But the settings that you choose yourself, like the depth of a tree, are not. These are called hyperparameters. They are like the dials on a radio. And turning them by hand, one at a time, is slow and unreliable. Let's make the computer do it.

Parameters are learned
Hyperparameters are chosen
Let the computer turn the dials

Grid search

The simplest approach is to try every combination. Grid search takes a list of values for each setting. With a pipeline, we name each setting with the step name, two underscores, and the setting. Four values of C, and four of gamma, gives sixteen combinations. Each gets five fold cross validation. That means eighty models trained, and you wrote only one line. Run it. It reports the best settings, and the best score.

grid.pypython

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)

pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.001, 0.01, 0.1, 1]}

search = GridSearchCV(pipe, grid, cv=5).fit(X, y)

print("Combinations tried:", len(search.cv_results_["params"]))
print("Best settings:", search.best_params_)
print("Best score:", round(search.best_score_, 4))

Output

Combinations tried: 16
Best settings: {'svc__C': 10, 'svc__gamma': 0.01}
Best score: 0.9789

Random search

The trouble with grids is that the cost explodes. Five settings, with five values each, means over three thousand combinations. Random search takes a different approach. It samples random combinations, and you decide how many. Research has shown that it usually finds great settings far sooner, because only a few of the dials really matter. Here we try fifteen random combinations, and notice that we search the learning rate on a log scale. Run it, and we match our best result, in reasonable time.

random_search.pypython

import time
from scipy.stats import loguniform, randint
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import RandomizedSearchCV

X, y = fetch_california_housing(return_X_y=True)
X, y = X[::4], y[::4]

space = {"learning_rate": loguniform(0.01, 0.3),
         "max_leaf_nodes": randint(10, 80),
         "min_samples_leaf": randint(5, 60)}

start = time.time()
search = RandomizedSearchCV(HistGradientBoostingRegressor(random_state=0),
                            space, n_iter=15, cv=3, random_state=0).fit(X, y)

print("Tried 15 of a huge number of combinations")
print("Best R2:", round(search.best_score_, 4))
print("Best:", {k: round(float(v), 3) for k, v in search.best_params_.items()})
print(f"Time: {time.time() - start:.0f}s")

Output

Tried 15 of a huge number of combinations
Best R2: 0.6924
Best: {'learning_rate': 0.062, 'max_leaf_nodes': 42.0, 'min_samples_leaf': 36.0}
Time: 61s

Smarter search with Optuna

We can do better still. Optuna learns as it goes. It looks at earlier results, and spends its effort on the promising areas. You write an objective function, which takes a trial, suggests settings, and returns a score. Then you run twenty trials. It's the same idea as random search, but it gets smarter with every attempt. For big projects, this is the professional choice.

optuna_search.pypython

import optuna
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import cross_val_score

optuna.logging.set_verbosity(optuna.logging.WARNING)
X, y = fetch_california_housing(return_X_y=True)
X, y = X[::4], y[::4]

def objective(trial):
    model = HistGradientBoostingRegressor(
        learning_rate=trial.suggest_float("lr", 0.01, 0.3, log=True),
        max_leaf_nodes=trial.suggest_int("leaves", 10, 80),
        min_samples_leaf=trial.suggest_int("leaf_size", 5, 60),
        random_state=0)
    return cross_val_score(model, X, y, cv=3).mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=20)

print("Best R2:", round(study.best_value, 4))
print("Best:", {k: round(v, 3) for k, v in study.best_params.items()})

Output

Best R2: 0.6877
Best: {'lr': 0.082, 'leaves': 73, 'leaf_size': 44}

Summary

Grid search for small spaces, random search for bigger ones, and Optuna when each run is expensive. But there is a trap hiding in all of them. The score that they report is too optimistic. Next, why that is, and how to get the truth.

Key takeaways

Grid: try everything
Random: sample widely
Optuna: learn as you search

Next up: Nested Cross Validation — why tuning scores flatter you

❮ Recommendation Systems Nested Cross Validation ❯