Hyperparameter Tuning

Every model has settings that you choose before training, like the depth of a tree, or the number of trees. They are called hyperparameters. The best values are rarely obvious. You could guess, but a smarter way is to let the computer search. In this lesson, you will use Optuna to do exactly that.

• Settings you choose before training
• Guessing is slow
• Let the computer search

Grid, random, and smart search

There are three ways to search. Grid search tries every combination on a grid. It is thorough, but the cost explodes with every extra setting. Random search tries random combinations, which is often better, because it explores more different values. Smart search, called Bayesian optimisation, learns from earlier trials. After each trial, it chooses the next settings in promising areas. Optuna uses this idea, and usually finds good settings in far fewer trials.

Tuning with Optuna

Here is a tuning job, in code. The objective function builds a model, with settings suggested by Optuna. Number of trees, between twenty and two hundred. Depth, between two and twelve. And the minimum samples per leaf. Then it scores the model, with three fold cross validation, and returns the score. We create a study that maximises that score, and run twenty trials. At the end, we print how many trials ran, the best score, and the best settings.

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from churn import make_data

optuna.logging.set_verbosity(optuna.logging.WARNING)
df = make_data(2000, seed=1)
X, y = df.drop(columns="churn"), df["churn"]

def objective(trial):
    model = RandomForestClassifier(
        n_estimators=trial.suggest_int("n_estimators", 20, 200),
        max_depth=trial.suggest_int("max_depth", 2, 12),
        min_samples_leaf=trial.suggest_int("min_samples_leaf", 1, 20),
        random_state=0,
    )
    return cross_val_score(model, X, y, cv=3).mean()

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=20)
print("trials run:", len(study.trials))
print("best score:", round(study.best_value, 3))
print("best params:", study.best_params)

trials run: 20
best score: 0.864
best params: {'n_estimators': 115, 'max_depth': 5, 'min_samples_leaf': 20}

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Tuning without fooling yourself

A few warnings. Always tune with cross validation, or a validation set, never on the final test set, or you will fool yourself. Record every trial in your experiment tracker. Set a budget, because there are diminishing returns, and a tiny gain may not matter in production. And remember that tuning cannot fix bad data, or a wrong problem. First get the data right, then tune.

Summary