Tuning Boosting

Boosting has a lot of settings, and they can look intimidating. The good news is that only a few really matter. And one of them, the number of trees, can be chosen for you automatically, with a beautiful trick called early stopping.

• Many settings, few that matter
• Trees can be chosen for you
• Early stopping

Early stopping

We allow up to three thousand trees, which is far more than we need. Early stopping sets aside a slice of the training data, and watches the score there after each new tree. If it fails to improve for twenty five rounds in a row, training stops. Run it. It used far fewer trees than we allowed. We saved time, and we avoided overfitting, and we never had to guess the right number.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

model = HistGradientBoostingRegressor(
    max_iter=3000, learning_rate=0.05, early_stopping=True,
    validation_fraction=0.15, n_iter_no_change=25, random_state=0)
model.fit(X_tr, y_tr)

print("Trees allowed:", 3000)
print("Trees used:   ", model.n_iter_)
print("Test R2:      ", round(model.score(X_te, y_te), 3))

Trees allowed: 3000
Trees used:    508
Test R2:       0.845

The settings that matter

Here are the settings that matter. The learning rate. Lower is usually better, but needs more trees. A value around point zero five is a great start. Tree size. Shallow trees, with a depth of three to eight, are usually best. Subsampling. Giving each tree a random part of the rows and features adds diversity. And regularisation, such as the minimum size of a leaf. Set the learning rate, use early stopping for the tree count, then tune the rest.

Quick check

A question for you. Your boosting model has a training score of point nine nine, and a test score of point seven. What should you do first? A, add more trees. B, raise the learning rate. C, make the trees shallower, and use early stopping. D, remove your test set. The answer is C. The large gap means overfitting, so we simplify the model.

Summary