Gradient Boosting

A random forest is a crowd working separately. Boosting is a team working in sequence. The first model makes a rough guess. The second studies only the mistakes of the first. The third fixes what is left. Each one is weak by itself, but together they become the most successful algorithm ever made for tables of data.

• Forest: a crowd working apart
• Boosting: a team in sequence
• The champion for tabular data

Fixing the mistakes

Watch this. We start with one flat guess, the average. It fits poorly, and the gaps between the line and the dots are the errors. Now, here is the key move. We train a small tree, not on the prices, but on those errors. We add a little of it to our prediction, and the line bends closer. Then we measure the new errors, and repeat. Each tree shrinks what is left. See the remaining error falling on the right. The word gradient here means the same thing it did in gradient descent. We are walking downhill, but with trees as our steps.

Boosting by hand

This idea is so simple that we can code it in a few lines. Start by predicting the mean. Then loop. Find the residual, which is whatever we have got wrong so far. Fit a tiny tree to that residual. Add a small fraction of it to our prediction. That fraction is the learning rate, and it stops any one tree from having too much influence. Run it. By round fifty, the error has dropped dramatically. You have just written the core of an algorithm that wins competitions.

import numpy as np
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 6, 200)).reshape(-1, 1)
y = np.sin(x).ravel() + rng.normal(0, 0.15, 200)

prediction = np.full(200, y.mean())
learning_rate = 0.3

for round_ in range(1, 51):
    residual = y - prediction
    tree = DecisionTreeRegressor(max_depth=2).fit(x, residual)
    prediction += learning_rate * tree.predict(x)
    if round_ in (1, 5, 10, 50):
        print(f"round {round_:>2}  error {((y - prediction) ** 2).mean():.4f}")

round  1  error 0.2787
round  5  error 0.0539
round 10  error 0.0227
round 50  error 0.0115

The real thing

Scikit learn's fast version is called hist gradient boosting. It groups values into bins, so it runs quickly, even on millions of rows. It handles missing values with no extra work, and it needs no scaling. We train five hundred trees. Run it. We now have the lowest error of the course so far, and an R squared around point eight four. Your linear model, in the same units, was stuck at about point six. This is the moment you walk into your boss's office with a big smile.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

model = HistGradientBoostingRegressor(max_iter=500, learning_rate=0.08,
                                      random_state=0).fit(X_tr, y_tr)

print(f"MAE ${mean_absolute_error(y_te, model.predict(X_te)) * 1e5:,.0f}")
print(f"R2  {model.score(X_te, y_te):.3f}")

MAE $29,947
R2  0.845

Summary