Random Forests

In nineteen oh six, at a country fair, eight hundred people guessed the weight of an ox. Many guesses were wildly wrong. But the average of all the guesses was almost exactly correct. Individually, people are noisy. Together, their errors cancel out. What if we built a crowd of trees?

• 800 guesses at a country fair
• The average was nearly perfect
• Errors cancel out

A forest votes

That is a random forest. We train hundreds of trees. But if they were all the same, the crowd would be pointless. We need diversity, and we get it in two ways. Each tree sees a random sample of the rows. And at each split, it can only consider a random handful of features. So each tree is a little different, and a little wrong in its own way. Then they all vote, or for regression, we average them. The noise cancels, and the signal remains.

One tree against a forest

Let's run the contest. One tree, against a forest of two hundred trees. N jobs minus one uses all your computer's cores. Run it. The forest slashes the error, to around thirty four thousand dollars. And we did almost no tuning. Remember our linear model, at over fifty thousand? That is a huge improvement. Your boss will be very pleased. There is also a bonus here, the out of bag score. Since each tree skips some rows, those rows can be used as a free validation set.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor

X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

one = DecisionTreeRegressor(random_state=0).fit(X_tr, y_tr)
forest = RandomForestRegressor(n_estimators=200, n_jobs=-1,
                               oob_score=True, random_state=0).fit(X_tr, y_tr)

print(f"One tree  MAE ${mean_absolute_error(y_te, one.predict(X_te)) * 1e5:,.0f}")
print(f"Forest    MAE ${mean_absolute_error(y_te, forest.predict(X_te)) * 1e5:,.0f}")
print(f"Forest R2 {forest.score(X_te, y_te):.3f}")
print(f"Out-of-bag R2 {forest.oob_score_:.3f}")

One tree  MAE $47,582
Forest    MAE $33,767
Forest R2 0.795
Out-of-bag R2 0.805

What did the forest learn?

Forests are harder to read than one tree, but they can tell us what they used. Feature importances shows how much each feature reduced the error. We sort them, and draw simple bars. Med inc dominates, followed by average occupancy, and then location. A quick warning though. These importances can favour features with many unique values. Later in this course, we will meet a more trustworthy method.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor

data = fetch_california_housing()
forest = RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=0)
forest.fit(data.data, data.target)

ranked = sorted(zip(forest.feature_importances_, data.feature_names),
                reverse=True)
for share, name in ranked:
    print(f"{name:<11} {share:6.3f}  {'#' * int(share * 60)}")

MedInc       0.521  ###############################
AveOccup     0.137  ########
Latitude     0.093  #####
Longitude    0.091  #####
HouseAge     0.053  ###
AveRooms     0.045  ##
Population   0.032  #
AveBedrms    0.029  #

Summary