Capstone: The Contest
A fair bake-off.
Now we find the champion. Remember our rule. Start with a baseline, because without one, you don't know whether any of this is working. Then give each model exactly the same fair test.
- Find the champion
- Baseline first
- Every model gets the same test
Step three: the contest
Step three. We set up four contenders. A dummy baseline, which always guesses the average. Ridge. A random forest. And boosting. Each one is a pipeline, using our feature step, so nothing can leak. They all use the same five folds. Run it. The baseline's error is huge, which shows how much our models are adding. Ridge improves a lot. The forest is much stronger. And boosting takes first place. On tables of data, that's what we expected.
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
CITIES = {"SF": (37.77, -122.42), "LA": (34.05, -118.24),
"SD": (32.72, -117.16), "SJ": (37.34, -121.89)}
class HouseFeatures(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
X = X.copy()
X["BedroomShare"] = X["AveBedrms"] / X["AveRooms"]
X["PeoplePerRoom"] = X["AveOccup"] / X["AveRooms"]
for name, (lat, lon) in CITIES.items():
X[name] = np.hypot(X["Latitude"] - lat, X["Longitude"] - lon)
return X
df = fetch_california_housing(as_frame=True).frame
df = df[df["MedHouseVal"] < 5].reset_index(drop=True)
train, test = train_test_split(df, test_size=0.2, random_state=42)
X_train, y_train = train.drop(columns="MedHouseVal"), train["MedHouseVal"]
X_test, y_test = test.drop(columns="MedHouseVal"), test["MedHouseVal"]
from sklearn.dummy import DummyRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
models = {
"Baseline": DummyRegressor(),
"Ridge": make_pipeline(HouseFeatures(), StandardScaler(), Ridge()),
"Forest": make_pipeline(HouseFeatures(),
RandomForestRegressor(60, n_jobs=-1, random_state=0)),
"Boosting": make_pipeline(HouseFeatures(),
HistGradientBoostingRegressor(random_state=0)),
}
folds = KFold(5, shuffle=True, random_state=0)
for name, m in models.items():
s = -cross_val_score(m, X_train, y_train, cv=folds,
scoring="neg_mean_absolute_error").mean()
print(f"{name:<9} MAE ${s * 100000:,.0f}")Baseline MAE $77,743 Ridge MAE $46,143 Forest MAE $29,575 Boosting MAE $28,448
Step four: tune the champion
Step four. Now that we know the winner, let's tune it. Optuna searches over the learning rate, tree size, leaf size, and regularisation. We use three folds, ten trials, and a third of the rows, to keep this quick. Notice that we're still only using the training data. Run it. It reports the best settings that it found. The gain from tuning is usually modest, and that's how it goes. Most of our improvement came from cleaning, features, and picking the right kind of model.
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
CITIES = {"SF": (37.77, -122.42), "LA": (34.05, -118.24),
"SD": (32.72, -117.16), "SJ": (37.34, -121.89)}
class HouseFeatures(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
X = X.copy()
X["BedroomShare"] = X["AveBedrms"] / X["AveRooms"]
X["PeoplePerRoom"] = X["AveOccup"] / X["AveRooms"]
for name, (lat, lon) in CITIES.items():
X[name] = np.hypot(X["Latitude"] - lat, X["Longitude"] - lon)
return X
df = fetch_california_housing(as_frame=True).frame
df = df[df["MedHouseVal"] < 5].reset_index(drop=True)
train, test = train_test_split(df, test_size=0.2, random_state=42)
X_train, y_train = train.drop(columns="MedHouseVal"), train["MedHouseVal"]
X_test, y_test = test.drop(columns="MedHouseVal"), test["MedHouseVal"]
import optuna
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
optuna.logging.set_verbosity(optuna.logging.WARNING)
folds = KFold(3, shuffle=True, random_state=0)
def objective(trial):
m = make_pipeline(HouseFeatures(), HistGradientBoostingRegressor(
learning_rate=trial.suggest_float("lr", 0.02, 0.2, log=True),
max_leaf_nodes=trial.suggest_int("leaves", 15, 90),
min_samples_leaf=trial.suggest_int("leaf", 5, 50),
l2_regularization=trial.suggest_float("l2", 0.0, 5.0),
max_iter=400, random_state=0))
return -cross_val_score(m, X_train[::3], y_train[::3], cv=folds,
scoring="neg_mean_absolute_error").mean()
study = optuna.create_study(sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=10)
print(f"Best MAE: ${study.best_value * 100000:,.0f}")
print({k: round(v, 3) for k, v in study.best_params.items()})Best MAE: $30,169
{'lr': 0.057, 'leaves': 58, 'leaf': 5, 'l2': 3.088}Summary
Our champion is chosen and tuned, using nothing but training data. Now for the moment of truth. It's time to break the seal, and open the test set for the very first time.
Key takeaways- Baseline shows the value added
- Boosting wins on tables
- Tuning adds a modest gain