Capstone: The Moment of Truth
Opening the sealed test set.
All through this project, the test set has stayed locked away. No model has seen it. No decision has touched it. This is the moment that all of that discipline was for. Whatever number comes out now, you can trust it.
- The test set has stayed sealed
- No model has seen it
- This number can be trusted
The final score
We train the final model on all of the training data, using the tuned settings. Then we predict, once, on the test set. We report more than one number. M A E and R squared for the team. And for your boss, we report how many estimates fall within ten and twenty percent of the true price. Finally, we split the errors by price band. Run it. The error is far below where we started. Most homes land within twenty percent of the true price. And see the bands. The error grows with price, as you'd expect, so expensive homes need more care.
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
CITIES = {"SF": (37.77, -122.42), "LA": (34.05, -118.24),
"SD": (32.72, -117.16), "SJ": (37.34, -121.89)}
class HouseFeatures(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
X = X.copy()
X["BedroomShare"] = X["AveBedrms"] / X["AveRooms"]
X["PeoplePerRoom"] = X["AveOccup"] / X["AveRooms"]
for name, (lat, lon) in CITIES.items():
X[name] = np.hypot(X["Latitude"] - lat, X["Longitude"] - lon)
return X
df = fetch_california_housing(as_frame=True).frame
df = df[df["MedHouseVal"] < 5].reset_index(drop=True)
train, test = train_test_split(df, test_size=0.2, random_state=42)
X_train, y_train = train.drop(columns="MedHouseVal"), train["MedHouseVal"]
X_test, y_test = test.drop(columns="MedHouseVal"), test["MedHouseVal"]
import numpy as np
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.pipeline import make_pipeline
final = make_pipeline(HouseFeatures(), HistGradientBoostingRegressor(
learning_rate=0.06, max_leaf_nodes=50, min_samples_leaf=15,
max_iter=400, random_state=0)).fit(X_train, y_train)
pred = final.predict(X_test)
errors = abs(pred - y_test)
print(f"Test MAE: ${mean_absolute_error(y_test, pred) * 100000:,.0f}")
print(f"Test R2: {r2_score(y_test, pred):.3f}")
print(f"Within 10%: {(errors / y_test < 0.10).mean():.0%}")
print(f"Within 20%: {(errors / y_test < 0.20).mean():.0%}")
for name, mask in [("Under $150k", y_test < 1.5),
("$150k-$300k", (y_test >= 1.5) & (y_test < 3)),
("Over $300k", y_test >= 3)]:
print(f"{name:<12} MAE ${errors[mask].mean() * 100000:,.0f}")Test MAE: $27,333 Test R2: 0.823 Within 10%: 47% Within 20%: 75% Under $150k MAE $19,826 $150k-$300k MAE $26,179 Over $300k MAE $49,913
From fifty three thousand to here
Look how far you've come. We started with linear regression on raw features, with an average error of about fifty three thousand dollars. Better data, better features, and a better model each took a chunk out of that. And your first test, in lesson one, wasn't even an honest test. Everything you see now is real.
| key | value |
|---|---|
| Lesson 1 | a model with no honest test |
| Linear model | about $53,000 error |
| Cleaning + features | better inputs |
| Boosting + tuning | your best honest score |
The report to your boss
A good report has no jargon. It says what we built, how well it works, and where it fails. It works best on mid-priced homes. It should not be used for homes above the cap. It should be retrained as the market moves, and we'll monitor it for drift. Being honest about limits is what builds trust.
Summary
Congratulations. You've taken a real problem from a blank page, all the way to an honest result. That's the whole job, from start to finish. HomeWorth launches next month. But the clinic has one more project, with very different challenges. Next, we'll predict who's about to leave.
Key takeaways- One clean look at the test set
- Report in business language
- State the limits honestly