Capstone: The Brief

This is what it all led to. Your boss calls you in. HomeWorth wants to launch an instant valuation product for its customers. They'll get a price in seconds, instead of waiting for three days. You own it from start to finish. Everything you've learned comes together in the next few lessons.

• The final project
• An instant valuation product
• You own it from start to finish

The brief

Before any code, professionals write down the brief. The business goal is to replace a three day wait with an instant estimate. The target is the median house value. Success means, on unseen data, a typical error that's small enough for a useful first estimate, beating the simple baselines. And we must state the limits. It can't price homes above the cap, so those go to a human expert.

Step one: honest data

Step one. We remove the capped districts, since their prices aren't real, and our product will send those to a person. Then, right away, before any exploring, we lock away twenty percent as the test set. We won't touch it again until the very end. Run it. We have about nineteen thousand six hundred rows, and the discipline starts here.

import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

df = fetch_california_housing(as_frame=True).frame
df = df[df["MedHouseVal"] < 5].reset_index(drop=True)

train, test = train_test_split(df, test_size=0.2, random_state=42)

print("Rows kept:", len(df))
print("Train:", len(train), " Test:", len(test))
print("Mean price:", round(train["MedHouseVal"].mean(), 2))

Rows kept: 19648
Train: 15718  Test: 3930
Mean price: 1.92

Step two: a feature transformer

Step two. We take everything we learned in feature engineering, and we package it as a proper class. By inheriting from these two base classes, our features become a real scikit learn step, that can sit inside a pipeline, and be saved with the model. Transform adds the ratios and the distances to the four big cities. Run it on one sample, and we see the new columns. Our domain knowledge is now safely wrapped up inside the model.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin

CITIES = {"SF": (37.77, -122.42), "LA": (34.05, -118.24),
          "SD": (32.72, -117.16), "SJ": (37.34, -121.89)}

class HouseFeatures(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X = X.copy()
        X["BedroomShare"] = X["AveBedrms"] / X["AveRooms"]
        X["PeoplePerRoom"] = X["AveOccup"] / X["AveRooms"]
        for name, (lat, lon) in CITIES.items():
            X[name] = np.hypot(X["Latitude"] - lat, X["Longitude"] - lon)
        return X

sample = pd.DataFrame({"AveRooms": [5.0], "AveBedrms": [1.0], "AveOccup": [3.0],
                       "Latitude": [37.8], "Longitude": [-122.3]})
print(HouseFeatures().fit_transform(sample).round(2).T.to_string())

                    0
AveRooms         5.00
AveBedrms        1.00
AveOccup         3.00
Latitude        37.80
Longitude     -122.30
BedroomShare     0.20
PeoplePerRoom    0.60
SF               0.12
LA               5.53
SD               7.23
SJ               0.62

Summary