How Good Is It?

Your boss has one question for you. If a customer uses this model, how wrong will the price be? Saying the M S E is point five will get you a confused look. You need numbers that a business person can understand. There are three, and each one tells you something different.

• How wrong, in dollars?
• MSE alone is not enough
• Three numbers to know

The three regression scores

M A E is the mean absolute error. It is the average size of the mistake, in the same units as the price, and it is the easiest to explain. R M S E is the square root of M S E. It is also in dollars, but it punishes big misses more. And R squared is a score that says how much of the variation in prices the model explains. One is perfect. Zero means you might as well always guess the average.

Scoring the model

Scikit learn has a function for each. We make predictions, then compute the three scores. We multiply by a hundred thousand, to get real dollars. Run it. An average error of more than fifty thousand dollars. An R squared around point six, so we explain about sixty percent of the variation. It is a respectable start. But it is not good enough for HomeWorth. Telling a family their home is off by fifty thousand would not go down well.

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.metrics import (mean_absolute_error,
                             mean_squared_error, r2_score)

X, y = fetch_california_housing(return_X_y=True)
pred = LinearRegression().fit(X, y).predict(X)

mae = mean_absolute_error(y, pred)
rmse = np.sqrt(mean_squared_error(y, pred))
r2 = r2_score(y, pred)

print(f"MAE:  ${mae * 100000:,.0f}")
print(f"RMSE: ${rmse * 100000:,.0f}")
print(f"R2:   {r2:.3f}")

MAE:  $53,116
RMSE: $72,410
R2:   0.606

Looking at the errors

Numbers summarise, but pictures diagnose. This chart plots each prediction against its error. If the model were perfect, every dot would sit on the red line. Good errors look like random noise around that line. But look at this. There is a clear diagonal band, a straight edge cutting through the dots. That is the price cap we found earlier, damaging our model. The chart also shows that errors get bigger for expensive districts. A chart of errors is called a residual plot, and it should be a habit.

import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression

X, y = fetch_california_housing(return_X_y=True)
pred = LinearRegression().fit(X, y).predict(X)
errors = y - pred

plt.figure(figsize=(9, 4.5))
plt.scatter(pred, errors, s=2, alpha=0.2)
plt.axhline(0, color="tab:red")
plt.xlabel("Predicted price")
plt.ylabel("Error")
plt.tight_layout()
plt.savefig("residuals.png", dpi=120)
print("Biggest miss:", round(abs(errors).max(), 2))

Biggest miss: 7.35

Summary