Seeing the Data

Here is a famous warning from statistics. Four completely different datasets can have exactly the same averages. You would only see the difference by drawing them. So before any model, a professional always draws the data. Today, that habit saves HomeWorth from a nasty surprise.

• Averages can lie
• Always draw the data
• A surprise is waiting

Two charts, one surprise

We make two charts, side by side. On the left, a histogram of prices, showing how many districts fall into each price band. On the right, a scatter plot, with income along the bottom and price going up. Now look at the left chart. See the tall spike at the far right? And on the scatter, a hard ceiling, a perfectly flat line across the top. Nearly a thousand districts sit exactly at five. Those prices were not real. The data was cut off.

import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing

df = fetch_california_housing(as_frame=True).frame

fig, axes = plt.subplots(1, 2, figsize=(10, 4.5))
axes[0].hist(df["MedHouseVal"], bins=50, color="tab:orange")
axes[0].set_title("House prices")
axes[1].scatter(df["MedInc"], df["MedHouseVal"], s=2, alpha=0.2)
axes[1].set_title("Income vs price")
plt.tight_layout()
plt.savefig("explore.png", dpi=120)

print("Districts at the cap:", (df["MedHouseVal"] >= 5).sum())

Districts at the cap: 992

Why the ceiling matters

Why does this matter? A district worth a million dollars is recorded as five hundred thousand. If we train on that, the model learns a lie. It will never predict above the ceiling, however rich the district. This is called a capped target, and you would never have found it by looking at a list of numbers. We have two choices. Remove those rows, or accept that our model cannot price the most expensive homes, and tell the business clearly.

Summary