Train, Test, and Honesty

Remember all those scores we reported to your boss? There is something uncomfortable to admit. They were all measured on the same data the model trained on. In other words, we let the student mark their own practice paper. It is time to fix that.

• An uncomfortable truth
• We marked our own paper
• Time to fix that

Split before you do anything

The fix is simple. Before you do anything, set aside a portion of the data. Typically twenty percent. This is the test set. Lock it away. The model trains only on the rest. At the very end, you unlock the test set once, and use it to get an honest score. It is like a sealed exam paper. If you peek at it, even a little, the score means nothing.

The split in code

Scikit learn gives us train test split. We pass X and y, and ask for twenty percent to be held back. Random state fixes the shuffle, so that we get the same result every time. Now we fit, only on the training rows. Then we measure only on the test rows. Run it. Over sixteen thousand rows to learn from, and over four thousand locked away. The error is a little worse than before. Good. This is the first number we can actually trust.

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

X, y = fetch_california_housing(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

model = LinearRegression().fit(X_train, y_train)

print("Rows to learn from:", len(X_train))
print("Rows locked away:  ", len(X_test))
mae = mean_absolute_error(y_test, model.predict(X_test))
print(f"Honest test MAE: ${mae * 100000:,.0f}")

Rows to learn from: 16512
Rows locked away:   4128
Honest test MAE: $53,320

Three rules of honest testing

Follow three rules. First, split early, before you explore too deeply, so that nothing from the test set shapes your ideas. Second, never fit anything on the test set, not the model, and not even a scaler. Third, use it rarely. Each time you check the test score and change something, you learn a tiny bit about it, and the score becomes a little less honest.

Summary