Why Did It Decide That?

A customer's home is valued well below what they expected. They ask why. Answering, the algorithm said so, is not good enough, and in many countries it may even be illegal. Your powerful boosted model has hundreds of trees. Nobody can read that. So how do we look inside?

• A customer asks why
• The algorithm said so won't do
• How do we look inside?

Permutation importance

Permutation importance is a wonderfully simple idea. Take a trained model, and shuffle one column, so that it becomes meaningless. Then measure how much the score falls. If the score collapses, that feature mattered. If nothing changes, it did not. It is done on held-out data, so it measures real usefulness. Run it. Look at this. Latitude and longitude rank very high, along with income. The model relies on location far more than the older method suggested.

from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

data = fetch_california_housing()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, random_state=0)

model = HistGradientBoostingRegressor(random_state=0).fit(X_tr, y_tr)
result = permutation_importance(model, X_te, y_te, n_repeats=5, random_state=0)

order = result.importances_mean.argsort()[::-1]
for i in order:
    print(f"{data.feature_names[i]:<11} {result.importances_mean[i]:.3f}")

Latitude    0.831
Longitude   0.657
MedInc      0.527
AveOccup    0.149
AveRooms    0.046
HouseAge    0.039
Population  0.009
AveBedrms   0.007

Explaining one prediction

Global importance tells us about the whole model. But our customer wants to know about one house. S H A P does this. It comes from game theory. Imagine that the features are players on a team, who together produce the final prediction, and we want to divide the credit fairly. Start from the average prediction. Each feature then pushes it up, or down. High income adds a lot. Location adds a little. An older house takes some away. The bars add up exactly to the final price.

SHAP in code

The shap library has a fast explainer for tree models. We compute values for one district. It starts from the average prediction, and each feature adds or subtracts. Run it. You can read this directly to a customer. Your price is above average, mainly because of income and location. That is an explanation that a person can understand, and even challenge.

import numpy as np
import shap
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor

data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
model = HistGradientBoostingRegressor(random_state=0).fit(X, y)

explainer = shap.TreeExplainer(model)
sample = X.iloc[:300]
values = explainer.shap_values(sample)

print(f"Average prediction: {explainer.expected_value.item():.2f}")
print(f"This district:      {model.predict(sample.iloc[[0]])[0]:.2f}")
for name, v in sorted(zip(X.columns, values[0]), key=lambda p: -abs(p[1]))[:5]:
    print(f"  {name:<11} {v:+.2f}")

Average prediction: 2.07
This district:      4.16
  MedInc      +1.77
  Longitude   +0.44
  Latitude    -0.43
  AveRooms    +0.21
  HouseAge    +0.09

How does a feature act?

One more tool. A partial dependence plot shows how the prediction changes as one feature moves, averaging over everything else. On the left, as income rises, the predicted price climbs steadily, then levels off, partly because of that price cap. On the right, house age has a much gentler effect. These charts help you check that a model behaves sensibly. Be careful though. They describe the model, not cause and effect.

import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.inspection import PartialDependenceDisplay

data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target
model = HistGradientBoostingRegressor(random_state=0).fit(X, y)

fig, ax = plt.subplots(figsize=(10, 4.5))
PartialDependenceDisplay.from_estimator(model, X, ["MedInc", "HouseAge"], ax=ax)
plt.tight_layout()
plt.savefig("pdp.png", dpi=120)
print("Each line: predicted price as one feature changes")

Each line: predicted price as one feature changes

Summary