Explainability

A customer asks why they were refused. A manager asks why the model changed its mind. A regulator asks how decisions are made. Answering, because the model said so, is not good enough. Explainability tools help you open the black box, and also help you find bugs. Let's try one.

• Why did it decide that?
• Useful for people and for debugging
• Global and local explanations

Global and local explanations

Explanations come in two kinds. A global explanation describes the model as a whole. Which features matter most, across all customers. It helps you sanity check the model, and find bugs, like a feature that should not matter, but does. A local explanation describes one prediction. Why was this customer flagged? It helps support teams, and affected people, understand a decision. Different tools produce each kind, and good practice uses both.

Permutation importance

Here is a global method, permutation importance. It is beautifully simple. Shuffle one feature, breaking its link to the answer, and measure how much the model's score drops. If the score collapses, the feature mattered. If nothing changes, it did not. We train a model, run the method ten times for stability, and sort the features by their importance. Then we print a small bar for each. Tenure, support calls, and charges matter most, and contract matters least. This matches what we know about the data, which is reassuring.

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from churn import make_data

train, test = make_data(4000, seed=1), make_data(1500, seed=2)
features = ["tenure", "charges", "calls", "contract"]
model = RandomForestClassifier(n_estimators=100, max_depth=6, random_state=0)
model.fit(train[features], train["churn"])

result = permutation_importance(model, test[features], test["churn"], n_repeats=10, random_state=0)
ranked = sorted(zip(features, result.importances_mean), key=lambda p: -p[1])
for name, score in ranked:
    print(f"{name:9s} {'#' * int(score * 1000):30s} {score:.3f}")

tenure    #############                  0.014
calls     ###########                    0.012
charges   #########                      0.009
contract  #####                          0.005

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Other explanation tools

There are more tools. SHAP assigns each feature a share of the credit, for each single prediction, and is the most popular choice for local explanations. LIME fits a simple model around one prediction. Partial dependence plots show how the average prediction changes as one feature moves. And, for simple models, like linear models and small trees, the model itself is the explanation. If you need to explain decisions, consider choosing a simpler model, even at a small cost in accuracy.

Summary