Slice-Based Evaluation

An overall score is an average, and averages hide things. A model can be excellent for most customers, and poor for a few important groups. Slice based evaluation means measuring quality separately, for meaningful segments of the data. It is simple, and it finds problems, that no overall number will ever show.

• Averages hide problems
• Measure by segment
• Find the weak slices

Quality by segment

Here is slice based evaluation, in code. We train a model, and score a test set. Then we group the customers in two ways. By tenure, new, middle, and long term. And by contract type, monthly, one year, and two year. For every group, we print how many customers it contains, the accuracy, and the recall. Look closely. Accuracy is highest, for long term, and two year customers, above ninety percent. But their recall is close to zero. The model almost never finds the rare customers who leave, in those groups. A single overall score would never have shown it.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, recall_score
from churn import make_data

train, test = make_data(4000, seed=1), make_data(3000, seed=2)
features = ["tenure", "charges", "calls", "contract"]
model = LogisticRegression(max_iter=500).fit(train[features], train["churn"])
test["pred"] = model.predict(test[features])
test["tenure band"] = pd.cut(test["tenure"], [0, 12, 36, 100], labels=["new (<1y)", "mid (1-3y)", "long (3y+)"])
test["contract type"] = test["contract"].map({0: "monthly", 1: "one year", 2: "two year"})

print(f"overall accuracy {accuracy_score(test['churn'], test['pred']):.3f}\n")
for column in ["tenure band", "contract type"]:
    print(column)
    for name, part in test.groupby(column, observed=True):
        print(f"  {name:12s} n={len(part):5d}  accuracy {accuracy_score(part['churn'], part['pred']):.3f}  recall {recall_score(part['churn'], part['pred']):.2f}")

overall accuracy 0.867

tenure band
  new (<1y)    n=  499  accuracy 0.756  recall 0.25
  mid (1-3y)   n= 1014  accuracy 0.820  recall 0.15
  long (3y+)   n= 1487  accuracy 0.937  recall 0.01
contract type
  monthly      n= 1623  accuracy 0.824  recall 0.20
  one year     n=  755  accuracy 0.895  recall 0.01
  two year     n=  622  accuracy 0.944  recall 0.00

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Choosing slices

Which slices should you check? The ones where mistakes matter most, and where behaviour is likely to differ. New versus long term customers. Regions, and languages. Device types, and channels. Rare categories, and edge cases. Groups relevant to fairness. And slices based on data quality, for example, rows with missing values. Compute them in your evaluation pipeline, and track them over time. Add a gate, that rejects a model, if any important slice gets much worse, even when the average improves.

Summary