Fairness in Machine Learning

A large company once built a hiring tool, and trained it on ten years of past hiring decisions. The history mostly involved men, so the model learned to mark down applications that mentioned women's organisations. Nobody programmed that. It simply learned the past. A model is a mirror, and sometimes we do not like what it shows.

• A hiring tool trained on history
• It learned the bias in the past
• A model is a mirror

Bias from biased records

Let's see how this happens. We simulate two groups of applicants, with exactly the same true skill. But past records were harder on group one, so their scores are lower. The model never sees the group label. It only sees the record. Run it. Group one is approved far less often, and fewer of its qualified people are found, though the two groups are equally skilled. Removing the sensitive column did not help, because the bias lives in the other features.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score

rng = np.random.default_rng(0)
n = 4000
group = rng.integers(0, 2, n)
skill = rng.normal(0, 1, n)
qualified = (skill + rng.normal(0, 0.5, n) > 0).astype(int)
# past decisions were harder on group 1, so the record is biased
record = skill - 0.8 * group + rng.normal(0, 0.5, n)

X = np.column_stack([record])
pred = LogisticRegression().fit(X, qualified).predict(X)

for g in (0, 1):
    m = group == g
    print(f"Group {g}: approved {pred[m].mean():.2f}"
          f"   qualified people found {recall_score(qualified[m], pred[m]):.2f}")

Group 0: approved 0.62   qualified people found 0.90
Group 1: approved 0.38   qualified people found 0.66

What fairness can mean

Fairness has several definitions. Demographic parity asks that each group is approved at the same rate. Equal opportunity asks that qualified people in each group have the same chance of being found. Calibration asks that a score of seventy percent means the same thing in every group. Here is the hard truth. Mathematically, you usually cannot satisfy all of them at once. So choosing a definition is a human decision, about values.

What you can do

So what can you do? Check your data, and ask who is missing, and how the labels were made. Always measure performance per group, not just overall. Include the people affected in the discussion. Keep a person in the loop for high stakes decisions. And sometimes the right answer is simply, we should not automate this at all.

Summary