Fairness

A company built a model to screen job applications, trained on ten years of past hiring. It learned to mark down anything mentioning a women's college. Nobody asked it to do that. It simply learned the patterns of the past. Data records what happened, not what should have happened.

• A model screens job applications
• It learned from past hiring
• Data records what happened

Checking selection rates

Before you ship a decision system, compare its outcomes across groups. Here, a score decides who gets an offer. The score runs a little higher for group A. Run it. Group A gets the offer sixty two percent of the time, and group B thirty eight. The ratio is about point six two. In US hiring, there's a rule of thumb called four fifths. A ratio under point eight is a warning sign which needs investigating. It doesn't prove discrimination, but you can't ignore it.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 4000
group = rng.choice(["A", "B"], n)
score = rng.normal(np.where(group == "A", .55, .45), .15)
offer = score > .5
df = pd.DataFrame({"group": group, "offer": offer})

rates = df.groupby("group")["offer"].mean()
print(rates.round(3).to_string())
print("Ratio:", round(rates.min() / rates.max(), 2))

group
A    0.618
B    0.385
Ratio: 0.62

What to do about it

Fairness has no single formula, and different definitions can't all be met together. So decide with the people affected. Measure outcomes by group. Look at where the data came from. Remember that removing a sensitive column doesn't work, because other columns can stand in for it. Keep a person involved in serious decisions. And record your reasoning.

Summary