Bias in Language Models

Embeddings learn from billions of sentences written by people. So they learn what people wrote, including our stereotypes. In twenty sixteen, researchers showed that a famous model completed man is to programmer as woman is to homemaker. Nobody put that in. It was simply in the data.

• They learn from what people wrote
• Including our stereotypes
• It was in the data

Measuring a bias

We make a direction in meaning, by subtracting a sentence about a man from a sentence about a woman. Then we project some job titles onto that direction. Plus means closer to the woman sentence, and minus means closer to the man sentence. Run it. Nurse and receptionist lean one way. Mechanic, engineer, and carpenter lean the other. This is a modern model, in twenty twenty six. If you ranked job applicants with these embeddings, that lean could quietly affect the results.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
jobs = ["nurse", "engineer", "receptionist", "carpenter", "librarian", "mechanic"]
words = model.encode(["she is a woman", "he is a man"] + [f"a {j}" for j in jobs], normalize_embeddings=True)
gap = words[0] - words[1]

for job, v in sorted(zip(jobs, words[2:]), key=lambda p: -float(p[1] @ gap)):
    print(f"{job:<14} {float(v @ gap):+.3f}")

nurse          +0.233
receptionist   +0.165
librarian      +0.040
mechanic       -0.053
engineer       -0.057
carpenter      -0.117

What to do about it

You can't fully remove this, but you can manage it. Test your system across names, genders, and dialects. Swap words in test sentences, and check that outputs don't change unfairly. Keep people involved in decisions that affect someone's life. Be especially careful with hiring, lending, and medical uses. And check your training data, as much of today's text is written by a small part of the world.

Summary