Evaluating Text Models

Scribe scores ninety percent. Is that good? It depends. What if the rare legal threats, the most important messages of all, are the ones it misses? And here's a stranger question. Do even humans agree on the right answer?

• Ninety percent. Is that good?
• What about rare legal threats?
• Do humans even agree?

Reading a report

Here's a typical result. Ten messages, and most are delivery problems. Accuracy looks fine at eighty percent. But now read the report, class by class. Legal has a recall of zero. Scribe missed the only legal threat. Macro average treats all classes equally, so it reveals this kind of failure. Weighted average hides it. For rare but important classes, always read each row.

from sklearn.metrics import classification_report, confusion_matrix

truth = ["delivery"] * 6 + ["refund"] * 3 + ["legal"] * 1
pred  = ["delivery"] * 6 + ["refund"] * 2 + ["delivery"] + ["delivery"]

print(classification_report(truth, pred, zero_division=0, digits=2))

              precision    recall  f1-score   support

    delivery       0.75      1.00      0.86         6
       legal       0.00      0.00      0.00         1
      refund       1.00      0.67      0.80         3

    accuracy                           0.80        10
   macro avg       0.58      0.56      0.55        10
weighted avg       0.75      0.80      0.75        10

Do humans agree?

Text labels come from people, and people disagree. Is this message angry, or just firm? Two labellers annotate the same eight messages. Plain agreement is seventy five percent. But two people flipping coins would also agree half of the time. Cohen's kappa corrects for chance. Run it. Kappa is well below the plain number. A good rule of thumb. If your humans only agree seventy five percent of the time, then no model can honestly be expected to reach ninety five. Always measure this first.

from sklearn.metrics import cohen_kappa_score

alice = ["angry", "fine", "angry", "fine", "angry", "fine", "fine", "angry"]
bilal = ["angry", "fine", "fine",  "fine", "angry", "angry", "fine", "angry"]

same = sum(a == b for a, b in zip(alice, bilal)) / len(alice)
print(f"Plain agreement: {same:.2f}")
print(f"Kappa:           {cohen_kappa_score(alice, bilal):.2f}")

Plain agreement: 0.75
Kappa:           0.50

Summary