Scoring Masks

A doctor's model reports ninety nine point eight percent accuracy at finding tumours. It sounds incredible. In fact, that model predicts nothing at all, every single time. Tumours cover so little of an image, that doing nothing scores brilliantly.

• 99.8% at finding tumours
• It predicts nothing at all
• Doing nothing scores brilliantly

IoU, Dice, and the trap

Here are two overlapping squares, a true mask and a predicted one. I o U is the overlap divided by the union. Dice is twice the overlap, divided by the two areas added together. Run it. They're related, and Dice is always the larger number. Now the trap. Here's a tiny object, and a model which predicts nothing. Run it. Its pixel accuracy is ninety nine point eight percent, and its Dice score is zero. So for small objects, never use pixel accuracy. Doctors use Dice for this exact reason.

import numpy as np

truth = np.zeros((100, 100), bool); truth[20:70, 20:70] = True
pred = np.zeros((100, 100), bool); pred[30:80, 30:80] = True

both = (truth & pred).sum()
iou = both / (truth | pred).sum()
dice = 2 * both / (truth.sum() + pred.sum())
print(f"IoU {iou:.3f}, Dice {dice:.3f}")

tiny = np.zeros((100, 100), bool); tiny[50:55, 50:55] = True
blank = np.zeros((100, 100), bool)
pixels = (tiny == blank).mean()
print(f"Predict nothing, pixel accuracy {pixels:.1%}, Dice 0.000")

IoU 0.471, Dice 0.640
Predict nothing, pixel accuracy 99.8%, Dice 0.000

Summary