Scoring Detectors

Every detector paper quotes a number called MAP. Most people use it, and few can explain it. It's actually simple. And once you've worked through one example by hand, you'll never be confused by it again.

• Every paper quotes mAP
• Few can explain it
• One example fixes that

Average precision by hand

A detector gives us eight boxes, with confidence scores, sorted from high to low. Each one is either correct or wrong, judged by its overlap with a real object. There are six real objects. Walk down the list. After each box, precision asks how many so far are correct, and recall asks how many objects we've found. Run it. See the first line? One box, correct, so precision is one, and recall is a sixth. As we accept lower scores, recall rises, and precision wobbles down. Average precision summarises that whole trade off in one number. Here, it's point seven one.

import numpy as np

scores = np.array([.95, .90, .85, .80, .70, .60, .50, .40])
correct = np.array([1, 1, 0, 1, 1, 0, 1, 0])
truth = 6

tp = correct.cumsum()
precision = tp / np.arange(1, len(correct) + 1)
recall = tp / truth

for s, p, r in zip(scores, precision, recall):
    print(f"Score {s:.2f}: precision {p:.2f}, recall {r:.2f}")

ap = sum(np.max(precision[recall >= t], initial=0) for t in np.linspace(0, 1, 11)) / 11
print(f"Average precision: {ap:.3f}")

Score 0.95: precision 1.00, recall 0.17
Score 0.90: precision 1.00, recall 0.33
Score 0.85: precision 0.67, recall 0.33
Score 0.80: precision 0.75, recall 0.50
Score 0.70: precision 0.80, recall 0.67
Score 0.60: precision 0.67, recall 0.67
Score 0.50: precision 0.71, recall 0.83
Score 0.40: precision 0.62, recall 0.83
Average precision: 0.712

Reading detection scores

MAP is just the average of this score across all of your classes. Then there are the overlap settings. MAP fifty counts a box as correct if it overlaps by at least half. MAP fifty to ninety five averages across stricter limits, so that it rewards tight boxes. Small objects often score far lower, so check those separately.

Summary