Cleaning Up and Scoring
NMS and mean average precision.
Ask a detector about one apple, and it might say, here, and here, and here, with a dozen overlapping boxes. Then how do you decide whether a detector is any good, when it can both miss and over-count? Let's solve both of these problems.
- A dozen boxes per apple
- Keep just one
- How good is a detector?
Non-maximum suppression
Here's the whole algorithm. Sort the boxes by score. Keep the best one. Delete any others that overlap it heavily. Repeat. Run it. Three overlapping boxes collapse into the single best one, and the separate box stays. A setting of point five is common. Lower it and you may delete real neighbours, like apples touching each other.
def iou(a, b):
x1, y1, x2, y2 = max(a[0], b[0]), max(a[1], b[1]), min(a[2], b[2]), min(a[3], b[3])
both = max(0, x2 - x1) * max(0, y2 - y1)
area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
return both / (area(a) + area(b) - both)
def nms(boxes, scores, limit=0.5):
order = sorted(range(len(boxes)), key=lambda i: -scores[i])
keep = []
while order:
best = order.pop(0)
keep.append(best)
order = [i for i in order if iou(boxes[best], boxes[i]) < limit]
return keep
boxes = [(10, 10, 60, 60), (12, 14, 62, 64), (8, 8, 58, 58), (100, 100, 150, 150)]
scores = [0.9, 0.8, 0.7, 0.85]
print("Boxes kept:", nms(boxes, scores))Boxes kept: [0, 3]
Mean average precision
To score a detector, we rank its detections by confidence, and walk down the list. At each step, we compute precision and recall. Run it. Going down, recall rises, as we find more apples. Precision wobbles down, as more mistakes creep in. The area under that curve is called average precision. Averaged over all classes, it's mean average precision, or MAP, the standard score for detectors.
import numpy as np
scores = np.array([0.95, 0.9, 0.85, 0.8, 0.7, 0.6, 0.5, 0.4])
hit = np.array([1, 1, 0, 1, 1, 0, 1, 0])
total_real = 6
tp = np.cumsum(hit)
fp = np.cumsum(1 - hit)
precision = tp / (tp + fp)
recall = tp / total_real
for s, p, r in zip(scores, precision, recall):
print(f"confidence {s:.2f} precision {p:.2f} recall {r:.2f}")
ap = np.trapezoid(precision, recall)
print("Area under the curve:", round(float(ap), 3))confidence 0.95 precision 1.00 recall 0.17 confidence 0.90 precision 1.00 recall 0.33 confidence 0.85 precision 0.67 recall 0.33 confidence 0.80 precision 0.75 recall 0.50 confidence 0.70 precision 0.80 recall 0.67 confidence 0.60 precision 0.67 recall 0.67 confidence 0.50 precision 0.71 recall 0.83 confidence 0.40 precision 0.62 recall 0.83 Area under the curve: 0.529
Summary
NMS keeps the best box, and MAP gives you one score. You now understand what's inside a detector. Let's run a real one, and then train it on your own data.
Key takeaways- NMS keeps the best box
- Precision and recall curve
- mAP is the standard score