Object Detection

So far, Aria can say, this is a photo of a dog. But a self driving car needs to know where the dog is, how many there are, and whether there's a child behind it. Answering what isn't enough. We also have to answer where.

• What isn't enough
• We also need where
• And how many

What a detector outputs

A detector gives a list of boxes. For each box, four numbers for its position, a class, and a confidence score. Modern detectors like YOLO, which stands for you only look once, split the image into a grid. Each cell predicts boxes, all at the same time, in one pass of the network. That's why they can run in real time on video.

Measuring a box

How do we score a predicted box? Intersection over union. Divide the area where the boxes overlap, by the total area they cover. A perfect match gives one, and no overlap gives zero. We test three boxes against the truth. Run it. A great box scores high, and a poor one low. A common rule says that a detection counts as correct when this is above point five.

def iou(a, b):
    x1, y1 = max(a[0], b[0]), max(a[1], b[1])
    x2, y2 = min(a[2], b[2]), min(a[3], b[3])
    overlap = max(0, x2 - x1) * max(0, y2 - y1)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return overlap / (area(a) + area(b) - overlap)

truth = (10, 10, 60, 60)
print("Great box: ", round(iou(truth, (12, 12, 62, 62)), 3))
print("Okay box:  ", round(iou(truth, (30, 30, 80, 80)), 3))
print("Wrong box: ", round(iou(truth, (70, 70, 90, 90)), 3))

Great box:  0.855
Okay box:   0.22
Wrong box:  0.0

Removing duplicates

Detectors tend to fire many boxes around one object. Non maximum suppression cleans these up. Keep the box with the top score, remove any others that overlap it too much, and repeat. We've got three overlapping boxes, and one separate. Run it. It kept the best of the cluster, and the separate box. Our Computer Vision course builds real detectors, and trains them on custom data.

import torch
from torchvision.ops import nms

boxes = torch.tensor([[10, 10, 60, 60], [12, 12, 62, 62], [14, 8, 58, 64],
                      [100, 100, 150, 150]], dtype=torch.float32)
scores = torch.tensor([0.9, 0.85, 0.7, 0.8])

keep = nms(boxes, scores, iou_threshold=0.5)
print("Boxes before:", len(boxes))
print("Kept:", keep.tolist())

Boxes before: 4
Kept: [0, 3]

Summary