Object Tracking

If you detect apples frame by frame, and simply add them up, you'll count the same apple thirty times. You need to know that this apple is the same one as in the last frame. That's tracking, and it turns detections into identities.

• Thirty frames, one apple
• Don't count it thirty times
• Detections become identities

Keeping identities

A tracker gives each object an ID, and keeps it from frame to frame. Even when objects cross paths or briefly vanish, a good tracker holds on. In each new frame, we detect again, and then link each detection to an existing track.

Matching tracks to detections

How do we link them? Compare every track with every new detection, using I o U. High overlap means it's probably the same object. The Hungarian algorithm finds the best overall pairing. Run it. Track one links to the detection that barely moved, even though the list order swapped. Trackers like SORT and ByteTrack add a motion model, and Ultralytics gives you this in one line, with model dot track.

import numpy as np
from scipy.optimize import linear_sum_assignment

def iou(a, b):
    x1, y1 = max(a[0], b[0]), max(a[1], b[1])
    x2, y2 = min(a[2], b[2]), min(a[3], b[3])
    both = max(0, x2 - x1) * max(0, y2 - y1)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return both / (area(a) + area(b) - both + 1e-9)

tracks = {1: (10, 10, 60, 60), 2: (100, 40, 150, 90)}
detections = [(108, 44, 158, 94), (16, 12, 66, 62)]

ids = list(tracks)
cost = np.array([[1 - iou(tracks[i], d) for d in detections] for i in ids])
rows, cols = linear_sum_assignment(cost)

for r, c in zip(rows, cols):
    print(f"Track {ids[r]} continues as detection {detections[c]}")

Track 1 continues as detection (16, 12, 66, 62)
Track 2 continues as detection (108, 44, 158, 94)

Counting across a line

Tracks make counting easy. Draw a virtual line. When an ID crosses it, count it, just one time. Here are three tracks, with their positions over time. We check each step, to see if it crossed. Run it. Two passed downward, and one went back up. That's exactly how AppleEye counts fruit on the belt, and it's how shop people-counters work too.

tracks = {7: [120, 180, 240, 310], 8: [300, 260, 240, 200], 9: [100, 150, 190, 220]}
line = 250

going_down = going_up = 0
for tid, ys in tracks.items():
    for a, b in zip(ys, ys[1:]):
        if a < line <= b:
            going_down += 1
        elif a >= line > b:
            going_up += 1

print("Apples passing down the line:", going_down)
print("Passing back up:", going_up)

Apples passing down the line: 1
Passing back up: 1

Summary