Motion and Video

A video is simply lots of photos, shown quickly. But between each frame is a very valuable signal. Motion. Our eyes are extremely sensitive to it, and machines can use it too, to find things without knowing what they are.

• Video is many photos
• Motion is valuable
• Find things by movement

Optical flow

Optical flow estimates how each pixel moved between two frames. We draw a circle, then move it a little in the second frame, by eight pixels right and three down. Farneback flow estimates the motion at every pixel. Run it. It gets close to the true movement. That's used in video compression, in stabilising shaky footage, and in self driving cars.

import cv2
import numpy as np

a = np.zeros((120, 160), np.uint8)
b = np.zeros((120, 160), np.uint8)
cv2.circle(a, (60, 60), 18, 255, -1)
cv2.circle(b, (68, 63), 18, 255, -1)

flow = cv2.calcOpticalFlowFarneback(a, b, None, 0.5, 3, 15, 3, 5, 1.2, 0)
mask = a > 0

print("Average motion in x:", round(float(flow[..., 0][mask].mean()), 1), "pixels")
print("Average motion in y:", round(float(flow[..., 1][mask].mean()), 1), "pixels")

Average motion in x: 7.2 pixels
Average motion in y: 2.7 pixels

Background subtraction

With a fixed camera, there's an easy way to find movement. Learn what the empty scene looks like, then flag anything that differs. Here's a simulated belt. A bright object arrives partway through. Run it. It flags the moving pixels, and ignores the slight noise in the background. Traffic counters and security cameras have been using this for decades, because it's cheap.

import cv2
import numpy as np

sub = cv2.createBackgroundSubtractorMOG2(history=40, varThreshold=25, detectShadows=False)
rng = np.random.default_rng(0)

for f in range(60):
    frame = np.full((120, 160), 90, np.uint8) + rng.integers(0, 8, (120, 160), dtype=np.uint8)
    if f > 40:
        cv2.circle(frame, (10 + (f - 40) * 6, 60), 12, 220, -1)
    fg = sub.apply(frame)

print("Moving pixels in the last frame:", int((fg > 0).sum()))
print("Share of the image:", f"{(fg > 0).mean():.1%}")

Moving pixels in the last frame: 440
Share of the image: 2.3%

Summary