Fixing Perspective
Making a tilted camera square.
There's no room to mount the camera straight above the belt, so it's fixed on a pole, at an angle. Now apples at the far end look smaller, and all the size measurements are wrong. But there's a remarkable fact. If everything lies on a flat surface, one simple matrix can undo the tilt completely.
- The camera sits at an angle
- Far apples look smaller
- One matrix undoes the tilt
Four points are all you need
A homography maps one flat plane onto another. Mark four points on the belt in the photo, then say where they should be in a square view. That's enough to calculate the whole mapping. Then each pixel can be moved. It's how document scanner apps straighten a page, and how sports graphics draw lines on a pitch.
Warping and unwarping
We simulate a tilted camera, by warping a flat image with four chosen corners. Then we fix it by calculating the opposite mapping, from the same four points. Finally, we compare the result with the original. Run it. The average difference is under half a brightness level, out of two hundred and fifty five, so we've recovered the flat view almost perfectly. On a real system, you'd mark four known points on the belt once, during setup, and then the same matrix fixes every frame.
import cv2
import numpy as np
from belt import belt
img, _, _ = belt(seed=3)
tilt = np.float32([[60, 40], [590, 10], [630, 470], [10, 440]])
flat = np.float32([[0, 0], [640, 0], [640, 480], [0, 480]])
camera = cv2.warpPerspective(img, cv2.getPerspectiveTransform(flat, tilt), (640, 480))
fixed = cv2.warpPerspective(camera, cv2.getPerspectiveTransform(tilt, flat), (640, 480))
print("Average difference after fixing:", round(cv2.absdiff(img, fixed).mean(), 1))
cv2.imwrite("persp.png", np.hstack([camera, fixed]))Average difference after fixing: 0.4
Also uses belt.py
import numpy as np
import cv2
def belt(n=7, seed=1, size=(480, 640), bruises=2, light=1.0):
"""Draw a simulated conveyor belt of apples. Returns the image and the true boxes."""
rng = np.random.default_rng(seed)
h, w = size
img = np.full((h, w, 3), (70, 75, 80), np.uint8)
img = cv2.add(img, rng.normal(0, 6, img.shape).clip(0, 30).astype(np.uint8))
boxes, spots, taken = [], [], []
while len(boxes) < n:
r = int(rng.integers(32, 48))
x, y = int(rng.integers(r + 5, w - r - 5)), int(rng.integers(r + 5, h - r - 5))
if any((x - a) ** 2 + (y - b) ** 2 < (r + c + 8) ** 2 for a, b, c in taken):
continue
taken.append((x, y, r))
red = int(rng.integers(150, 225))
cv2.circle(img, (x, y), r, (35, 40, red), -1, cv2.LINE_AA)
cv2.circle(img, (x - r // 3, y - r // 3), r // 4, (90, 90, min(255, red + 30)), -1, cv2.LINE_AA)
boxes.append((x - r, y - r, x + r, y + r))
for x, y, r in taken[:bruises]:
cv2.ellipse(img, (x + r // 3, y + r // 4), (r // 3, r // 4), 20, 0, 360, (25, 45, 70), -1, cv2.LINE_AA)
spots.append((x, y, r))
img = cv2.GaussianBlur(img, (3, 3), 0)
return np.clip(img * light, 0, 255).astype(np.uint8), boxes, spotsSummary
Four points can square up any flat surface. Next, the same idea can join several photos into one.
Key takeaways- Homographies map flat planes
- Four points define one
- Set it up once