Stitching Images
One wide view from two cameras.
The belt is too wide for a single camera, so there are two, with some overlap. To count properly, you need one wide image. That's exactly how your phone makes a panorama, and now you know enough to build one.
- Two overlapping cameras
- You need one wide image
- That's how a panorama works
Building a panorama
We cut one long scene into two overlapping views. SIFT finds distinctive points in each. We match them, and use the ratio test, which keeps a match only if it's clearly better than the second best. Many of those matches will still be wrong. So RANSAC tries random small sets of matches, and keeps the transform that the largest number agree on. Run it. Thirty matches, and RANSAC trusted twenty one of them. And the shift that it found is exactly three hundred pixels, which is the true answer. It worked it out from the images alone.
import cv2
import numpy as np
from belt import belt
scene, _, _ = belt(n=14, size=(420, 1000), seed=2)
left, right = scene[:, :650], scene[:, 300:]
sift = cv2.SIFT_create()
ka, da = sift.detectAndCompute(left, None)
kb, db = sift.detectAndCompute(right, None)
pairs = cv2.BFMatcher().knnMatch(da, db, k=2)
matches = [m for m, n in pairs if m.distance < 0.75 * n.distance]
src = np.float32([kb[m.trainIdx].pt for m in matches])
dst = np.float32([ka[m.queryIdx].pt for m in matches])
H, inliers = cv2.findHomography(src, dst, cv2.RANSAC, 3.0)
pano = cv2.warpPerspective(right, H, (1000, 420))
pano[:, :650] = left
print(f"Matches: {len(matches)}, kept by RANSAC: {int(inliers.sum())}")
print("Shift found:", round(float(H[0, 2])), "pixels, truth 300")
cv2.imwrite("pano.png", pano)Matches: 30, kept by RANSAC: 21 Shift found: 300 pixels, truth 300
Also uses belt.py
import numpy as np
import cv2
def belt(n=7, seed=1, size=(480, 640), bruises=2, light=1.0):
"""Draw a simulated conveyor belt of apples. Returns the image and the true boxes."""
rng = np.random.default_rng(seed)
h, w = size
img = np.full((h, w, 3), (70, 75, 80), np.uint8)
img = cv2.add(img, rng.normal(0, 6, img.shape).clip(0, 30).astype(np.uint8))
boxes, spots, taken = [], [], []
while len(boxes) < n:
r = int(rng.integers(32, 48))
x, y = int(rng.integers(r + 5, w - r - 5)), int(rng.integers(r + 5, h - r - 5))
if any((x - a) ** 2 + (y - b) ** 2 < (r + c + 8) ** 2 for a, b, c in taken):
continue
taken.append((x, y, r))
red = int(rng.integers(150, 225))
cv2.circle(img, (x, y), r, (35, 40, red), -1, cv2.LINE_AA)
cv2.circle(img, (x - r // 3, y - r // 3), r // 4, (90, 90, min(255, red + 30)), -1, cv2.LINE_AA)
boxes.append((x - r, y - r, x + r, y + r))
for x, y, r in taken[:bruises]:
cv2.ellipse(img, (x + r // 3, y + r // 4), (r // 3, r // 4), 20, 0, 360, (25, 45, 70), -1, cv2.LINE_AA)
spots.append((x, y, r))
img = cv2.GaussianBlur(img, (3, 3), 0)
return np.clip(img * light, 0, 255).astype(np.uint8), boxes, spotsRANSAC is everywhere
RANSAC deserves a moment, because it's one of the most useful ideas in all of computer vision. The idea is simple. Assume that much of your data is wrong. Pick a tiny random sample, fit a model, and count how many other points agree. Repeat, and keep the winner. It's used for stitching, robot navigation, and 3D reconstruction.
Summary
Match, filter with RANSAC, and warp. Next, how to measure exactly what your camera's lens is doing.
Key takeaways- The ratio test filters matches
- RANSAC handles wrong matches
- It recovers the shift exactly