How Cameras Work

How big is that apple? A photo can't tell you. A small apple close by looks the same as a large one far away. To measure the real world, we have to understand how a camera turns three dimensions into two.

• How big is that apple?
• Near small equals far large
• 3D to 2D loses depth

The pinhole camera

The simplest camera is a box with a tiny hole. Light from the top of a tree passes through the hole, and lands at the bottom of the back wall. So the image is upside down. Further objects look smaller, because rays are closer together. Every camera follows this model, more or less. And it explains why one image can't tell you distance.

Projecting points

We can write this as one matrix, called K. It holds the focal length, and the image centre. Multiply a three D point by K, and divide by its depth. Here are three points. The first two have the same x and y, but one's twice as far away. Run it. The far one lands closer to the centre of the image. That's why railway lines appear to meet in the distance.

import numpy as np

fx, fy, cx, cy = 800.0, 800.0, 320.0, 240.0
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])

points = np.array([[0.1, 0.0, 2.0], [0.1, 0.0, 4.0], [-0.2, 0.1, 2.0]])

for X in points:
    u, v, w = K @ X
    print(f"3D {X}  ->  pixel ({u / w:.0f}, {v / w:.0f})")

3D [0.1 0.  2. ]  ->  pixel (360, 240)
3D [0.1 0.  4. ]  ->  pixel (340, 240)
3D [-0.2  0.1  2. ]  ->  pixel (240, 280)

Calibrating a camera

How do you find K for a real camera? Take photos of a known pattern, usually a chessboard, from several angles. Calibrate camera works out the focal length, the centre, and the lens distortion. Here we simulate twelve views of a board, with a known camera. Run it. It recovers the focal length of eight hundred, and the centre, with a tiny error. Every serious measurement system starts with this step.

import cv2
import numpy as np

objp = np.zeros((6 * 9, 3), np.float32)
objp[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2)

K_true = np.array([[800, 0, 320], [0, 800, 240], [0, 0, 1]], float)
rng = np.random.default_rng(0)
objs, imgs = [], []
for _ in range(12):
    r = rng.uniform(-0.4, 0.4, 3)
    t = np.array([rng.uniform(-4, -2), rng.uniform(-3, -1), rng.uniform(12, 20)])
    pts, _ = cv2.projectPoints(objp, r, t, K_true, None)
    objs.append(objp); imgs.append(pts.astype(np.float32))

err, K, dist, _, _ = cv2.calibrateCamera(objs, imgs, (640, 480), None, None)
print("Focal length found:", K[0, 0].round(1))
print("Centre found:", K[0, 2].round(1), K[1, 2].round(1))
print("Pixel error:", round(err, 4))

Focal length found: 800.0
Centre found: 320.0 240.0
Pixel error: 0.0

Summary