Seeing in 3D

A robot arm picking apples must know exactly where each one sits in space. Photos are flat, but the world isn't. Depth sensors and clever new methods now let machines capture complete three dimensional scenes.

• Robots need real positions
• Photos are flat
• The world isn't

From depth to a point cloud

With a depth map and the camera matrix, we can turn each pixel back into a real three D point. It's just our projection formula, run in reverse. Here's a simple depth map, with a near box. Run it. Thousands of points in space, measured in metres, and we can compute that the box is a little under ten centimetres wide. Now a robot knows where to reach.

import numpy as np

focal, cx, cy = 500.0, 80.0, 60.0
depth = np.full((120, 160), 2.0)
depth[40:80, 60:100] = 1.0

v, u = np.mgrid[0:120, 0:160]
z = depth
x = (u - cx) * z / focal
y = (v - cy) * z / focal
points = np.stack([x, y, z], axis=-1).reshape(-1, 3)

print("Points in 3D:", len(points))
print("Width of the near box:", round(float(40 * 1.0 / focal), 3), "metres")

Points in 3D: 19200
Width of the near box: 0.08 metres

Neural radiance fields

In twenty twenty, NeRF amazed the field. Train a small network on a few photos of a scene, and it can render that scene from new angles, with realism. The network learns the colour and density at any point in space. Two small ideas make it work. Encode positions as sine waves, so that fine detail can be learned. Then, along each ray, add up how much light gets caught. Run it. You can see both ideas. A newer method called Gaussian splatting does this far faster, and it's now in real products.

import numpy as np

def encode(p, n=4):
    return np.concatenate([f(2 ** i * np.pi * p) for i in range(n) for f in (np.sin, np.cos)])

point = np.array([0.1])
print("One number becomes:", len(encode(point)), "numbers")
print(encode(point).round(2))

samples = np.linspace(2.0, 6.0, 6)
density = np.array([0, 0, 1.5, 3.0, 0.2, 0])
step = samples[1] - samples[0]
weight = 1 - np.exp(-density * step)
print("Light caught at each depth:", weight.round(2))

One number becomes: 8 numbers
[ 0.31  0.95  0.59  0.81  0.95  0.31  0.59 -0.81]
Light caught at each depth: [0.   0.   0.7  0.91 0.15 0.  ]

Summary