Vision for Robots

Everything so far has been about looking. A robot has to look and act, in a world that changes as it moves. If it's wrong by a centimetre, it drops the apple. And it needs to know where it is. That turns vision into a completely different kind of problem.

• So far, just looking
• A robot looks and acts
• A centimetre matters

What robots need

A robot picking apples needs a sequence of things. It detects the fruit. It works out the fruit's position in 3D, using stereo or depth cameras. It converts that into arm coordinates, through a step called hand eye calibration. It plans a path around the branches. And it keeps watching as it moves, correcting all the way. That final loop is called visual servoing.

Knowing where you are

How does a drone know where it is, with no GPS? Visual SLAM. It tracks features between frames, builds a map, and locates itself in that map, both at once. Your phone does this for augmented reality. Robot vacuums use it. It's built from pieces that you've already learned, such as keypoints, matching, and RANSAC.

Training robots in simulation

Real robots are slow and costly to train, and they break. So researchers train in simulation, running thousands of copies, far faster than real time. The problem is that a simulation isn't real. Domain randomisation fixes that, by varying the colours, the lighting, and the textures, wildly. Reality then looks like just one more variation. It's our augmentation lesson, applied to whole worlds.

Summary