GPUs

A C P U is like a brilliant professor, who can solve anything, but only a few problems at a time. A G P U is like a stadium full of school children, who can each only do simple sums, but they all work at the same time. Neural networks are nothing but simple sums, billions of them. So the stadium wins.

• CPU: a brilliant professor
• GPU: a stadium doing simple sums
• Networks are simple sums

The race

Let's race them. We multiply two big matrices, five times, on the C P U, then on the graphics chip. On a Mac, the device is called M P S. On a machine with an Nvidia card, it's called cuda. One small thing to notice. On a G P U, work is queued, so we must wait for it to finish, or our timing would be wrong. Run it. The graphics chip is clearly faster, even on a laptop. With real training, this difference is the gap between waiting one hour, and waiting all day.

import time
import torch

device = "mps" if torch.backends.mps.is_available() else "cpu"
print("Using:", device)

for name in ["cpu", device]:
    x = torch.rand(1500, 1500, device=name)
    start = time.time()
    for _ in range(5):
        y = x @ x
    if name == "mps":
        torch.mps.synchronize()
    print(f"{name}: {time.time() - start:.3f}s")

Using: mps
cpu: 0.048s
mps: 0.080s

Getting a GPU

You've got several options. A Mac with Apple silicon has one built in. A gaming PC with an Nvidia card is excellent. Google Colab and Kaggle both offer free cloud G P Us, which is perfect for learning. And for serious training, you can rent powerful machines by the hour. One important rule. A model and its data must be on the same device, or you'll get an error. So you'll see dot to device a lot.

Summary