Adversarial Examples

In twenty fourteen, researchers made a discovery that shocked the field. They took a photo of a panda, and added a tiny, carefully chosen pattern of noise. To you, the photo didn't change at all. But the network was now very confident that it was looking at a gibbon. Neural networks don't see the way that we do.

• A panda, a tiny pattern of noise
• You see no change
• The network sees a gibbon

Attacking a model

The attack uses gradients, in reverse. Training changes weights to lower the loss. An attack changes pixels to raise it. We take the sign of the gradient, for each pixel, and nudge the image by a small amount epsilon. This is called the fast gradient sign method. Run it. Accuracy starts high. With a tiny amount of noise, it drops. With a little more, the model is badly broken, on images that you'd still read easily.

import torch
from torch import nn
from torchvision import datasets, transforms

train = datasets.MNIST("data", train=True, transform=transforms.ToTensor())
test = datasets.MNIST("data", train=False)
X = test.data[:1000].float().unsqueeze(1) / 255
y = test.targets[:1000]

torch.manual_seed(0)
model = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for _ in range(2):
    for a, b in torch.utils.data.DataLoader(train, batch_size=128, shuffle=True):
        loss = nn.functional.cross_entropy(model(a), b)
        opt.zero_grad(); loss.backward(); opt.step()

X_adv = X.clone().requires_grad_()
nn.functional.cross_entropy(model(X_adv), y).backward()

for eps in [0.0, 0.05, 0.1, 0.2]:
    attacked = (X + eps * X_adv.grad.sign()).clamp(0, 1)
    acc = (model(attacked).argmax(1) == y).float().mean()
    print(f"noise {eps:<5} accuracy {acc:.3f}")

noise 0.0   accuracy 0.958
noise 0.05  accuracy 0.569
noise 0.1   accuracy 0.106
noise 0.2   accuracy 0.000

Why it matters

This is a real risk. Researchers have put small stickers on stop signs, that fooled classifiers. Language models have their own version, called prompt injection. The best defence is adversarial training, where you include attacked examples while training. It's also a reminder that high accuracy doesn't mean that a model truly understands. So for any system where safety matters, test it against an attacker.

Summary