Seeing Inside the Black Box

Saliency and Grad-CAM.

⏱ 2 min read▶ Video at 1:59:25🎓 Free

Watch this lesson in the Deep Learning courseWatch on YouTube ↗Full Deep Learning course

A famous project trained a network to spot wolves. It looked amazing, right up until researchers checked what it was looking at. It had learned that wolves appear in snow. It was a snow detector. A network can be right, for completely the wrong reasons. So we need ways to ask, what were you looking at?

From the video: Seeing Inside the Black Box

A brilliant wolf detector
It was detecting snow
Right for the wrong reasons

A saliency map

Remember that gradients tell us how much the output changes, when an input changes. Normally, we use that for the weights. But we can also take the gradient with respect to the pixels. That tells us which pixels matter the most for this decision. Run it. The bright areas are the pixels that most affect the score, and they trace the digit itself. So this model's looking at the right thing. If they'd lit up in the corners, we'd have a problem.

saliency.pypython

import torch
from torch import nn
from torchvision import datasets, transforms
import matplotlib.pyplot as plt

train = datasets.MNIST("data", train=True, transform=transforms.ToTensor())
torch.manual_seed(0)
model = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                      nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
                      nn.Flatten(), nn.Linear(32 * 7 * 7, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
dl = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True)
for i, (x, y) in enumerate(dl):
    loss = nn.functional.cross_entropy(model(x), y)
    opt.zero_grad(); loss.backward(); opt.step()
    if i == 250:
        break

image = train[7][0].unsqueeze(0).requires_grad_()
score = model(image)[0].max()
score.backward()
heat = image.grad[0, 0].abs()

fig, ax = plt.subplots(1, 2, figsize=(8, 4))
ax[0].imshow(image[0, 0].detach(), cmap="gray"); ax[0].set_title("Input")
ax[1].imshow(heat, cmap="hot"); ax[1].set_title("What mattered")
for a in ax: a.axis("off")
plt.tight_layout()
plt.savefig("saliency.png", dpi=120)
print("Prediction:", model(image).argmax().item())

Output

Prediction: 3

Grad-CAM

Pixel maps can be noisy. Grad Cam gives a smoother picture. It uses the last convolution layer, which has rich meaning, and a rough sense of where things are. We grab that layer's output with a hook. We find how much the winning class depends on each map, and combine them. Run it. The maps are seven by seven, so we get a small heat map, which we can stretch over the image. It shows which regions drove the decision, and it's a standard tool for checking vision models.

gradcam.pypython

import torch
from torch import nn
from torchvision import models

model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT).eval()

store = {}
model.layer4.register_forward_hook(lambda m, i, o: store.update(maps=o))

x = torch.rand(1, 3, 224, 224)
scores = model(x)
target = scores[0].argmax()

grads = torch.autograd.grad(scores[0, target], store["maps"])[0]
weights = grads.mean(dim=(2, 3), keepdim=True)
cam = (weights * store["maps"]).sum(1).relu()[0]

print("Feature maps:", list(store["maps"].shape))
print("Heat map:", list(cam.shape))

Output

Feature maps: [1, 512, 7, 7]
Heat map: [7, 7]

Summary

Saliency and Grad Cam ask a model where it's looking, and they can catch wolf detectors before they ship. But that's not the only strange weakness that networks have. Next, the attack that can fool a model, with changes that you can't even see.

Key takeaways

Right can be for the wrong reason
Gradients can point at pixels
Grad-CAM shows regions

Next up: Adversarial Examples — fooling a neural network

❮ Deep Reinforcement Learning Adversarial Examples ❯