Convolutions

A convolution sounds intimidating, but you've used one if you've ever applied a filter to a photo. Blur, sharpen, and edge detect are all convolutions. The only difference in deep learning is that the network invents its own filters.

• You've used these on photos
• Blur, sharpen, edge detect
• The network invents its own

Watching a filter slide

On the left is a tiny image, a white square outline. In the middle is a filter, a small grid of three by three numbers. This one looks for vertical edges, with negatives on the left, and positives on the right. Now we slide it across the image. At each position, we multiply the overlapping numbers and add them up. Where there's no edge, the result is zero. On a dark to bright edge, it's strongly positive. Bright to dark gives a negative. The result on the right is called a feature map. It's a map of where this pattern was found.

Filters on a real image

Let's apply three hand made filters to a real image, of a piece of clothing. Conv two d does the sliding. One finds vertical edges, one finds horizontal edges, and one blurs. Run it. Each filter pulls out a different view of the image. For decades, engineers designed filters like these by hand. A convolutional network learns its own, with backpropagation, using the very same training loop that you already know.

import torch
import torch.nn.functional as F
from torchvision import datasets
import matplotlib.pyplot as plt

image = datasets.FashionMNIST("data", train=False).data[3].float() / 255
x = image[None, None]

kernels = {
    "Vertical edges": [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]],
    "Horizontal edges": [[-1, -2, -1], [0, 0, 0], [1, 2, 1]],
    "Blur": [[1 / 9] * 3] * 3,
}
fig, ax = plt.subplots(1, 4, figsize=(10, 3))
ax[0].imshow(image, cmap="gray"); ax[0].set_title("Original")
for a, (name, k) in zip(ax[1:], kernels.items()):
    out = F.conv2d(x, torch.tensor(k, dtype=torch.float32)[None, None], padding=1)
    a.imshow(out[0, 0], cmap="gray"); a.set_title(name)
for a in ax: a.axis("off")
plt.tight_layout()
plt.savefig("filters.png", dpi=120)
print("Same image, three learned-style filters")

Same image, three learned-style filters

Three settings of a convolution

A convolution layer has three main settings. Kernel size is the size of the filter, and three by three is the usual choice. Stride is how far it jumps with each step. A stride of two halves the image. And padding adds a border, so that the edges aren't lost, and the size stays the same. In a real layer, there are many filters at once, often dozens, each learning to spot its own pattern.

Summary