Filters and Convolution

Almost all of image processing, and all of the deep networks that came later, are built on one single operation. Slide a small window over the picture. At each spot, combine the pixels under it. That's a convolution, and it's the most important idea in this course.

• One single operation
• Slide a small window
• The most important idea here

A filter sliding

Here's a small filter sliding across a simple image. It multiplies the numbers under it, adds them together, and writes one new number. This filter looks for vertical edges. Where there's nothing, it gives zero. Where brightness changes, it gives a strong signal. The result is a map of where this pattern exists.

Removing noise

Different filters do different jobs. Here's a noisy photo, like a phone camera in the dark. A box blur simply averages neighbours. A Gaussian blur gives more weight to the centre, and it looks more natural. A median blur takes the middle value, and it's great at removing specks. Run it, and compare them. The noise level drops a lot. Almost every vision pipeline begins with a little smoothing, as noise causes false results later.

import cv2
import numpy as np
import matplotlib.pyplot as plt
from skimage import data

gray = data.camera()
noisy = np.clip(gray + np.random.default_rng(0).normal(0, 25, gray.shape), 0, 255).astype(np.uint8)

box = cv2.blur(noisy, (5, 5))
gauss = cv2.GaussianBlur(noisy, (5, 5), 0)
median = cv2.medianBlur(noisy, 5)

fig, ax = plt.subplots(1, 4, figsize=(10, 3))
for a, im, n in zip(ax, [noisy, box, gauss, median], ["Noisy", "Box", "Gaussian", "Median"]):
    a.imshow(im, cmap="gray"); a.set_title(n); a.axis("off")
plt.tight_layout(); plt.savefig("blur.png", dpi=110)
print("Noise before:", round(float(np.std(noisy.astype(int) - gray)), 1))
print("After Gaussian:", round(float(np.std(gauss.astype(int) - gray)), 1))

Noise before: 23.8
After Gaussian: 11.1

Summary