Generating Images
Diffusion and inpainting.
So far, we've taught computers to look. Now, we'll teach them to imagine. In just a few years, image generation went from blurry blobs to pictures that win art contests. And for Orchard Vision, it's surprisingly practical, not just fun.
- From looking to imagining
- Blurry blobs to art prizes
- Surprisingly practical
The path from image to noise
Diffusion models learn by watching images get destroyed. Here's a photo, slowly mixed with random noise, until nothing remains. Run it. Look along the row. A generator is trained to undo a single small step of this. Then we start with pure noise, and run it backward, step by step, guided by your words. Our Deep Learning course builds a real one from scratch.
import numpy as np
import matplotlib.pyplot as plt
from skimage import data, transform
image = transform.resize(data.astronaut(), (128, 128)) * 2 - 1
rng = np.random.default_rng(0)
fig, ax = plt.subplots(1, 5, figsize=(10, 2.4))
for a, keep in zip(ax, [1.0, 0.8, 0.5, 0.2, 0.0]):
noisy = np.sqrt(keep) * image + np.sqrt(1 - keep) * rng.normal(size=image.shape)
a.imshow(((noisy + 1) / 2).clip(0, 1)); a.set_title(f"{keep:.0%} image"); a.axis("off")
plt.tight_layout(); plt.savefig("noise.png", dpi=110)
print("Generators learn to walk this path backward")Generators learn to walk this path backward
Filling in missing pixels
A very useful version of this is inpainting. Remove part of an image, and fill it in. Here's a simple classic method, which spreads neighbouring colours into the gap. Run it. It fills in the black box, with a smooth guess. Diffusion models do this with real understanding, and that's how phones can remove people from photos.
import cv2
import numpy as np
import matplotlib.pyplot as plt
from skimage import data
image = data.astronaut()
mask = np.zeros(image.shape[:2], np.uint8)
cv2.rectangle(mask, (220, 60), (300, 130), 255, -1)
broken = image.copy()
broken[mask > 0] = 0
fixed = cv2.inpaint(broken, mask, 5, cv2.INPAINT_TELEA)
fig, ax = plt.subplots(1, 2, figsize=(8, 4))
ax[0].imshow(broken); ax[1].imshow(fixed)
for a in ax: a.axis("off")
plt.tight_layout(); plt.savefig("inpaint.png", dpi=110)
print("Filled pixels:", int((mask > 0).sum()))Filled pixels: 5751
Summary
Diffusion learns to reverse noise, and inpainting fills gaps. So why does a fruit farm care about generated images? Because the hardest thing to get is examples of rare problems.
Key takeaways- Forward adds noise
- Generators walk it backward
- Inpainting fills gaps