How Diffusion Works

How can a machine create a brand new picture, from just a sentence? The answer is one of the most surprising ideas in A I. It starts with pure static noise, like an old television with no signal, and gradually removes the noise, until a picture appears. In this lesson, you will see how.

• Start with pure noise
• Remove noise, step by step
• A picture appears

Step one: destroying a picture

First, the training idea. Take a real photograph. Now add a little random noise. Then a little more. Again, and again, for many steps, until nothing remains, but pure static. This forward process is easy, and needs no learning. The clever part is that, at every step, we know exactly how much noise we added.

Step two: learning to undo it

Now the learning begins. We show a neural network a noisy picture, and ask, what noise was added? It predicts the noise, and we compare it with the truth. Over millions of images, the network gets very good at it. In effect, it learns what real pictures look like, from the inside, because to remove noise correctly, it must know what should be there instead.

Step three: creating from nothing

And now the magic. To create a new picture, we start with pure random noise. The network predicts the noise, and we remove a small part of it. Then again. And again. About fifty times. With each step, shapes emerge from the static. First, rough colours, then outlines, then details. At the end, a brand new image, that never existed before. Different starting noise, gives a different picture.

Why this works so well

Diffusion became the dominant way to make images, for good reasons. Training is stable, unlike older methods. It produces sharp, detailed, and varied pictures. It can be guided, by text, and by other images. And it scales, bigger models, and more data, keep improving quality. Famous examples include Stable Diffusion, D A L L E, Midjourney, and Imagen.

Summary