Text to Image

Pure diffusion makes random pictures. To get the picture you asked for, the words must steer the process. In this lesson, you will learn how text and images are connected, and what the guidance dial does.

• Random pictures are not enough
• Words must steer the process
• Connecting text and images

Connecting words and pictures

First, the model needs to understand your prompt. A text encoder turns your words into embeddings, the same idea as before. A famous model called C L I P, was trained on hundreds of millions of pictures with captions, so that a picture and its caption, land close together in the same map of meaning. Now the denoising network can be told, at every step, remove the noise in a way that fits this description.

1. Your prompta cat in a spacesuit
2. Text encoderwords become embeddings
3. Shared meaning mappictures and captions align
4. Guided denoisingeach step fits the prompt

The guidance dial

A setting called guidance scale, controls how strongly the prompt steers the picture. Low guidance gives the model freedom. Pictures look natural and varied, but may ignore parts of your prompt. High guidance forces it to follow the words closely, but pictures may look harsh, over saturated, or strange. Most users keep it in the middle, and adjust only when needed.

The latent trick

Working on every pixel would be slow. So Stable Diffusion uses a trick. A compressor, called an autoencoder, squeezes the image into a much smaller hidden space, called the latent space. Diffusion happens there, which is far cheaper. At the end, a decoder expands the result, back into a full picture. This is why modern image models can run on an ordinary graphics card.

Summary