Text to Image

You type, a cat wearing a spacesuit on the moon. Seconds later, a picture exists, that no one has ever made. That felt like magic when it first appeared. But you already know nearly all of the pieces. Let's put them together.

• Type a sentence, get a picture
• It felt like magic
• You already know the pieces

The pieces of Stable Diffusion

A modern image generator has three networks. A text encoder turns your sentence into a list of vectors. A U Net, the shape that you just learned, predicts noise, and it's told about your sentence through cross attention. And a V A E works in a compressed space, which makes everything far cheaper. Generation starts with random noise, in that small space. The U Net removes noise over a few dozen steps, guided by the words. Finally, the V A E decoder turns the result into a full image.

1. VAE decoderlatent to image
2. U-Net with cross attentionremoves noise, listens to text
3. Random noisein latent space
4. Text encodersentence to vectors

Text as vectors

Here's the first step. A text encoder turns each prompt into numbers. Here we use a small text model. We encode three prompts, and compare them. Run it. A red sports car, and a fast crimson automobile, are close in meaning, despite having almost no words in common. And fruit is far away. So the network that's drawing receives meaning, not just letters.

import torch
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")

prompts = ["a photo of a red sports car",
           "a fast crimson automobile",
           "a bowl of fresh fruit"]
vectors = model.encode(prompts, convert_to_tensor=True)

print("Prompt becomes:", list(vectors.shape))
print("Car vs automobile:", round(util.cos_sim(vectors[0], vectors[1]).item(), 3))
print("Car vs fruit:     ", round(util.cos_sim(vectors[0], vectors[2]).item(), 3))

Prompt becomes: [3, 384]
Car vs automobile: 0.481
Car vs fruit:      0.155

Classifier-free guidance

Here's a clever trick. How strongly should the image follow your words? In each step, the U Net makes two predictions, one with your text, and one with no text. We take the gap between them, and we push further in that direction. The guidance scale says how much. Run it. A scale of one uses the plain result. A larger scale exaggerates what the text adds. Around seven is a popular choice. Too high, and images look harsh and burned. That's the slider that you see in image generator apps.

import torch

torch.manual_seed(0)
empty = torch.tensor([0.20, -0.10, 0.05])
with_text = torch.tensor([0.32, -0.02, 0.01])

for scale in [1.0, 7.5, 20.0]:
    guided = empty + scale * (with_text - empty)
    print(f"guidance {scale:<5} {guided.round(decimals=2).tolist()}")

guidance 1.0   [0.3199999928474426, -0.019999999552965164, 0.009999999776482582]
guidance 7.5   [1.100000023841858, 0.5, -0.25]
guidance 20.0  [2.5999999046325684, 1.5, -0.75]

Summary