An image generator turns a field of static into your picture by removing the noise a little at a time

Under the picture is random static

The finished picture is the wrong place to start. Underneath it is something that never appears on screen: a field of random static, the visual version of an untuned television, that the system begins from. A text-to-image generator does not open a blank canvas and lay down strokes, and it does not rummage through a stored collection for the closest match. It starts from an image that is pure noise and takes a little of that noise away, over and over, until a coherent picture is what remains.¹

That one sentence hides every interesting part. What is the noise, exactly, and what does it mean to take it away? Why would removing noise from static ever leave a face rather than more static? And how does a string of words settle which face? Each answer rests on the one before it, so they come in order, beginning with the only thing the network was ever trained to do.

The network is trained to predict the noise

The reverse process only makes sense once the training does, so the training comes first. A diffusion model is built around a single network, and that network learns exactly one task.

Start with a real photograph. Add a small, measured amount of random noise, a fine grain scattered evenly across it, so the image is slightly corrupted but still plainly itself. Add another small amount, and another. Repeat this enough times and the photograph dissolves into pure static, every trace of the original gone. This gradual slide from a clean image to noise is the diffusion the method takes its name from. In the paper that established the approach, it runs over a thousand small steps, each adding a sliver more noise than the last.¹

The network is trained on that slide, run backward. Take one of those partly corrupted images, together with the record of how much noise it holds, and ask the network for a single thing: the noise that was added.¹ Because the original photograph is known, the exact noise is known too, so each guess can be scored against the truth and the network nudged until its guesses land close. Run this across an enormous set of images, each paired with a caption, at every level of corruption from a faint haze to near-total static, and the network gets good at one narrow skill.¹

The network is never shown a clean image to copy, and it is never asked to compose or to draw. It only ever looks at a noisy image and estimates the noise in it. That single ability is the ground everything the system does later is built on.

Generation runs the denoiser backward from static

Generation is that trained skill, run in the only direction it has. Begin with a fresh field of pure random static, noise and nothing else. Ask the network to estimate the noise in it, and subtract a small fraction of that estimate. What is left is an image very slightly less noisy than the static it came from. Feed that back in, ask again, subtract again. Each pass removes a little more noise, and after enough passes a clear image sits where the static was.¹

There is no canvas holding a drawing in progress, and no list of objects being placed. At every moment the thing in hand is just an image, noisy at first and cleaner each pass, and the picture is whatever remains once the noise is gone.¹ Coarse shapes settle first and fine detail arrives late, because the early passes have so much noise to cut through that only large structure survives.¹

How many passes? Two different numbers get quoted, and they answer different questions. The original method used a thousand steps, one for each level of noise it was trained on.¹ A thousand runs of a large network per image is slow, so deployed generators use faster sampling methods, worked out after the original, that reach a comparable picture in far fewer steps. A common default is around fifty.² The thousand is the training-time design. The fifty is what a product actually runs. They are not one number measured two ways.

One more shortcut explains a lot about how the fast systems are built. Running a large network over every pixel of a high-resolution image, fifty times over, is expensive. So one influential design does the denoising not on the pixels but on a compressed code of the image, a smaller grid of numbers that a second network can expand back into full pixels in a single pass at the end. That compressed grid is called a latent space, and denoising there instead of in pixels is what lets Stable Diffusion run on an ordinary graphics card.³ Not every system works this way. Google's Imagen and OpenAI's DALL-E 2 denoise in actual pixels, generating a small image and then running further diffusion steps to enlarge it in stages.⁴⁵ The subtract-the-noise loop is identical in all of them. Only the space it runs in differs.

The prompt steers each denoising step

Nothing so far involves the words. The loop as described would wander from static to some plausible image with no regard for what was asked. The prompt has to enter, and it does so through a second network whose only job is to turn text into numbers.

A separate model, a text encoder, reads the prompt and turns it into a block of numbers that stand for its meaning. Which encoder a system uses varies. The released version of Stable Diffusion uses CLIP,² a model trained to line up images and their captions in a shared space of meaning, on four hundred million image-and-text pairs pulled from the web.⁶ Imagen uses a large language model, T5, frozen and trained on text alone, and its builders found that enlarging that text model improved the pictures more than enlarging the image network did.⁴ The original latent-diffusion work used yet another text encoder.³ The shape is always the same. A separate model encodes the prompt, and the denoiser reads that encoding.

The encoding is fed into the denoiser on every pass. Each time the network estimates the noise, it does so while reading the prompt's numbers, so the noise it chooses to remove is the noise standing between the current image and one that matches the words.³ This steering is not a final filter. It rides every one of the fifty passes, tilting each step toward the described picture.

How hard the prompt is enforced is itself a dial. Trained the right way, one network can run two ways on the same step: once reading the prompt and once ignoring it. The difference between those two estimates points from a generic image toward one that fits the words, and the system can walk further along that direction than the prompt alone would carry it, pushing the picture harder toward the text.⁷ Turn the dial up and images cleave more tightly to the prompt but grow more alike. Turn it down and they vary more freely but drift from what was asked.⁷ Users usually meet this as a guidance slider. One caution on the numbers: the research paper measures the strength on a scale where zero means no steering, while a common product slider is offset by one and defaults to around seven and a half.²⁸ The two scales describe the same control, and their numbers are not interchangeable.

What the noise-predictor never learned

Return to the one skill the network has. It estimates noise. It never learned letters, or the count of fingers on a hand, or the rule that a red cube on a blue cube is a different scene from a blue cube on a red one. Where those things come out right, they come out right because predicting noise well, across enough captioned images, happens to reproduce them. Where they break, they break for the same reason: nothing in the system holds them directly.

Two failures are familiar to anyone who has pushed these tools. Text inside a generated image often comes out as convincing gibberish, letter shapes in the right style that spell nothing. And attributes slip between objects: ask for a red ball beside a blue box and the colors may trade places.⁵ These are not rare edge cases. They are steady enough that the people who build the systems record them as known limitations.⁵

This is where the honest account stops. The builders can name the failures precisely, but why they happen is not settled. The team behind DALL-E 2 proposed an explanation, tracing both the swapped attributes and the garbled text to properties of the CLIP encoding the model reads from, and they were careful to present it as a hypothesis rather than a demonstrated cause.⁵ The denoising loop, the noise-prediction training, and the text conditioning are settled engineering, understood well enough to rebuild from scratch. The reasons for these particular failures are not. That line, between the parts that are worked out and the parts that are still guesses even for the builders, is the one to keep in view.