The diffusion paper behind today's image generators could not take a prompt

The paper that made diffusion generate real pictures

“Denoising Diffusion Probabilistic Models” is a 2020 paper by Jonathan Ho, Ajay Jain, and Pieter Abbeel at UC Berkeley, presented at the NeurIPS 2020 conference.¹ Its method is the one this course covers in the lesson on how an image generator works: a fixed forward process takes a real image and adds a little random noise to it, over and over, until the picture is pure static. A network then learns to run that chain backward, subtracting a little of the noise at each step until an image reappears.¹

What made the paper matter is a claim its authors kept deliberately small. Diffusion models were not new in 2020. What no one had shown was that they could produce sharp, believable images, and this paper showed exactly that.¹ Almost everything else people now attach to the phrase “diffusion model” came later, from other papers.

State of the art at 32 by 32

Start with what the paper put on the page. Its headline result is on CIFAR-10, a standard set of 32 by 32 color photographs of ten kinds of object.¹ On that set the model reached a Fréchet Inception Distance of 3.17, a standard score for how close a batch of generated images is to real ones, where lower is better. It was the best any method had reported at the time.¹ Both that figure and its companion Inception score of 9.46 are unconditional numbers: the model was given no label, no class, and no caption, and drew each image from noise alone.¹

To read 3.17, set it beside the field. The paper's own table puts the diffusion model against the strongest GAN of the day, the adversarial approach an earlier lesson covers on its own, and against the score-based model it improved on most directly.¹

On FID the diffusion model came first, 3.17 against the leading GAN's 3.26. On Inception score the GAN was still ahead, 9.74 against 9.46.¹ The paper led on one of the two standard measures and trailed by a hair on the other. The score-based model from a year earlier sat at an FID of 25.32, an order of magnitude worse, and it is the closest predecessor the paper beat outright.¹

That score came at a cost the paper is equally plain about. Producing one image means running the whole reverse chain, one network evaluation per step, and the paper set that chain to a thousand steps for every experiment.¹ Sampling a batch of 128 images took about 300 seconds on the authors' 2020 hardware, near 2.3 seconds an image, and the network doing the work held about 35.7 million weights.¹

The paper did reach past CIFAR-10, to 256 by 256 photographs of bedrooms, churches, cats, and faces, again unconditional.¹ There the results were more mixed: an FID of 4.90 on bedrooms and 7.89 on churches, but 19.75 on cats. The abstract sums the whole 256 by 256 effort up as “sample quality similar to ProgressiveGAN,” a GAN from three years before.¹ The ceiling was 256 pixels on a side, one class of image at a time.

Diffusion did not start here

The paper is often called the founding document of diffusion. It founded something narrower, and said so. The idea of building a generative model by slowly destroying the structure in data with a forward diffusion and then learning to reverse it was introduced five years earlier, in 2015, by Jascha Sohl-Dickstein and three coauthors.² The specific technical footing the 2020 paper builds on came in 2019 from Yang Song and Stefano Ermon. They estimated the direction toward more realistic data at many noise levels and sampled by following it. Their model reached an Inception score of 8.87 on CIFAR-10, the mark this paper passed.³ Ho, Jain, and Abbeel credit both openly.

So the contribution is exact. Not diffusion, but the first demonstration that diffusion generates high-quality images, together with the training recipe that got it there. That recipe is the part practitioners kept. Rather than have the network predict the cleaned-up image at each step, the paper rewrites the problem so the network predicts the noise that was added. It trains the network on a plain squared-error gap between the real noise and the predicted noise.¹ Predicting the noise, on that simple objective, is the move nearly every diffusion model since has inherited.¹ Calling the paper a founding document is fair for the modern wave of image generation. It is wrong for diffusion itself.

Where the prompt actually comes from

Now set the 2020 paper next to a modern text-to-image tool. There a typed sentence becomes a photoreal image at high resolution in a few seconds, on ordinary hardware. Every one of those parts, the sentence, the speed, the resolution, the hardware, comes from a paper published after this one.

The thousand-step wait was the first to go. Denoising Diffusion Implicit Models, from Jiaming Song, Chenlin Meng, and Stefano Ermon, appeared in October 2020, four months after the original paper. It keeps the trained network and changes only how the reverse chain is walked, producing comparable images 10 to 50 times faster.⁴ Fast sampling was not a later era. It arrived almost at once.

Running on ordinary hardware came from moving the work off the pixels. The 2020 model denoises the full image directly, which is what makes 256 pixels a strain. Latent Diffusion Models, from Robin Rombach and colleagues in 2021, run the same process in the compressed latent space of a pretrained autoencoder, a network trained to squeeze an image down to a small code and rebuild it. Working on that small code instead of the full picture cuts the computation enough to reach high resolution on modest hardware.⁵ That paper is the basis of Stable Diffusion, released to the public in August 2022 and able to run on a consumer graphics card with as little as 2.4 gigabytes of memory.⁶

The typed prompt came from the same Latent Diffusion paper, which added cross-attention layers, a mechanism that lets the denoising network read a text description while it works.⁵ Making the words actually bite took one more idea: classifier-free guidance, from Jonathan Ho and Tim Salimans in 2022. It trains a single model to run both with and without the caption, then pushes each sample toward the captioned version. The trade is some variety for images that follow the prompt more closely.⁷

Photoreal pictures from a caption were shown at scale by two 2022 systems. DALL-E 2, from OpenAI, first turns the caption into a compact numeric description of the image it should produce, then lets a diffusion model paint the picture from that description.⁸ Imagen, from Google, feeds the text through a large frozen language model and cascades diffusion models up to high resolution, reporting a COCO image-quality score of 7.27 on the same lower-is-better scale.⁹ These are the papers behind the products. The 2020 paper is behind these papers.