Stable Diffusion does its denoising on a compressed grid, not the full image

Every denoising step ran over every pixel

“High-Resolution Image Synthesis with Latent Diffusion Models” is a paper by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, from the CompVis group at LMU Munich. It was posted to arXiv in December 2021 and presented at the CVPR 2022 conference.¹ It starts from a method this course has already covered: diffusion, which builds a picture by starting from pure noise and removing a little at a time, over many steps, until an image is left.

The trouble the paper names is what that costs. Each denoising step runs a full pass of a neural network, and the diffusion models before this one ran every pass over every pixel. A 512-pixel-square color image is already more than 780,000 numbers, three color values at each of its 262,144 pixels, and one image takes many steps. The authors put the bill plainly: training the strongest such models “often takes hundreds of GPU days.”¹ Raising the resolution makes every step more expensive, which is what kept high-resolution diffusion on large clusters.

An autoencoder shrinks the image to a grid

The paper's move is to stop running diffusion on the picture at all. It splits image generation into two stages. The first is a separate network called an autoencoder. Its encoder learns to write an image down as a much smaller grid of numbers, and its decoder learns to rebuild the full image from that grid.¹ The two halves are trained together so the round trip loses as little as the eye will notice. The result is a compact stand-in for the image, its latent, small enough to work on directly.

How much smaller is set by one number the paper calls the downsampling factor, f, the factor by which each side of the image shrinks.¹ At f = 8, each side of the grid is one eighth the side of the image, so the grid has sixty-four times fewer positions to process than the picture does. And the autoencoder is trained once and then reused across many different diffusion models, so the cost of learning to compress is paid a single time.¹

The factor cannot be pushed without limit. The authors test f from 1 up to 32 and find the useful range in the middle, at factors of roughly four through sixteen.¹ At f = 1 there is no compression at all, which is ordinary pixel-space diffusion. Past the useful range, too much compression throws away detail the decoder cannot put back, and image quality stalls after only a few training steps.¹ The reason to move off the pixels shows up directly in the scores. The paper measures quality with Fréchet Inception Distance, a standard number for how close a batch of generated images is to real ones, where lower is better.¹ Hold the diffusion network's budget fixed, and the pixel-space model at f = 1 trails the latent model at f = 8 by about thirty-eight points of that distance after two million training steps.¹

The compression is not free, and the authors say so. Rebuilding an image from its latent can become a bottleneck where pixel-level accuracy matters, and generating an image step by step is still slower than the single pass a GAN takes.¹ Moving into the latent space is a favorable trade, not a free win.

How the prompt reaches the picture

Generation then happens entirely on the latent. The model starts from a grid of random noise the size of a latent, runs the familiar denoising steps to turn it into the latent of a plausible image, and only at the very end passes that single latent through the decoder to produce pixels.¹ The expensive inner loop never touches a full-resolution image.

A bare version of this draws a picture from nothing in particular. To make it follow a typed prompt, the paper adds cross-attention layers inside the denoising network. These use the same attention mechanism an earlier lesson covers, here letting each denoising step read an encoded form of the prompt while it works.¹ The words have to become numbers first. Stable Diffusion does that with CLIP, a text encoder trained beforehand that turns a sentence into a list of vectors.² Cross-attention is the channel through which those numbers steer which image the noise becomes.

The model people ran was a bigger, separate training

That is the method. The paper also built models with it, and their scale is worth seeing plainly. The text-to-image model the paper reports is a 1.45-billion-parameter latent diffusion model trained on LAION-400M, a public set of 400 million captioned images.¹ Across standard benchmarks its scores are competitive with the best of the day.

Those are the paper's own results. The model that made the method famous was a different, larger build. In August 2022, CompVis, Stability AI, and their collaborators released Stable Diffusion: weights, code, and a model card, published openly on Hugging Face under a license that permitted commercial use.³ The model card records a separate training run. Stable Diffusion version 1.4 used the same downsampling factor of 8 and the CLIP text encoder above. It was trained on 256 A100 GPUs, over 225,000 steps at 512-pixel resolution, on subsets of the far larger LAION-5B collection, filtered for aesthetics and against watermarks.² For the 512-pixel images it trained on, the latent the diffusion network actually worked on was a 64-by-64 grid.²

This is the distinction the paper-versus-product confusion erases. The academic result and the public product share an architecture and one of their authors, but they are separate trainings, on different data, at different scale. The paper's efficiency claim is real and specific. For a given level of quality, working in the latent space costs far less than working in pixels, and the compressor is trained only once.¹ It is not a claim that image generation became cheap in absolute terms. The product that used the method still took 256 of the most capable GPUs of its day to train.²

What the open weights set loose

Releasing the weights openly is what set Stable Diffusion apart from the image models before it, which ran behind company services. Anyone could download the model, run it, inspect it, and retrain it.³ That openness also meant the training data was now the public's to examine, and two findings followed.

First, the data. In December 2023, the Stanford Internet Observatory reported that LAION-5B, the collection Stable Diffusion was trained from, contained known child sexual abuse material. Using established detection methods, the authors identified 3,226 entries of suspected such material, much of it confirmed by third parties.⁴ The report named Stable Diffusion 1.5 specifically and recommended that models based on it without added safety measures be withdrawn; LAION took the dataset down.⁴ This finding attaches to the product's training set, the billions-strong LAION-5B, not to the 400-million-image subset the CVPR paper's own experiments used.

Second, copyright. Getty Images sued Stability AI in the United States in 2023, alleging that more than 12 million of its photographs were copied through the LAION data used to train the model.⁵ That is an allegation by one party, and the first court to reach a judgment did not accept its central theory. In November 2025 the England and Wales High Court rejected Getty's secondary-infringement claim, finding, as reported, that the model weights do not themselves store the training images, after Getty had already dropped its primary copyright claim at trial.⁶

The paper's own claims hold up. The method works, and the efficiency it reported is real and has lasted. Its two-stage design is now the default for image generation, and Stability's own follow-up, SDXL, released the next year, is built on the same move with a larger network.⁷ The trouble that trails Stable Diffusion, the abuse material found in its dataset and the copyright fights over it, comes from the data the product was trained on, not from the paper's way of making pictures.