Image generators put a prompt's colors on the wrong objects

The bench comes out yellow

A text-to-image model takes a line of text and returns a picture meant to match it. Give it one object with one attribute, a red apple or a wooden chair, and it usually obliges. Give it two objects that each carry their own attributes, and it starts to slip.

When researchers testing Stable Diffusion asked it for a yellow bow and a brown bench, it drew the bow yellow and then painted the bench yellow too, dropping the brown the prompt had assigned to it.¹ Both objects arrived. The color meant for one of them landed on both.

One prompt is an anecdote. The same slip shows up wherever binding is measured at scale. On T2I-CompBench, a benchmark of thousands of compositional prompts, Stable Diffusion v1.4 scores 0.38 out of 1 at putting the named color on the named object, and only 0.12 at getting spatial relations like “on top of” right.² A separate study generated 1,350 DALL-E 2 images from basic relation prompts and asked 169 people to judge them; about 22% matched the relation requested.³

The colors and the objects are all there. Only the pairing between them is wrong. To see why, follow the prompt through the two steps that stand between the words and the picture.

The path from words to image

The picture is not drawn from the prompt directly. First the words are handed to a text encoder, a network that reads the prompt and turns it into a list of vectors, one for each word or word-piece. Most image systems use a CLIP-style encoder, trained by matching whole captions to whole images across 400 million pairs.⁴ That training is the point to hold on to, and this course covers CLIP itself in its own lesson.

Those vectors are what the image model consults while it works. A generator like Stable Diffusion builds the picture by denoising: it starts from a field of noise and refines it step by step, the subject of an earlier lesson. At each step it reads the text vectors through cross-attention, the operation that lets one set of things attend to another. Here each region of the emerging image attends to the words and pulls in the ones that describe it.⁵ Attention has its own lesson; what matters for binding is that cross-attention is the only channel through which the prompt reaches the canvas.

Two places, then, can bend the pairing: the encoder that writes the vectors, and the cross-attention that reads them.

Where the binding comes loose

The tidy story blames the encoder. CLIP's text encoder reads left to right behind a causal mask, so each word's vector is blended with the words before it; by the time “bench” is encoded it carries traces of “yellow,” “bow,” and “brown” alike.⁶ And CLIP was never trained to keep an adjective attached to its noun. Its objective rewards a single pooled sense of the whole caption, matched against the whole image, with no signal that “yellow” modifies “bow” and not “bench.”⁴ On this account the encoder hands the image model a loose bundle of concepts, and the pairing is already lost in the text, before the image model runs at all.

That story is too tidy. If the encoder held all the blame, repairing the encoding would repair the picture. StructureDiffusion tries exactly that. It parses the prompt and feeds structured noun-phrase encodings into the image model, and the payoff is small, lifting two-object color accuracy from 19.2% to 22.7%.⁶ Attend-and-Excite leaves the encoder untouched and works only on the cross-attention, pushing each object's word to claim a region of the image, and it recovers far more.¹

The gap it closes points at the same place. When Stable Diffusion draws a two-object prompt, the whole prompt matches the image reasonably well, around 0.83 on a CLIP similarity score. The worse-served object matches at only 0.60 to 0.63: one object is rendered while the other is dropped or mis-colored. Steering the cross-attention raises that second object to 0.69 to 0.72.¹

This does not settle where the fault lives. Each method reports gains on its own prompts and metrics, and each favors the part it chose to touch, so the numbers cannot be compared directly.¹⁶ What they agree on is narrower and firmer: reach into the text side or the attention side and binding improves, which locates the fault in those two parts and not somewhere else.

The failure is not particular to one encoder either. A separate study of DALL-E 2, which uses a different text encoder, catalogs the same behavior: one attribute word modifying two objects, a single word playing two roles, a property leaking from one object to its neighbor.⁷

How far the explanation reaches

Two things are settled. The failure is systematic, not occasional: it is measured on Stable Diffusion by T2I-CompBench, on DALL-E 2 by the relation study, and reproduced by every fix paper that first had to document the problem it set out to solve.²³¹⁶ And the cause lives in the text conditioning and the cross-attention, the two parts that carry the words into the image, because that is where every working fix reaches in.¹⁶

Two things are open. The first is how the blame divides. The fixes disagree about how much of the problem sits in the encoding and how much in the attention, and their evidence is competing methods rather than a clean split.¹⁶ The second is whether any of this generalizes. The fixes move the numbers, but the ceiling stays low; even after correction, two-object color is right about 22.7% of the time.⁶ A 2025 review argues the shortfall is architectural, that continuous attention is ill-suited to the discrete job of keeping each attribute on its own object, and that incremental fixes will not close the gap.⁸

Under all of it sits one plain fact, and it is where the explanation reaches bottom: the model was never handed the sentence's structure. Nothing about which adjective governs which noun survives into the vectors the picture is drawn from, so nothing downstream can reliably put it back. That is the step below which no further part of the system would change the answer.