Self-attention can't tell "dog bites man" from "man bites dog"

Attention sees a bag of words

Every layer of a transformer is built on one operation, self-attention, and an earlier lesson worked through its arithmetic. For each word, the operation returns a weighted average of all the other words, and it sets the weights itself by measuring how well each word matches the one doing the asking.

The measuring is where order goes missing. To weigh how much "dog" matters to "bit", the operation compares those two words' vectors and reads off a single number. The comparison looks at the two vectors and nothing else. It has no term for the fact that "dog" came right before "bit", or three words after it. So the blend the operation builds for "bit" is assembled from the same parts, with the same weights, wherever in the sentence the other words happen to sit.

This is not a quirk of one model. A transformer is, by construction, invariant to the order of its input: permute the words and the math hands back the same answers, permuted the same way.¹ The original transformer paper stated the consequence plainly. With no recurrence and no convolution to track sequence, the model has to be handed information about position from outside, or it cannot use order at all.²

One caveat. That blindness is exact for the bare operation. The models you actually use are decoders that read left to right: when one processes a word, the causal mask hides every word that comes after it, so each word sees only the ones before. The masking leaks a little order for free. A word can, in effect, count how many words precede it, which is a rough stand-in for where it sits, and a model given nothing else can still recover some of that.³ But the count is a weak signal, and the operation underneath is still order-blind, so every production model supplies position on purpose.

Position gets added to the vectors

The first fix is the one still drawn in most diagrams. Before the words reach attention, the model adds a second vector to each one that says where it sits. The original paper used a fixed pattern of sine waves: each position gets its own set of values, built from waves of different wavelengths, summed straight into the word's vector.² Nothing is learned. The pattern is identical for every model trained this way. Now "dog" in slot two and "dog" in slot four reach attention as different vectors, and the operation that could not see order has order baked into what it reads.

The next models learned the pattern instead of fixing it. GPT-1 kept the idea of one position vector added per slot, but let training choose the numbers rather than fix them by formula.⁴ BERT did the same on the other side of the architecture, summing a learned position vector into every input token alongside the word itself.⁵ Learning the vectors buys a little and costs something concrete: a learned table has a fixed number of rows. BERT's stops at 512, so a 513th word has no position vector at all, and the model simply cannot take a longer input.⁵

From a word's slot to the gap between words

Absolute position, the fact that a word sits in slot five, is not obviously the thing a model needs. What usually matters is the gap between two words. "Not" three words before "good" does specific work, wherever in the sentence the phrase falls. Shaw and colleagues rebuilt the encoding around that gap. Instead of stamping each word with its slot at the input, they added a learned representation of the distance between two words directly into the attention comparison. Now the operation weighs "dog" against "bit" knowing the two sit one apart.⁶ On translation this beat absolute position by 1.3 BLEU points on English-to-German, a small but real gain, and adding absolute position back on top of it bought nothing further.⁶

The scheme running inside most models today is a later relative idea: rotary position embedding, or RoPE. Rather than add a position vector, RoPE rotates each word's query and key, the two projections attention compares, by an angle set by the word's position. When the operation later compares two words, their rotations combine, and the comparison comes out depending only on the difference between their positions.⁷ The gap falls out of the geometry. LLaMA, the open model family a great deal of current work is built on, dropped absolute embeddings and put RoPE at every layer, which is how a 2021 paper became a default.⁸

Past the training length, the schemes diverge

A relative scheme sounds like it should handle any length. If the model only cares about the gap between two words, why should a longer sentence trouble it? It does. Models using RoPE fail to work past the sequence length they were trained on: feed one a passage longer than anything in training and its output degrades.⁹ Whole methods exist only to stretch a trained RoPE model to longer inputs after the fact. Position interpolation, NTK scaling, and YaRN are all patches of this kind, and YaRN reports doing the stretch with roughly ten times fewer tokens than earlier fixes.⁹ If RoPE handled length on its own, none of them would need to exist.

You can watch the failure. ALiBi adds no position vectors at all. Instead it subtracts a penalty from each attention comparison in proportion to how far apart the two words sit, and it was built to extrapolate.¹⁰ Its authors trained models at one length and tested them well past it.

A model trained on 1024 tokens keeps its footing out past 2048, while the sinusoidal and rotary curves shoot upward the moment the input passes the length they trained on. It does this while training 11% faster.¹⁰ ALiBi buys that by construction. Its penalty grows with distance, so far-apart words are pushed toward irrelevance, which is why length stops hurting and also why the model reads within a narrower window than the raw input length suggests.

So which encoding is best? No one scheme has won, and the winner depends on the test. Relative beat absolute on translation.⁶ ALiBi beat sinusoidal and rotary on length.¹⁰ And on a set of small reasoning tasks, a decoder given no positional encoding at all, leaning entirely on the order the causal mask leaks, generalized to longer inputs better than absolute, rotary, and ALiBi, at no extra cost.¹¹ That last result comes from small models on algorithmic problems, not the large general-purpose models most people use, and none of the sources here names a single scheme as settled best for those.¹ This is the floor of the descent, the step below which nothing changes the answer: the people building the systems are still guessing here too.