GPT-4 names Tom Cruise's mother but not Mary Lee Pfeiffer's son

Mostly right forward, mostly wrong in reverse

Ask GPT-4 who Tom Cruise's mother is and it answers Mary Lee Pfeiffer, correctly, most of the time. Ask the same model who Mary Lee Pfeiffer's son is, the identical fact from the other end, and it fails more often than not.¹ This is not a one-off. Berglund and colleagues ran the same test, both directions, on roughly 1,500 real parent-child pairs pulled from public records.¹

The celebrity test leaves an opening: maybe GPT-4 simply knows famous parents better than famous children, for reasons unrelated to training. Berglund's team closed that opening with a synthetic version: invented people, sentences of the exact form "[Made-up name] is [description]," and a model finetuned on those sentences alone. Asked forward, it answered correctly. Asked the same fact backward, "who fits [description]?", it scored no better than a random guess.¹ Fame and ambiguity are not doing the work here. A fact trained in one direction is not available from the other. Researchers call this the reversal curse.

Both experiments run on the same underlying mechanism: next-token prediction, the training method the earlier lesson on autoregressive generation explains. What follows works out exactly what one step of that training does to a sentence like "Tom Cruise's mother is Mary Lee Pfeiffer" that leaves the forward direction intact and never touches the reverse.

What one training update on "A is B" changes

Training that sentence into a model does not file it anywhere a lookup could pull from either end. It runs through ordinary next-token training: the model reads the tokens of "Tom Cruise's mother is" in order, and at the position right after "is," it produces a probability over every possible next token. The actual next token, "Mary," gets compared against that guess, and one gradient step shifts every weight a small amount toward the guess that turns out right. The same happens for "Lee," then for "Pfeiffer." This is ordinary, measured engineering: what a training step does to every sentence in a training set.

The literal claim, stated plainly and settled: the loss for this sentence is computed in one direction only. Nothing in this training step asks the model to predict "Tom Cruise" from "Mary Lee Pfeiffer's son is," because that string of tokens was never in front of it. Unless the training data separately contains a sentence written in that order, no gradient nudges the network toward answering it. That is the mechanical fact the reversal curse rests on. A weight update that makes the tokens of A predict the tokens of B does not, by that same update, make the tokens of B predict the tokens of A. A second, separate update would be needed, computed on a sentence written the other way, and ordinary training text mostly supplies no such sentence.

Zhu and colleagues confirmed this directly inside a trained network. Analyzing models after autoregressive training on directional text, they found the attention weight matrices connecting a description to a name come out asymmetric, structured to compute the forward direction and not its mirror.² That asymmetry is the measured shape training leaves behind, inspected in real networks rather than inferred from behavior alone.

A fact is a direction, not an address

Training builds a set of directions in a high-dimensional space, the representations the earlier lesson on word embeddings describes. A transformer's layers push those directions around so that, in the context "Tom Cruise's mother is," the arrangement makes "Mary Lee Pfeiffer" the likely next token. No second operation runs for free that arranges things so "Tom Cruise" follows "Mary Lee Pfeiffer's son is." That would take the same kind of gradient the first arrangement took, computed on the reverse direction, and the training data supplied none.

A database offers the wrong comparison here. It stores "mother: Mary Lee Pfeiffer" under a key, and a second index lets you query it by the value too, so either direction is a lookup. Nothing in a next-token predictor works this way.

Grosse and colleagues measured this asymmetry directly in a real trained model, using influence functions, a technique for tracing which training examples pushed a given output which way. They found that when the order of two key phrases in a training example is flipped, the measured influence of that example on related outputs drops to near zero.³ The information behind "who is Tom Cruise's mother" and the information behind "who is Mary Lee Pfeiffer's son" behave, in their data, like two separate facts. They are not two phrasings of one.

Allen-Zhu and Li put a number on how little room there is to store a fact at all: on the order of two bits of factual knowledge per parameter.⁴ Their separate analysis of extraction finds the same direction-dependence. What a model can pull back out of its weights depends on the specific shape of question it was trained to answer, not on whether the fact merely sits somewhere inside, waiting for any query that names it.⁵

Whichever order you predict is the only order you learn

Push one level deeper: why can't a single training step teach both directions at once? The constraint is in what an autoregressive model is allowed to see while training. At the token position for "Mary," the model attends only to the tokens before it: "Tom Cruise's mother is." The token "Tom" never sits in a position where it predicts anything about a sentence that starts with "Mary." The two directions are not two readings of one computation. They are two different computations, and training only ran one of them.

Kitouni and colleagues name this the factorization curse: an autoregressive model commits, at training time, to one order for predicting the tokens of a fact, and that order is the only one it learns to compute.⁶ Training on "Tom Cruise's mother is Mary Lee Pfeiffer" teaches one conditional: the mother's name, given the son's. It teaches nothing about the reverse conditional, because computing that would mean predicting different tokens from a different starting point, a computation the training sentence never asked for.

This is not a property of one architecture family alone. Wu, Yang, and Wang tested BERT, a model trained to fill in a blank using tokens on both sides rather than only the tokens before it. They found it does not carry the reversal curse the way autoregressive models do.⁷ That is independent confirmation, from a team with no stake in Berglund's paper: change what a model is trained to predict from, and the direction-dependence goes with it.

What closes the gap, and what does not

If the cause is which direction training predicts in, the fix should be training in both directions at once. Golovneva and colleagues tested exactly that: pretraining on a sentence together with a copy that has the token order reversed, so the model gets a gradient update in both directions for the same fact. They report the method can "mitigate the reversal curse, and even completely eliminate it in certain cases."⁸ Kitouni and colleagues tested that same reversed-token mechanism on their own tasks and report the opposite: reversing tokens "does not improve backward retrieval," and reversing named entities specifically "leads to undesirable behaviour."⁶ Neither paper cites the other. This is not one team's result sitting inside the other's broader scope. Two primary sources tested the same fix and got opposite answers, and that disagreement stands unresolved.

A different line of work changes the training objective itself rather than patching the data. Masked diffusion models do not commit to a left-to-right order at all. During training, random tokens are hidden and predicted from whatever tokens remain, in no fixed order, so both directions of a fact get a gradient at some point. Nie and colleagues found the advantage holds at scale: a masked diffusion model with 1.1 billion parameters kept a meaningfully higher reverse accuracy than autoregressive models many times its size, including a 175-billion-parameter GPT-3.⁹ This is exactly what the training-dynamics account above predicts: a method that refuses one predicting order produces less of the asymmetry. No production chat model in wide use trains this way, so the asymmetry in this lesson still applies to the model most readers use.

One question this literature can now answer with real measurements: whether making an autoregressive model bigger fixes this by itself. Lin and colleagues tested the reversal gap on open models ranging from 7 billion to 70 billion parameters, across the Llama, Vicuna, and Mistral families, and found it holding regardless of size or model family.¹⁰ That is a measured result, not a guess: scale alone has not closed the gap in anything tested so far. What is still open sits one layer beneath it. Exactly which computation, in which layer, distinguishes a model trained one way from one trained the other is not fully mapped for any production model. Zhu's asymmetric attention matrices are evidence toward an account, not the account itself. Whether some future training recipe could close the gap without paying the cost of full masked-diffusion training is untested. And whether a person who learns "Tom Cruise's mother is Mary Lee Pfeiffer" and effortlessly answers "who is her son" runs a computation categorically different from a weight update is a question this literature does not touch. That comparison stays open.