A chatbot can't keep a word it never wrote down

The answers that can't belong to one word

Someone asks a chatbot to pick a secret five-letter word, keep it hidden, and answer letter guesses, the way a person holds a word in hangman. The chatbot agrees and says it has one in mind. The guessing starts. The exchange below is reconstructed to show the shape of the failure, not a saved log, but the pattern is the one the measured studies report.

No single word carries all three answers and comes out as "table," because "table" has no O. The answers are not merely wrong. They are impossible together.

Research groups that run the game as a controlled test find the same thing. One study is built around hangman itself. It catalogs the two ways the game breaks: the model confirms the true answer and a wrong one alike, or it denies the word it then reveals. It keeps playing even when asked whether a specific letter is in the word it never chose.¹ Another group ran a plain yes-or-no version. A model asked to hold an imagined object contradicted itself as the questions piled up, once agreeing an object was both heavier than an elephant and lighter than a cat. Across 200 games, GPT-4o contradicted itself in 173 of them, and a smaller model, GPT-4o mini, in all 200.²

There is no word to hide

The natural explanation is that the model picked a word, is holding it in mind, and either got sloppy or decided to cheat. That belief is not naive. It is the belief the game runs on, and it describes exactly how a human opponent works: they choose a word, keep it, and check each guess against it.

The model does none of that, because it wrote nothing down. A plain chat keeps no private memory that survives from one turn to the next. The application rebuilds the conversation each time by resending the whole visible transcript as fresh input, the resend covered in an earlier lesson. The sharper point here is not that the model forgets what was said. It is that it cannot keep what was never said. Each reply is generated one token at a time, and every token is conditioned only on the tokens already in that input. The GPT-2 paper writes the model's estimate as a product of per-token probabilities, each one given the tokens before it.³ The chat interfaces say the same in their own words. Anthropic's Messages API "is stateless, which means that you always send the full conversational history to the API,"⁴ and OpenAI's guide states that "each text generation request is independent and stateless."⁵

So when the instruction is "pick a word, keep it secret, don't tell me," the model produces no tokens that name a word. Nothing about a chosen word gets written into the transcript, and the transcript is the whole of what the next turn can see. There is no secret, because there is nowhere for a secret to sit. A person told to pick a card and not show it still holds a card, and can turn it over later. A plain chat told to pick a word and say nothing holds nothing, and has nothing to turn over.

Performing a commitment, not holding one

This is the difference between holding a commitment and performing one. A person who holds a word answers from it. The model instead produces the answers of someone who is holding a word, generating each yes or no to fit the transcript so far, with no word behind any of them. It is not lying about a secret it holds. It has no secret to lie about.

A recent result makes the boundary exact. An agent that can see only the shared transcript cannot both leave the secret undetermined and stay consistent with it; with no word written down, there is nothing to be consistent with.¹ The fix follows straight from that. Write the word into the model's working memory, as a tagged note like <secret>planet</secret> that rides along in the context, and the word now exists somewhere later answers can be checked against. In the same study, models left to keep the word "in their head" answered consistently in roughly 2 to 26 percent of games, depending on the model and task. The same models, given an explicit place to write the word down, rose to between 56 and 100 percent.¹

Where a model can hold a word

The clean claim is narrow and worth stating exactly: a plain chat, asked to keep a word and tell no one, holds no commitment across turns. That leaves real exceptions, and they come down to a single question, whether the word gets written somewhere.

Within one reply, it can. A reasoning model produces a chain of thought before its answer: OpenAI's o1 "thinks before it answers" and "can produce a long chain of thought before responding to the user," and the person reading is shown only a summary of it.⁶ Any model can also be told to work on a scratchpad, writing its intermediate steps as ordinary generated tokens before the final answer.⁷ In both cases the commitment lives in tokens the model wrote, even when those tokens are hidden from the reader. What it cannot do is hold a commitment in nothing at all.

Those written-but-hidden tokens usually do not survive the turn. In a standard chat, a reasoning model's internal tokens are discarded between turns.¹ A word it settled on while composing one reply is gone by the next, unless the product keeps it. Whether it does is a product choice, not a fact about the model. Even the working state inside a single generation, the cached keys and values the model builds as it decodes, is scoped to that one request and is not carried to the next.⁸ What does cross turns is state the product writes down where it can be read. Memory features and server-side stores do exactly that. As one practitioner put it, such memory "is stored somewhere in plain English; you can just go and look at it."⁹

Two things, then, are settled engineering: a plain chat holds no commitment across turns, and any commitment that does survive is written down somewhere readable. What is a product choice is whether a given tool keeps that writing. A reasoning model that drops its thoughts keeps nothing; a memory feature that saves them, or a tool that files the word away, keeps it.

What you trust when you trust it to remember

The hangman game is a clean case of a habit that shows up wherever a chatbot is trusted to keep something it never wrote. A plan it "agreed" to but did not record. A number it "chose" and withheld. A running tally it says it is keeping. If the thing is not in the visible transcript or in a store that can be read back, it is not anywhere, and the next reply reconstructs it from scratch.

Newer models play the game more coherently, which can look like progress on this exact problem. It is not. A model generating locally consistent answers is a better imitation of a commitment, not a held one. As Murray Shanahan puts it, it is "absolutely inherent in the way large language models are built" that the model will not commit at the start of the conversation to what the object is.¹⁰ The practical rule is the one the fix already showed. To make a chatbot keep something, make it write it down, in the conversation or in a store where it can be read back and checked.