gpt-3.5-turbo-instruct plays legal chess; the chatbots don't

A confident move the board doesn't allow

Play chess against a chatbot by typing the moves, and a few turns in it often does something no chess player would: it names a move the position does not allow. It slides a bishop straight through one of its own pawns, or captures a piece that left the board ten moves ago, or castles while in check. It does this in the same fluent tone as a legal move.

That is the reputation, and it is only half true. Dynomight, who ran fifty games each against a range of models, reported that "no other LLM is remotely close" to one exception.¹ The exception is gpt-3.5-turbo-instruct, an older completion model from OpenAI that plays a full game at club strength and almost never offers an illegal move. One evaluation counted five or fewer illegal moves across 8,205 moves against the engine Stockfish.² A player rated 1900 at blitz lost a ten-game blitz match to it, nine to one.³ gpt-4, meanwhile, mostly loses by attempting an illegal move.²

So the behavior is real but not universal, which rules out the two explanations that come first. You might guess the model looked up the rules and slipped, or that it fails by guessing at random. Neither fits. There is no rulebook in the loop for it to consult, and a model guessing at random could not play a legal game at club strength. The real question is narrower: what produces the illegal move, and why does it nearly vanish for one model and not the others?

There is no board, only the moves as text

The model is not shown a chessboard. It receives the game as text in the notation players use to record games, PGN: 1.e4 e5 2.Nf3 Nc6 and on down the line, one move after another.⁴ Its whole task is the one it performs on any text, taught in autoregressive generation: predict what comes next, one token at a time. The next move is just the next stretch of that text.

The model has no separate board it sets the position up on and checks a move against. Nowhere in the process is a move tested for legality before it is written. Whatever the model knows about where the pieces are, it holds inside the same statistics it uses to pick the next token. A move is legal to it only in the sense that legal-looking continuations are the ones its training made likely, and its play is exactly as reliable as those statistics.

This much is settled engineering, and it answers the first guess directly: the model is not consulting the rules and failing, because the rules are written down nowhere. It reconstructs legal play, when it does, from the shape of millions of games.

Small models trained on transcripts build a board anyway

"No board" is easy to overread. It does not mean the model has no sense of where the pieces are. For small models built to study this, the opposite is true. Train a modest network on nothing but game transcripts, with no rules and no diagram, and it builds an internal picture of the board that a researcher can read out of its activations.

The clearest demonstration used Othello, a simpler game. Li and colleagues trained a network only to predict legal Othello moves, then probed its activations for the board. A straight linear read stalled above 20% error, while a two-layer probe recovered the board with 1.7% error, which they called an emergent nonlinear representation.⁵ Nanda and colleagues showed it was simpler than "nonlinear" suggested: encode each square as mine-or-yours relative to whoever is to move, and a linear probe reads the board with about 99.6% accuracy.⁶ The network plays both colors, so it tracks "one of my pieces," not "a black piece."⁷ Edit that board along the probe's directions and the model's next move changes to match, which shows the representation is used, not decorative.⁷

Chess is the case that matters here, and Adam Karvonen ran it. A 50-million-parameter network trained character by character on about 16 million online games plays legal chess more than 99.6% of the time and wins most games against a weak Stockfish.⁸ A linear probe reads 99.2% of squares correctly across 10,000 games, and every game is unique by move ten, so the model generalizes rather than replaying memorized lines.⁹ A later study recovered the board from seven networks without training a probe at all, answering the worry that the probe invents the structure it finds.¹⁰ For these small, purpose-built models, the internal board is settled.

The late-game collapse belongs to the weak models

A tidy story explains when the illegal moves appear. A game starts in well-trodden opening lines the training data covers densely, then wanders into rarer and rarer positions, and the guesses get worse as the examples thin out. For weak and chat-tuned models, that is roughly what happens. Adam Yedidia found that ChatGPT-3.5, without a careful prompt, could not make a legal move by about move fourteen; fed the running game score, it lasted to around move twenty or thirty before slipping.¹¹

The strong model refuses to follow the pattern. Mathieu Acher measured gpt-3.5-turbo-instruct's illegal-move rate across hundreds of games and found it roughly constant over the game's length rather than climbing as the game grew long.⁴ Dynomight watched the same model play legal moves deep into the late game, and in positions that never occurred in any recorded game in history.¹⁴ Errors compounding as the game leaves the opening is the failure mode of a weak model, not a law of how these systems play chess. When the statistics are good enough, novelty and length stop mattering much.

The models sort into a gradient, and one adversarial game set shows how wide it is.

These rates are not properties of the model alone. The same instruct model touched an illegal move in 16% of Acher's games but made five or fewer across 8,205 moves in Karvonen's evaluation.² The two harnesses prompt the model differently and allow different numbers of retries, and the human who lost to it allowed five before counting a forfeit.³ Legality is a property of prompt and model together, so a figure has to carry the setup that produced it.

Why one model sits so far ahead is not settled. Dynomight lists several candidates and rules none out: the chat interface and its instruction tuning degrading play, how much chess sat in each model's training data, and OpenAI's own data choices.¹ One widely repeated explanation, not confirmed against any primary OpenAI document here, is that only games between players rated 1800 or higher were kept for pretraining.¹² Prompting closes much of the gap: with three example games and a trick that makes the model rewrite the full move list first, Dynomight lifted gpt-4o to about 1540 Elo, still short of the instruct model's roughly 1750.¹⁴ The measurements are solid. The cause of the gap is an open question.

The engine can only make legal moves

A chess engine like Stockfish, or AlphaZero, is built the opposite way: it is given the rules and an explicit representation of the position, and it chooses a move by searching among the moves the rules allow.¹³ An illegal move is not something it usually avoids. It cannot express one, because the only moves it ever considers are legal. AlphaZero reached world-class play "given no domain knowledge except the game rules," and those rules were part of its world, not something inferred from text.¹³

The chatbot was given neither the board nor the rules, only a stream of move text, and it plays legal chess only to the extent that predicting that text well produces legal moves. For one model the statistics are good enough that legality comes almost for free. For the others they are not, and a bishop slides through a pawn.