Rich Sutton's essay has four examples and zero citations
Each of Sutton's four examples for search beating human knowledge carries more hand-built structure than his 1,100-word essay says, from Deep Blue's tuned evaluation function to the human-labeled game data inside the AlphaGo run he actually cites.
Language models draw no line between instructions and data
System text, user text, and anything pasted or retrieved land in one stream of tokens, and only training, not architecture, decides which span the model treats as a command.
Google's Gemini illustrated 1943 German soldiers with Black and Asian faces
A rule meant to diversify Gemini's people pictures fired on every prompt, even ones that history had already answered, and Google pulled the feature three weeks after launch.
The system human judges ranked best at the 2005 NIST evaluation scored 6th on BLEU
BLEU scores a translation by counting matching word sequences against a human reference rather than reading what it means, and a 1-2 point difference between two systems' scores is a coin flip on whether people actually judge one better.
A shutdown button only works on an AI unsure of its own goal
A 2016 proof shows a system defers to a human only when it doubts its own goal, and Palisade's 2025 tests found some models resisting shutdown up to 97% of the time even under the instruction that told them most explicitly to comply.