The Evidence
Bahdanau's 2014 translator chose which source words to read for each word it wrote
The model now cited as the origin of attention ran on WMT'14 English-French alone, and its authors credited a 2013 handwriting network for the idea.
ELMo set records on six language tasks without replacing the models it improved
The Allen Institute made a word's vector depend on its whole sentence, with a method BERT discarded within a year.
LSTM learned to carry a value across 1,000 time steps, on a network of a few thousand weights
The speech recognition and translation the name now carries came years later, on a three-gate cell Hochreiter and Schmidhuber's paper never described.
Backpropagation's famous paper showed a network building features no one designed
Its main demonstration was a kinship puzzle of twenty-four people, and the backward calculus it is remembered for inventing had been in print since the early 1970s.
Flamingo beat fine-tuned models on six benchmarks with 32 examples
DeepMind bridged a frozen vision model and a frozen language model with two small trained parts, then let a few examples in the prompt do the rest.
AlphaStar reached Grandmaster while capped at 22 actions every five seconds
The rank came from anonymous games on Blizzard's European ladder, and the January show match everyone remembers ran a looser, less constrained agent.
The 2016 WaveNet paper never reports its own generation speed
Van den Oord and DeepMind's 2016 model scored 4.21 out of 5 in its own listening test, against a natural recording that scored 4.55 in the same test.
GPT-1 pre-trained a single model on unlabeled books and beat purpose-built systems on nine of twelve datasets
A 2018 OpenAI preprint made generative pre-training the default way to build a language model, and paired it with the per-task fine-tuning its own descendants would drop.
FlashAttention does more arithmetic to move less data
Dao and coauthors' 2022 paper tiles the score matrix through a GPU's on-chip memory, cutting one training configuration's HBM traffic from 40.3GB to 4.4GB and its runtime from 41.7 to 7.3 milliseconds.
Turing predicted 70 percent by 2000. GPT-4.5 scored 73 in 2025.
The study behind that headline kept Turing's five-minute clock, and swapped his average interrogator for paid volunteers and his generic machine for a coached persona.
The 2009 ImageNet paper never measured the accuracy it is famous for
That paper is a database: 3.2 million web images hung on a hierarchy borrowed from a dictionary of English, their labels checked by crowd vote, finished three years before the record fell.
Mamba dropped attention to make long sequences cheap. Its authors put some back
Mamba matches a transformer twice its size on short-context reasoning up to 2.8B parameters, and loses to a far smaller one at looking things up in a long context.
Kingma and Welling made random sampling differentiable
A generative model first trained on handwritten digits, whose real legacy became the compressed latent space that Stable Diffusion runs inside.
Stable Diffusion does its denoising on a compressed grid, not the full image
Robin Rombach's CompVis team published the method behind Stable Diffusion in 2022, the design its successors were built on.
PaLM was trained on a fourteenth of the data its size called for
A scaling result DeepMind posted a week before PaLM says Google built its biggest language model far too large for the 780 billion tokens it saw.
ReAct searched Wikipedia between thoughts and still trailed chain-of-thought on HotpotQA
Yao's team prompted PaLM-540B with one or two examples, and reading a real Wikipedia line before each thought erased the invented facts that were chain-of-thought's most common failure mode.
AlphaZero beat Stockfish twice, under different rules each time
Everyone quotes the one-minute-per-move match from the 2017 preprint, not the peer-reviewed 1000-game rematch that gave Stockfish tournament time and still ended 155 to 6.
T5 recast every language task as text in, text out, and lost at translation
Google's controlled bake-off, run on 750 gigabytes of web text the team cleaned itself, crowned the encoder-decoder architecture that large language models would soon move past.
The biggest models now train with dropout turned down or off
Srivastava and Hinton's 2014 paper cut CIFAR-10 error to 12.61% and led the small benchmarks of its day, then showed in its own experiments why models trained on far more data need it less.
Meta's own Llama report can't certify four of its top scores
Meta scored its 405B model against GPT-4 and Claude itself, and the report's own contamination check leaves MMLU, HumanEval, and two more headline benchmarks blank.
A network's error peaks where it can just fit its training data, then falls again
Feed a network 4.5 times the training data and it can get worse, a limit one of the two 2019 papers behind double descent measured directly.
Google's 2017 layer split a model's size from what it computes
Noam Shazeer's team sent each token to a few expert sub-networks out of thousands, the design behind why Mixtral today lists 47 billion parameters but runs only 13 billion per token.
Anthropic taught an AI judge sixteen rules and trained a less harmful chatbot
Bai et al.'s 2022 paper measured that edge as a preference margin from 8,135 crowdworker harmlessness comparisons on Anthropic's own 52B models, the size behind a term now used far more loosely.
Segment Anything returns a mask and never a name
Kirillov and his Meta coauthors let the 2023 model draw 99.1% of its own billion-mask training set, then measured it losing to a task-specific model on the standard segmentation score.
August 2026One gradient step still fools image classifiers, a decade on
Goodfellow, Shlens, and Szegedy's 2015 paper blamed the panda-to-gibbon trick on models being too linear, a claim later work disputes, and proposed a defense that had to be rebuilt before it worked.
The diffusion paper behind today's image generators could not take a prompt
Ho, Jain, and Abbeel's 2020 paper reached a state-of-the-art score on the 32 by 32 CIFAR-10 set by learning to reverse a thousand steps of added noise.
Tested on Atari and robots in 2017, PPO now tunes billion-parameter models
Its one durable idea is a clip that keeps a policy update from overshooting, and the trust that the method aligns language models was built entirely by later papers.
Google packed a whole sentence into a single vector
The 2014 LSTM that gave machine translation its encoder-decoder shape leaned on a source-reversing hack that attention made unnecessary within months.
LoRA matched full fine-tuning by training 4.7M of GPT-3's 175 billion weights
The 2021 Microsoft paper that made cheap fine-tuning routine reported parity only on the constrained language benchmarks it ran.
The Vision Transformer lost to a plain CNN until it was fed 300 million images
Dosovitskiy and his coauthors pinned the reversal to Google's private 303-million-image set, and a qualifier in their own introduction named the missing ingredient a later model supplied on ImageNet alone.
Prompt engineering lifted CLIP's zero-shot ImageNet score by nearly five points
Radford, Kim and their OpenAI coauthors trained the 2021 model on 400 million captions they never released, and its label-free ImageNet result stayed strong on unfamiliar photos while collapsing on counting and satellite tiles.
Today's RAG pipelines fine-tune neither piece the paper trained together
Lewis et al. froze DPR's document encoder and fine-tuned its query encoder alongside BART. The RAG that ships today wires two frozen off-the-shelf pieces.
Adam's convergence proof was wrong, and the field kept using Adam
The 2018 counterexample that broke the proof drives Adam to the worst point of a one-dimensional convex problem, at hyperparameters no practitioner ever chooses.
Whisper transcribes audio it never trained on, and silence no one spoke
680,000 hours of loosely labeled web audio bought robustness on speech that clean-benchmark models fumble, and the decoder that delivered it now writes fluent text into medical transcripts.
DPO turns preference tuning into a single classification loss
Its simplicity won open models over from the RLHF pipeline, though the evidence that it matches PPO was small-scale and later work still puts PPO ahead on the hardest tasks.
The report that coined ‘foundation models’ ran no experiment
A 114-author, 214-page report chose the name to signal architectural stability and safety, then admitted, in the same section, that it could not say whether the foundation was trustworthy.
The knowledge distillation paper trained a network to recognize a digit it never saw
Hinton, Vinyals, and Dean's 2015 paper trained the small network on how the big one ranked every wrong answer, and that soft-target method is the part much of today's 'distillation' leaves out.
phi-1 matched code models 12x its size, but mostly on problems like its training set
Microsoft built the 1.3-billion-parameter model from about seven billion tokens of code that GPT-4 filtered and GPT-3.5 wrote, and it reached 50.6% on the HumanEval coding test.
The winning ticket turns up only after you train the whole network
Frankle and Carbin won a 2019 best-paper award for showing the subnetwork exists on small vision nets, and that it matches the full network only from the exact random weights it started with.
A small network memorized modular arithmetic, then generalized a million steps later
OpenAI named the effect grokking in 2022, on a 400,000-parameter network learning arithmetic tables over 97 symbols.
GPT-2 set records at predicting text and barely beat random at summarizing
The report people cite for calling the model too dangerous to release never mentions danger, misuse, or release.
The emergence jump returns when you plot AI skills against loss
A 2024 paper re-plotted the same exams against pre-training loss instead of model size, and found four hard tasks that sit at chance until loss falls past 2.2, even under the continuous metrics that were supposed to dissolve the jump.
The batch normalization paper got its own explanation wrong
Batch normalization trains deep networks in a fraction of the steps, yet a 2018 experiment found the internal-covariate-shift reason its own paper gave is not what makes it work.
king - man + woman = queen scored 65.6% on word2vec's own test
Tomas Mikolov and colleagues at Google reported the figure on a 19,544-question benchmark in 2013, and later analyses found the vector offset leans on excluding the very words the analogy starts from.
The deeper network fit its training data worse
Microsoft's ResNet swept the 2015 vision contests by adding each layer's input back to its output, the skip connection now wired into every transformer.
The GAN paper credited with deepfakes generated blurry thumbnails
Ian Goodfellow's 2014 paper invented adversarial training, then judged its blurry samples with a metric its own authors distrusted.
DQN reached ‘human-level’ Atari on 29 of the 49 games
The human in ‘human-level’ was one professional games tester playing about twenty short sessions, and the paper’s lasting contribution was two tricks that first made Q-learning stable on a neural network.
AlexNet won ImageNet 2012 on two gaming GPUs
Krizhevsky, Sutskever, and Hinton beat the runner-up by roughly ten points in 2012, a margin that pushed computer vision toward depth, data, and compute for good.
AlphaFold predicts a protein's shape, not what it does
DeepMind's 2021 paper measured how closely predicted protein shapes match the ones labs solve, and beat every rival on a blind test by a margin worth understanding before the Nobel-season shorthand runs past it.
AlphaGo's Nature paper never mentions Lee Sedol
Its one human match is a 5-0 win over Fan Hui, Europe's champion and a professional 2 dan, from a network trained first on 30 million human Go positions.
GPT-4's technical report declares 'no further details'
The 100-page report scores GPT-4 on 34 exams and states, in Section 2, that it gives no further details on how the model was built.
July 2026Rich Sutton's essay has four examples and zero citations
Each of Sutton's four examples for search beating human knowledge carries more hand-built structure than his 1,100-word essay says, from Deep Blue's tuned evaluation function to the human-labeled game data inside the AlphaGo run he actually cites.
DeepSeek trained a model to reason without ever showing it a worked example
DeepSeek's January 2025 paper proved reinforcement learning alone can teach a model to reason at length, and the $5.6 million price tag attached to that result belongs to a different model.
Two more papers turned a robot's backflip trick into InstructGPT's training recipe
Christiano et al. 2017 spent tens of thousands of human comparisons on Atari games and simulated robots, not one sentence of text, and getting that recipe to work on a language model took five more years.
The slogan ‘stochastic parrots’ outgrew the argument that coined it
The 2021 paper grounds two of its four risks in studies it cites, and leaves the other two, an argument about research priorities and the philosophy behind its slogan, without a citation under either.
BERT's near-human score on a reasoning test collapsed once the shortcut was closed
Devlin, Chang, Lee and Toutanova's 2018 model set new marks on eleven language tasks by training on masked words instead of the next one. One of its most human-looking scores, though, traced to a handful of words a benchmark let it exploit, not the argument the task was built to test.
The paper credited with starting the LLM era never trained a language model
Vaswani et al.'s 2017 paper reported BLEU translation scores on two language pairs and never once trained the kind of model it is now credited with inventing.
OpenAI's 2020 scaling paper got the curve right and the budget wrong
DeepMind's 2022 Chinchilla paper retrained the same comparison and found the right split was close to even, not Kaplan's five-to-two tilt toward model size.
Eight worked examples more than tripled a large model's math score
The 2022 Google paper showed the trick helps only around 100 billion parameters and larger, and it left open whether the written steps are how the model reaches its answer.
A big part of Apple's "reasoning collapse" was models hitting an output limit
The 2025 "Illusion of Thinking" study gets cited to prove reasoning models can't really reason, but its headline Tower of Hanoi failures were largely the models running out of room to print the answer.
A 1.3B model beat GPT-3 on the one metric OpenAI trained it to win
The 2022 InstructGPT paper made RLHF the recipe behind ChatGPT by optimizing which answers labelers preferred, and its real gains in truthfulness and safety were partial, not the guarantee the method is now credited with.
Microsoft's 'sparks of AGI' study tested a GPT-4 the public never got
A qualitative probing study by fourteen Microsoft researchers, run on an unreleased model no outsider can check and carrying a contamination problem the authors admit, now props up claims its own method was never built to support.
The sudden jump in AI benchmarks was mostly in the scoring
Wei's 2022 catalog of abilities that appear only past a scale threshold met a 2023 rebuttal that re-scored the same model outputs for partial credit and found most of the threshold in the ruler.
Few-shot GPT-3 beat fine-tuning on a handful of tasks and trailed it on most
The same tables that show genuine few-shot arithmetic also leave most benchmarks behind fine-tuning and cannot separate a capability the prompt taught from one the pretraining already held.
Chinchilla beat a model four times its size on the same compute
The paper's tables buy a training-compute-optimal ratio near twenty tokens per parameter, evidenced three ways, but not the settled law the field now quotes or the inference cost that makes everyone train past it.