The sudden jump in AI benchmarks was mostly in the scoring
Wei's 2022 catalog of abilities that appear only past a scale threshold met a 2023 rebuttal that re-scored the same model outputs for partial credit and found most of the threshold in the ruler.
Change which SWE-bench tasks are counted and GPT-4o's score doubles
SWE-bench grades a model by running a real repository's own tests in a sealed container, and OpenAI reports the same GPT-4o clearing that bar on 16% of the original 2,294 tasks but 33.2% of the 500 its annotators kept.
Zillow writes down the houses its model told it to buy
Its iBuying arm, Zillow Offers, paid an automated model's price for thousands of houses, then shut down in November 2021 once those houses were worth less than Zillow had paid.
A model can recognize strawberry's letters and still miscount them
The word arrives as the three integers 496, 675, and 15717, and counting its r's would mean recovering the spelling from those numbers and then running a tally the model was never built to run.