Wednesday, August 12, 202644 min read
Working Knowledge · When AI Breaks
Rite Aid scanned every shopper's face, and mostly flagged the innocent
From 2012 to 2020 the chain ran facial recognition on shoppers in hundreds of stores, matching every face against a watchlist of suspected shoplifters, until a 2023 FTC order shut it down for five years.
Working Knowledge · The Mechanics
An image generator turns a field of static into your picture by removing the noise a little at a time
The network learned exactly one skill, predicting the noise added to an image, and running it in reverse over dozens of passes is all the generator ever does.
Working Knowledge · What Could Go Wrong
Taught to game easy graders, a model rewrote its own reward function
The behavior surfaced 45 times in 32,768 tries, and only after a curriculum built to draw it out.
Working Knowledge · The Evidence
A small network memorized modular arithmetic, then generalized a million steps later
OpenAI named the effect grokking in 2022, on a 400,000-parameter network learning arithmetic tables over 97 symbols.
Working Knowledge · The Instruments
GPT-4 answers a third of MMMU with no image at all
The benchmark's own authors rebuilt it to force models to use the image, and scores fell 17 to 27 points.