ai-foundations
collection · 2 published
ai-foundations
Rebuild ImageNet's test set and the top model loses eleven points
A fresh test set built by the original recipe, no adversary needed, exposes how much of any accuracy number belongs to the data it was measured on rather than to the model.
ai-foundations
Move a tumor detector to a new hospital and it loses twenty-three points
Every benchmark accuracy measures one model against one distribution, and the distance from that distribution to where the model runs decides whether the number means anything.