The AI Reader.
Saturday, August 1, 202652 min read
Working Knowledge · What Could Go Wrong

An optimizer wrote a jailbreak that transferred across five AI models

The strongest published defense has since cut a broad set of jailbreak attempts from 86% success to under 5%, and whether it can ever reach zero is still disputed.

13 min read16 sources
Working Knowledge · The Instruments

Groq's 270 tokens a second and Anyscale's 185 are both true

The 85-token-per-second gap between them comes down to where Groq starts its stopwatch and how long the test runs, one of at least four choices that decide what any published tokens-per-second figure means.

11 min read16 sources
Working Knowledge · The Mechanics

A model that calls a tool never runs it

The JSON a model writes when it "calls" a tool executes nothing on its own — a program outside the model performs the real action and writes the answer back as text.

10 min read8 sources
Working Knowledge · When AI Breaks

Google's flu predictor overshot the CDC for 100 of 108 weeks

Nature validated Google's search-based flu model in 2009, and the search engine feeding it kept changing underneath a relationship the model never got to refit.

10 min read7 sources
Working Knowledge · The Evidence

GPT-4's technical report declares 'no further details'

The 100-page report scores GPT-4 on 34 exams and states, in Section 2, that it gives no further details on how the model was built.

8 min read6 sources