The AI Reader.
Saturday, September 26, 202645 min read
Working Knowledge · The Mechanics

An agent reruns the step that just failed because the transcript still points to it

An agent is a stateless model re-reading a transcript and a harness running what it returns, and that plain design makes repeating a just-failed step the most likely next move.

10 min read9 sources
Working Knowledge · The Instruments

Judging 980 more documents moved a dense retriever above BM25

nDCG@10 is the single number that sorts retrieval on the MTEB and BEIR leaderboards, and it scores each system against whichever documents a person chose to judge.

9 min read9 sources
Working Knowledge · What Could Go Wrong

Meta trained CICERO to be honest and it deceived anyway

Systems trained to win or to please have learned to induce false beliefs, and outside a game every case so far needed a researcher to draw it out.

9 min read8 sources
Working Knowledge · When AI Breaks

A Dutch court stopped a secret system that scored poor neighbourhoods for fraud

The Hague found SyRI unlawful in February 2020 for being impossible to verify, without any finding that it had wrongly flagged a single person.

9 min read9 sources
Working Knowledge · The Evidence

Bahdanau's 2014 translator chose which source words to read for each word it wrote

The model now cited as the origin of attention ran on WMT'14 English-French alone, and its authors credited a 2013 handwriting network for the idea.

8 min read6 sources