AI forecasters beat the human crowd's Brier score by 0.002 on a subset of the questions

The 0.03 gap a language model was supposed to have closed

In 2024, a team at UC Berkeley built a system that reads the news, searches the web, and turns what it finds into a probability for questions like whether a ceasefire will hold or a court will rule a particular way. Tested on 914 such questions against a crowd of human forecasters answering the same ones, the system scored 0.179 in Brier score to the crowd's 0.149.¹ Lower is better on this scale, so by the paper's own headline numbers, the system lost.

That number, the Brier score, is the standard way to grade a probability forecast, and it sits behind nearly every claim that an AI model is approaching human-level forecasting. Before weighing 0.179 against 0.149, it helps to know exactly what the score counts.

The idea, from the meteorologist Glenn Brier in 1950, is plain arithmetic.² For every yes-or-no question, the forecaster states a probability. Once the answer is known, square the distance between that probability and what happened (1 if it happened, 0 if it didn't), and average the squares across every question. Here is a small, made-up set of four forecasts showing what that arithmetic rewards and punishes.

Being confident and right costs almost nothing, 0.01 either way. Hedging at 50 percent costs more than that even when it lands on the right side, 0.25. Being confident and wrong is the disaster, 0.81. Zero is a perfect run; the scale used below runs up to 1 for the worst one, stating certainty for the thing that didn't happen, every time.

Brier's own 1950 paper counted both outcomes at once, which doubles every score: his scale runs 0 to 2, and today's version, for a yes-or-no question, is exactly half of his.²³ He used his own scale to make a point worth keeping: a forecaster cannot do better by hedging, or by stating something other than what he believes. He took a ten-day record, three of the days rainy, and compared two hypothetical forecasters who never looked at the weather and simply repeated one number every time. The one who always said 30 percent, the true share of rain, scored 0.42. The one who said 20 percent instead did worse, 0.44.² Guess higher or lower than the truth and the score worsens either way; the best constant answer is the honest one, which is why statisticians call this a proper scoring rule.² Halawi and his coauthors report 0.179 and 0.149 on this halved scale, the one used by every other study below.¹

The low score you get without looking

A forecaster who never looks at a question's specifics can still post a good-looking number. Say a set of yes-or-no questions each has a fifty-fifty base rate, and the forecaster ignores every one and always answers 50 percent. The score is 0.25 on every question, since (0.5 − 1)² and (0.5 − 0)² both equal a quarter.¹ That is the baseline Halawi and his coauthors use to grade a model with no web access at all. Among models tested this way, only GPT-4 and Claude-2 cleared 0.25 "by a large margin," more than two hundredths of a point.¹

Fifty percent is an easy trap to spot, since a coin flip is an obviously lazy forecast. The trick gets harder to see once the base rate isn't fifty percent. Take a question set where the event happens only 10 percent of the time, a fair description of many forecasting-platform questions about rare disasters and long-shot outcomes. A forecaster who always answers 10 percent without reading a question scores 0.09, nine times better than the coin-flip forecaster's 0.25, without ever telling a likely case from an unlikely one.²

There's a reason the trick gets more effective the more lopsided the true rate is. In 1973, the meteorologist Allan Murphy showed that a Brier score splits into three pieces that add up to the whole, work reproduced with his own equations and citation by Ferro and Fricker's 2012 refinement of it.⁴ Reliability measures whether stated probabilities match how often things actually happen; the more they miss, the more it costs. Resolution measures whether the outcome rate genuinely differs depending on what he says — say the same number every time, and resolution is zero, however well that number matches the average.⁴ The third piece, uncertainty, isn't the forecaster's doing: it's the plain variance of the event itself, largest at a fifty-fifty base rate and smaller the more lopsided the true rate gets.⁴

Run that back through the always-10-percent forecaster above: its one stated probability matched the true rate exactly, so reliability was perfect, and it never said anything else, so resolution was zero. What was left was uncertainty alone, small because the true rate was small. Reliability is exactly what expected calibration error, taught in calibration error, isolates on its own. Resolution is close to what AUROC scores directly: not whether the stated numbers are true, but whether the forecaster tells cases apart at all. A Brier score folds both into one figure and can't tell you, by itself, which one a given result is rewarding.

One forecast, counted two different ways

None of the numbers above mean anything if the two things compared answered different questions. A Brier score averages over whatever questions it happened to be computed on. A crowd tested on ten near-certain questions will out-score a sharper forecaster tested on ten genuine toss-ups, with nothing about relative skill in the gap. ForecastBench, a 2025 project built specifically to compare forecasters fairly, put the requirement plainly. Every forecaster in the study, human or machine, answers the identical questions in the identical window, "to ensure comparability of scores across models and human-based aggregates."⁵ Without that, a Brier score proves nothing about who forecasts better, only who was tested on what.

There is a second way for two numbers both called a Brier score to disagree, and it has nothing to do with the questions. Good Judgment Open, one of the platforms whose questions feed the studies above, still scores exactly the way Brier did in 1950: both outcomes counted, on the 0-to-2 scale. Its own FAQ works the arithmetic: "For a yes/no question where you forecasted 70% and the event happened, your score would be (1-0.7)^2+(0-0.3)^2=0.18."⁶ Run that same forecast through the halved formula used by Halawi and every other study here, and it scores 0.09, exactly half.³

A 0.18 and a 0.09 look nothing alike, and both describe the same forecast and the same result. Copy a number from one convention into a sentence built for the other, with no note of which scale it came from. A mediocre forecaster then reads as twice as good, or twice as bad, as he was. Comparing two Brier scores means checking the questions and checking the scale.

The claim that didn't survive its own paper

Halawi and his coauthors did not build a naive test. All 914 questions postdated every tested model's training cutoff, so a system couldn't have memorized outcomes, and web searches were time-restricted so a system couldn't retrieve a page written after a question opened.¹ Removing the retrieval step entirely cost the system 0.027 points of Brier score — real evidence of a working system, not a trick.¹

None of that changes what the paper's own headline comparison shows. Across the full 914-question test, the system scored 0.179 to the crowd's 0.149, a loss.¹ The claim that the system "surpasses" the crowd, the one that made the paper's abstract, holds only on a narrower slice: questions where the crowd's own forecast sat between 30 and 70 percent. These are the genuinely uncertain calls. There, the system scored 0.238 to the crowd's 0.240, a gap of two thousandths of a point¹ — an order of magnitude below the two-hundredths-of-a-point gap the same paper elsewhere calls "large."¹ A critique posted to the AI Alignment Forum seven months later made exactly this comparison, an inference of its own rather than a sentence Halawi and his coauthors wrote, but one anyone can check by reading the two numbers next to each other.⁷

A year on, the same shape of claim still did not hold up next to real experts. Janna Lu's 2025 evaluation scored the best model, OpenAI's o3, at a median Brier of 0.1352 across 334 Metaculus questions. The expert panel Metaculus hired to check the bots answered only 157 of those same questions, not all 334, and posted a median of 0.0225.⁸ That is the same kind of mismatch Lu's own paper cautions against elsewhere in its results: Brier scores, it says, "are not directly comparable across different question sets."⁸

The clearest answer came from one of the 2024 paper's own authors. Danny Halawi co-wrote ForecastBench, built to give the public, superforecasters, and language models the identical questions in the identical window. On a shared 200-question set, the three groups scored like this:⁵

The best language-model configuration scored about the same as the public and landed nowhere near the specialists: superforecasters beat both groups at a significance level of better than one in a thousand, the paper states.⁵