Humanity's Last Exam only admits questions today's models fail

The score starts near zero

Humanity's Last Exam is a set of 2,500 public questions in mathematics, the humanities, and the natural sciences, each with a single known answer that, in the authors' words, "cannot be quickly answered via internet retrieval."¹ It was built because the tests before it had run out of room. Models now top 90% on MMLU, a broad multiple-choice exam of general knowledge, so a passing score there no longer separates one model from the next.²

On Humanity's Last Exam they separate again, because almost all of them fail. As read on 2026-08-14, the top model on the official no-tool leaderboard, gemini-3.1-pro-preview, answered 46.44% of the questions correctly. The board is live and rises over time.³ At the January 2025 launch the best model reached 13.4%, and most sat in the single digits.² That gap, near zero at launch and under 47% today, is the number people quote. How it is produced explains both halves: why it starts so low, and why it rises so fast.

Only questions the models already miss

The questions come from people, not a model. Nearly 1,000 subject-expert contributors from more than 500 institutions across 50 countries wrote them, and reviewers holding a graduate degree in the field checked each one across two rounds.²

What makes the set unusual is the filter. Before a question is admitted, it is put to several frontier models. A multiple-choice question is kept only if those models, on average, do worse than random guessing; a short-answer question is kept only if the models cannot solve it.² The pipeline logged about 70,000 model attempts and produced roughly 13,000 questions that stumped the models, later narrowed to the released set.² The test is, by design, the questions the strongest models of the day get wrong.

The near-zero launch score follows directly from that rule. Every question was chosen because strong models missed it, so a low score measures a model against a pool built from those failures, not against knowledge in general. Raise the models and the filter has to be rebuilt, which is part of why the score climbs as fast as it does.

The composition is roughly what you would expect of an exam. Of the 2,500 public questions, a fifth to a quarter are multiple-choice, the rest are short-answer, and about a tenth carry an image. The paper and the official site give slightly different splits: 24% versus about 20% multiple-choice, and 14% versus about 10% multimodal.²⁴ The set has also moved. The January 2025 announcement described 3,000 questions.⁵ The release settled at 2,500, with a private held-out set kept back to check for overfitting.¹ Questions later flagged as flawed or web-searchable have been "removed and replaced."⁴

A model marks the answers

Grading is automatic. A separate model, o3-mini, compares each answer to the stored correct one and decides whether they match, allowing equivalent forms such as a decimal for a fraction.² No person marks the runs, which is what lets thousands of answers be scored the same way every time.

The score arrives with a second number that gets less attention: calibration error. Every model is asked for both an answer and its confidence, from 0 to 100%, and calibration error measures how far that confidence sits from how often the model is actually right.⁴ A model that is 90% confident but 10% correct is badly calibrated, and that pairing is the failure the benchmark was built to expose.

At launch the numbers were stark in both columns. The models scored in single digits and were confident anyway, with calibration error from 73% to 89%.² Being wrong is one thing. Being wrong while sure is what the second column records. Even now the leaders carry calibration error between 38% and 57%, so overconfidence shrinks as scores rise without clearing.³

The 26% that used tools

None of the numbers so far let the model use tools. The leaderboard runs every model closed-book, at temperature 0, on the public questions, with no browsing and no code.³ Allow tools and the score jumps. OpenAI's GPT-5 answered 24.8% with reasoning and no tools, and 42.0% when the same family was given web search and a Python tool.⁶ The questions did not change. The model's access did.

That gap is where the benchmark has misled people. On 12 February 2025, Fortune reported that OpenAI's deep research "can complete 26% of Humanity's Last Exam," and described the test as a benchmark for the frontier of human knowledge.⁷

Two things are wrong with reading it that way. The 26.6% run used web browsing and Python tools, while the roughly 9% scores it was called a "nearly threefold jump" over were no-tool leaderboard numbers.⁷ Setting a tool-assisted score against no-tool scores compares two different tests. And the questions are closed-ended academic problems with known answers, not a survey of everything a world-class expert holds, so "the frontier of human knowledge" claims more than the test measures.²

Put on one axis, the error is visible. The no-tool scores form one rising line, and the tool-assisted runs sit above it as separate points. Reading a point off the upper set as though it belonged to the lower line is exactly the jump Fortune described.

What a high score would not prove

So what would a high HLE score show? The authors are explicit about the limit.

A top score means a model answers hard, closed academic questions. It does not mean the model can carry out open-ended research.² The misreading runs against the benchmark's own stated scope, not against a critic's.

There is a second limit, in the answer key itself. FutureHouse, an AI research lab, audited 321 text-only chemistry and biology questions and found that 29%, give or take 3.7 points, had answer keys that conflict with the peer-reviewed literature.⁸ The HLE team's own re-check of a subset put the problematic share lower, at about 18%.⁸ Take the range as 18% to 29%: on that slice, close to a quarter of the questions can mark a model wrong for giving the better answer. Part of what a low score measures is the test's own errors.