A model's "37% hallucination rate" was its wrong-answer share on SimpleQA

A quiz score, reported as a hallucination rate

In March 2025, ABC's science desk reported that GPT-4.5, then OpenAI's newest model, "hallucinated 37 per cent of the time," and that the model before it, GPT-4o, "hallucinated 62 per cent of the time."¹ The figures are real, and they come from one specific test. GPT-4.5 sits at the top of OpenAI's public SimpleQA leaderboard, answering 62.5 percent of the benchmark's questions correctly and getting about 37 percent of them wrong.² SimpleQA is a factual-recall benchmark OpenAI published in late 2024: a fixed set of short, fact-seeking questions, each written to have one indisputable answer, and a grade for whether a model's response matches it.³

Calling the wrong share a "hallucination rate" is not the reporter's coinage. OpenAI's own o1 system card, from December 2024, lists SimpleQA inside a section headed "Hallucination Evaluations" and prints, beside each model's accuracy, a "hallucination rate (lower is better)" that is exactly the share of answers the grader marked wrong.⁴ The coverage carried OpenAI's own label to the next model. So one run of one test yields two headlines. "Answers 62.5 percent correct" and "hallucinates 37 percent of the time" describe the same model sitting the same quiz. Only the first label is fair to the test.

Every question stumped at least one of four models

A benchmark's number is only as meaningful as the questions behind it, and SimpleQA's questions were selected by a rule that shapes everything downstream. The finished set holds 4,326 questions.³ Each one was made the same way.

1. Write a question with a single answer

   A human trainer wrote a short question whose answer is one indisputable fact, backed by a specific reference web page, and chosen so the answer would not change over time.³

2. Confirm the answer independently

   A second trainer answered the same question without seeing the first trainer's answer, and the question was kept only if the two matched. On a 1,000-question recheck by a third trainer, the answers agreed about 94 percent of the time, and the authors estimate the finished set still carries roughly a 3 percent labeling error.³

3. Keep it only if a model gets it wrong

   The trainer then showed the question to four separate answers from GPT-4-class models. The question survived only if at least one of the four answered incorrectly. If all four got it right, the trainer was instructed to discard the question and write a harder one.³

SimpleQA is not a sample of the facts a model meets in a day. It keeps only the facts a strong model was already prone to miss. A SimpleQA percentage is therefore always a percentage on questions pre-selected for being hard for models of roughly that class. When OpenAI released the benchmark, the models it tested scored below half: GPT-4o answered 38.2 percent of the questions correctly, and Claude 3.5 Sonnet 28.9 percent.³ Those low numbers are the design working as intended, not a verdict that the models are bad at facts in general.

A wrong answer and a blank are graded apart

How each answer is graded matters as much as which questions get asked. Every response lands in one of three bins.³

• Correct: the answer contains the reference answer and does not contradict it.
• Incorrect: the answer contradicts the reference answer in any way, even when the contradiction is hedged.
• Not attempted: the answer neither states the reference answer nor contradicts it, which is what a model declining to commit looks like.

An automated grader, a prompted copy of ChatGPT, reads each response and files it in one of the three bins.³ Grading answers with a model has its own failure modes, which a separate lesson takes up. What matters here is the split itself, because it is the part the "hallucination rate" framing erases. A model that says "I'm not sure" is not scored the same as a model that states something false with confidence. Only the second is counted incorrect.

GPT-4o attempted almost everything and was wrong most of the time, leaving only 1.0 percent of questions unanswered and getting 60.8 percent of them wrong.³ Claude 3.5 Sonnet did the opposite, declining to answer 35.0 percent of the questions. Read only the raw correct rates, 38.2 for GPT-4o against 28.9 for Claude, and GPT-4o looks clearly stronger on facts. But of the questions each model actually tried, Claude was right more often, 44.5 percent against GPT-4o's 38.0.³ Which model is "better" depends on whether you reward answering or reward knowing when to stay quiet, and a single headline percentage buries the choice.

SimpleQA reports both. "Correct given attempted" is the score on only the questions the model chose to answer. The benchmark's single summary figure blends that with the overall correct rate, so a model cannot win by staying silent, nor by guessing at everything.³ The reason for measuring abstention at all is that a model which could tell which of its answers were shaky would decline those and answer the rest. Models are poor at that self-assessment and tend to overstate their confidence, a failure with its own lesson, which is exactly why the not-attempted grade has to be kept separate from the wrong one.³

How the incorrect share became a hallucination rate

When OpenAI's o1 system card renamed the incorrect share a "hallucination rate," it took a number defined as "answered a hard trivia question wrong." It then attached the name of a different and scarier event: a model inventing a fact that has no basis at all.⁴ Those are not the same thing. A model can miss a SimpleQA question because the answer is genuinely obscure and it never learned it, which is closer to a gap in knowledge than a fabrication. And because the questions were selected for being missable, the wrong-answer share is inflated by design against how often the same model would slip on an ordinary fact.

GPT-4.5's own system card does report a hallucination rate, but it measures that with a different test, PersonQA, a set of questions about facts on named people, rather than with SimpleQA.⁵ The 62.5 percent correct that circulates for GPT-4.5 comes instead from OpenAI's simple-evals leaderboard.² The "37 percent" paired with it is just the wrong-answer share of that same run, the leftover of 62.5, not anything the system card's hallucination table reports. Two numbers about one model, drawn from two different tests, get merged into a single claim about how often it hallucinates.

None of this makes SimpleQA a bad number. It makes it a narrow one. Its own authors write that it "only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer."³ The score is also tied to one language and one slice of facts. DeepSeek reported its V3 model answering 24.9 percent of SimpleQA's English questions correctly, while scoring 64.8 on a separate Chinese-language version of the same test.⁶ And a SimpleQA result says nothing about whether a model repeats the false beliefs people actually hold, which is a different measurement again, TruthfulQA, built from 817 questions designed around common human misconceptions.⁷

The trouble starts whenever the number is lifted out of that setting. One industry compilation lines up self-reported SimpleQA and hallucination scores from across labs, and warns that "relying solely on public scores like TruthfulQA or SimpleQA creates a false sense of readiness." That same table then files SimpleQA in one bucket with the misconception test and the people-facts test, the exact merge the score cannot support.⁸ Daswin de Silva, an AI researcher quoted in the same ABC report, put the limit plainly. The benchmark, he said, "is only testing for short, fact-based queries and that's not really the first-use case for ChatGPT."¹