GPT-4's technical report declares 'no further details'

One sentence in Section 2

The GPT-4 Technical Report opens by describing what the model can do. Its abstract states that GPT-4 "exhibits human-level performance on various professional and academic benchmarks... a score around the top 10% of test takers."¹ Two pages later, in a section titled "Scope and Limitations of this Technical Report," it describes what the report itself will not do: "GPT-4 is a Transformer-style model pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers. The model was then fine-tuned using Reinforcement Learning from Human Feedback (RLHF). Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."¹

That sentence, not the abstract, is where the document draws its own boundary. Everything the rest of the report publishes, benchmark scores, exam percentiles, contamination checks, red-team findings, sits on one side of the line. How the model was actually built sits on the other, by the report's own account.

The document runs 100 pages.¹ Fourteen of them are the numbered sections, Introduction through Conclusion, that make up the report proper.¹ The other 86 are acknowledgements, references, seven appendices, and a roughly 60-page System Card bundled in as Appendix H, itself part of the same PDF OpenAI published.¹ None of those 86 pages crosses back over the line Section 2 draws.

What the report measures itself

What the report does publish is a capability battery: 34 exams and standardized tests, from the Uniform Bar Exam to the LSAT, three sections of the GRE, fourteen Advanced Placement subjects, and three tiers of Leetcode coding problems, each given a raw score and an estimated percentile.¹ The method is stated plainly: "We did no specific training for these exams," and where a portion of a test's questions turned up in GPT-4's training data, the report reruns the exam with those questions removed and reports "the lower score of the two."¹

The contamination check behind that method is one of the report's more careful pieces of work. It compares 50-character substrings from each exam question against the training set, three substrings per question, and publishes the result exam by exam rather than as one aggregate figure.¹ For the Uniform Bar Exam specifically, the check found no overlap at all: zero percent contamination across all 400 questions, with an identical score whether the empty contaminated subset is counted or not.¹ The report's headline MMLU score gets similar precision on one point. The number is 86.4 percent, and the table names the method right next to it: five-shot.¹ Architecture, training compute, and dataset construction get none of that specificity. The exams do.

Competition, safety, and outside review

The report does not present its own gap as permanent. Immediately after declining detail, Section 2 adds: "We are committed to independent auditing of our technologies, and shared some initial steps and ideas in this area in the system card accompanying this release. We plan to make further technical details available to additional third parties who can advise us on how to weigh the competitive and safety considerations above against the scientific value of further transparency."¹

OpenAI's chief scientist, Ilya Sutskever, gave a fuller account of those two considerations the day after GPT-4 launched, calling them "self evident."² On competition: "It's competitive out there... GPT-4 is not easy to develop. It took pretty much all of OpenAI working together for a very long time to produce this thing."² On safety, in a separate interview the same week: "Safety is not a binary thing; it is a process. Things get complicated any time you reach a level of new capabilities."³

"They're disclosing nothing about the contents of their training set," wrote Ben Schmidt, a vice president at Nomic AI, the day of launch.² The report is narrower than that. It names two categories of data: "publicly available data (such as internet data) and data licensed from third-party providers."¹ It names the fine-tuning method too, Reinforcement Learning from Human Feedback. What it declines is everything after that: how much data, from where, filtered how. "No further details," the report's own phrase, describes what's missing more precisely than "nothing" does.

Nature's news desk covered the same launch and found researchers who were, in its words, excited about the model's abilities but frustrated "that they cannot yet access the technology, its underlying code or information on how it was trained."⁴ The system the report offers in place of that access is outside review, not open method. More than 50 outside experts were engaged to test the model for risk before release, including the Alignment Research Center, given early access specifically to assess "risks from power-seeking behavior."¹ That is a real form of scrutiny. It answers a different question than how the model was built, and OpenAI chose the experts who got to ask it.

A number the report didn't compute itself

Table 1's most quoted row is the Uniform Bar Exam: GPT-4 scores 298 out of 400, next to an estimated percentile of "~90th."¹ GPT-3.5 scores 213, "~10th," on the same row.¹ OpenAI did not administer that exam. The report's acknowledgements credit outside collaborators, Pablo Arredondo, Daniel Katz, and Michael Bommarito of Stanford CodeX, along with Shang Gao of Casetext, for running the simulated test.¹ That is the same team that separately published its own paper on the result, "GPT-4 Passes the Bar Exam."⁵

Katz and his coauthors' own topline figure is 297 points. A footnote adds that the best combination of prompt and settings would push the score "to 298 or higher."⁵ OpenAI's table prints 298: the number from that footnote, standing in for the paper's own headline result of 297. The percentile comes from the footnote right after it: "Using a percentile chart from a recent exam administration (which is generally available online), ChatGPT would receive a score below the 10th percentile of test-takers while GPT-4 would receive a combined score approaching the 90th percentile of test-takers."⁵ No chart is named. No comparison population is named. Those two sentences, offered in passing, are the entire origin of the number that became Table 1's headline column.

An earlier lesson recomputed the same roughly 298-point score against the population a claim about passing the bar exam actually implies: attorneys who passed it, not test-takers in general. Against that group, GPT-4's aggregate score lands near the 45th percentile, not the 90th.⁶

The contamination check traces to data OpenAI controlled and measured directly. The percentile traces to a population no document ever names, borrowed from a chart Katz and his coauthors gestured at and never cited themselves.⁵ The same report, on the same page, carries a number it verified next to a number it only repeated. The distinction holds for any claim sourced to this document: some numbers the report measured itself, the way it measured contamination; others it only repeated, the way it repeated a percentile it never checked. Architecture, training compute, and dataset construction are neither measured nor repeated. Section 2 says so directly, and says why, in the order it gives: competition, then safety.¹