Meta's own Llama report can't certify four of its top scores

The report grades its own product

In July 2024 Meta released a long technical report and, in the same week, the weights of the largest model it describes.¹ The report is titled "The Llama 3 Herd of Models." A line in its first table corrects the title before the reader gets far: "All results in this paper are for the Llama 3.1 models." The flagship is a dense Transformer with 405 billion parameters and a context window of up to 128,000 tokens, and it ships as Llama 3.1 405B.² The 8-billion and 70-billion models arrived in April as Llama 3; the multilingual, long-context, tool-using versions and the 405B are the July release the paper actually measures.²

Two things run through the document at once. It is an unusually detailed engineering record: it names the model's size, the compute spent to train it, and the broad shape of its training data. It is also a launch, and the abstract states that Llama 3 "delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks."¹ The report is candid about the system it built and confident about how that system ranks against its rivals. Meta measured the first directly. It largely scored the second itself.

What the 405B took to build

Start with what the report states plainly, because these are numbers only Meta could publish. Llama 3.1 is a family of three dense Transformers, at 8 billion, 70 billion, and 405 billion parameters.² A parameter is one of the adjustable numbers a network learns during training. "Dense" means every parameter is used on every token, unlike a mixture-of-experts design that routes each token through only a fraction of the network. Meta chose the dense design on purpose, to "maximize training stability" at this scale.²

The 405B was pre-trained on 15.6 trillion tokens of text, with a knowledge cutoff at the end of 2023.² A token is a chunk of text a few characters long; Llama 3's tokenizer packs English into about 3.94 characters per token, up from 3.17 for the Llama 2 tokenizer.² The stated training budget was 3.8 × 10²⁵ floating-point operations, run on up to 16,000 H100 GPUs.²

Those figures are hard to hold. The model card restates the compute in a unit that is easier to feel: 30.84 million GPU-hours on H100-80GB cards to train the 405B alone, and 39.3 million across the whole family.³ Each card draws 700 watts. Meta reports the training emitted 11,390 tons of CO₂-equivalent on a location-based measure, and zero on a market-based one after matching the load with renewable energy.³

The report also gives the shape of the training data without naming a single source inside it. The final mix is roughly 50% general-knowledge tokens, 25% math and reasoning, 17% code, and 8% multilingual.² That four-way split is the most the paper says about the corpus. Meta removed personal information and adult-content domains, but lists no sites, no datasets, and no named proportions.² The weights are public. The data that made them is described only in outline.

What Meta's "open" license actually grants

"Open" is the word carrying the most weight in the launch. What Meta released is the weights and several kinds of code: model code, inference code, and training and fine-tuning code.⁴ It did not release the training data, and the four-way mix above is as close as a downloader gets to it.² The accurate description is open weights under a conditional community license.

The conditions are real but bounded. A user must display "Built with Llama" and keep Meta's copyright notice.⁴ Any model trained on Llama or its outputs must put "Llama" at the start of its name.⁴ And a company with more than 700 million monthly active users on the release date cannot simply download it. It "must request a license from Meta," grantable at Meta's "sole discretion."⁴ That clause is aimed at Meta's largest rivals.

One restriction people carry over from Llama 2 is gone. Llama 2's license forbade using a model's outputs to improve any other large language model. The Llama 3.1 license drops that ban and asks only for the renaming.⁴ Training a competitor on Llama's outputs is allowed, so long as the competitor's name begins with "Llama."

Meta calls the 405B "the first frontier-level open source AI model."⁵ The Open Source Initiative, which authors the definition of that term, disagrees.

Both sides are reading the same license text, and the text settles the substance even where it does not settle the word. A downloader gets far more than any closed API allows: the actual weights, and the right to run and fine-tune them commercially.⁴ A downloader does not get the training data, an unconditional grant, or the OSI's blessing on the phrase "open source." Open weights under conditions is the description that survives both the launch blog and the rebuttal.

The comparison Meta scored itself

The claim that put the report on every timeline is the one in its abstract: comparable to GPT-4.¹ The evidence offered for it is Table 2, where the 405B is lined up against GPT-4, GPT-4o, and Claude 3.5 Sonnet across sixteen benchmarks.² Two facts about how that table was built matter before reading a single number. Meta "evaluate[d] the performance of other models ourselves and compare[d] the results with the reported numbers, selecting the best score" for each competitor.² The comparison is Meta's own run of its rivals. And the winners are not the ones the parity claim implies.

On the general, code, and reasoning rows, the best score usually belongs to GPT-4o or Claude 3.5 Sonnet, not the 405B.² The 405B leads on some rows, GSM8K among them, and beats the older GPT-4 (0125) almost everywhere. Against the newer closed models it lands at parity or a step behind, and the report itself concedes the 405B "trails Claude 3.5 Sonnet in capabilities such as coding and reasoning."²

Two checks beyond the benchmark table point the same way. Meta's human-preference evaluation placed the 405B roughly on par with GPT-4 and mixed against GPT-4o and Claude 3.5 Sonnet.² LMArena, which ranks models by blind human preference rather than benchmark accuracy, put the 405B at rank 3 on its hard-prompts board after correcting for answer length and formatting, behind GPT-4o and a top-tied Claude 3.5 Sonnet.⁷ Preference is not accuracy, so that ranking gives the table context rather than overturning it.⁷

Put together, the reading the numbers support is narrow and defensible. The 405B was the strongest openly available model at its release, competitive with the closed frontier and behind it on the hardest tasks. The stronger reading, that the table shows Llama beating GPT-4o and Claude, rests on Meta's own scoring of its own rivals, and the numbers in it do not carry that far.

The four scores the report leaves blank

The report's most useful admission sits in a table most readers never reach. Contamination is when test questions, or near-copies of them, have leaked into the training data, so a high score can mean the model saw the answers rather than learned the skill. Meta checked for it. The method scores each benchmark example by how far its text overlaps, in runs of eight tokens, with the pretraining corpus, and Table 15 reports the share of each test that looks contaminated and the points that contamination is estimated to have added.²

For several benchmarks the table returns a clean-looking answer. HellaSwag shows heavy overlap and an estimated 14.3-point gain for the 405B. GSM8K shows 41% of the test flagged but almost no gain. SQuAD shows nothing.² For four of the benchmarks in the flagship comparison, the report gives no estimate at all.

The report says why, in its own words: for those four, "even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate."² So much of each test appears to overlap the training data that the method cannot separate memorized answers from learned skill.

MMLU, HumanEval, and MBPP are among the headline general and code benchmarks in Table 2, the same table that carries the comparison to GPT-4.² Meta ran an honest check on whether contamination inflated those scores, and the check could not return an answer. On its own most famous numbers, the report cannot vouch for what it published, and it says so plainly.