The report that coined ‘foundation models’ ran no experiment

114 authors, 214 pages, no experiment

“On the Opportunities and Risks of Foundation Models” went up on arXiv on August 16, 2021, credited to the Center for Research on Foundation Models (CRFM) at Stanford’s Institute for Human-Centered AI. Its structured author list runs to 114 names, ending with Percy Liang as corresponding author.¹ The document has gone through three arXiv revisions since. Its current version runs 214 pages. The version circulating when the first critiques appeared ran 212.²

The report’s own table of contents lists six top-level parts: Introduction, Capabilities, Applications, Technology, Society, Conclusion, split into 26 numbered sections, each written by a different subset of the 114 authors. Its “Author Contributions” note says plainly that “not all the views expressed in this report are held by all the authors.”² None of the 26 sections is a Methods or Results section. A search of the full text for the ordinary language of a new finding turns up nothing: no “we conduct,” no “we trained,” no “our experiment,” no “we ran.”² The one number the report turns into a chart is not one it measured itself.

The report does call one part of itself an experiment, and it is not the models. “The writing of this report was an experiment,” it says: “we had over 100 people from different backgrounds come together to write a single report covering a wide range of aspects of foundation models.”² The abstract is careful about what it claims for the models themselves: “We call these models foundation models to underscore their critically central yet incomplete character.”² Incomplete is the operative word. The report is a survey and a research agenda organized under a new name. It does not report what that name’s subjects can do.

What the word ‘foundation’ was built to do

Section 1.1.1 of the report, titled “Naming,” is where the authors explain the word choice at length. Before this report, the models it discusses went by “large language models,” when they were models of language at all. GPT-3 and BERT fit that label. DALL-E, a model the report names as a third example in its own abstract, does not: it generates images from text, not text from text.² “Foundation model” is the report’s device for putting a language model and an image model under one name. It turns three separate models, across two fields, into a single paradigm shift. That is naming doing real work: it is the difference between a report about language models and a report about a change sweeping all of AI.

The same section, a few sentences later, states the limit on that claim: “At present, we emphasize that we do not fully understand the nature or quality of the foundation that foundation models provide; we cannot characterize whether the foundation is trustworthy or not.”² Both sentences are the report’s own, in the same section, a few lines apart. The first argues the word signals something about reliability. The second says the report cannot tell whether that reliability is there. Whether that gap matters, and how much, is not something the report itself weighs. Its readers did.

Marcus and Davis built their case on the report’s own hedge

Twenty-six days after the report posted, Gary Marcus and Ernest Davis published an essay-length rebuttal in The Gradient. Marcus is a psychology and neural science professor emeritus at NYU and founder of Robust.AI. Davis is an NYU computer scientist. CRFM itself republished the essay five weeks later as invited commentary on its own site.³⁴ Their argument goes straight at the section 1.1.1 hedge: “The report says, unironically, ‘we do not fully understand the nature or quality of the foundation that foundation models provide’, but then why grandiosely call them foundation models at all?”³ “Foundation models certainly sound cooler,” they write. “But sounding cooler doesn’t mean that those models provide the foundations AI so desperately needs.”³

They were not alone, and not only among researchers already skeptical of large models. Thomas Dietterich, a former AAAI president then at Oregon State, told Wired he suspected a motive the report’s naming section does not mention: “I was surprised that they gave these models a fancy name and created a center… That does smack of flag planting, which could have several benefits on the fundraising side.”⁵ The report’s own conflict-of-interest note names CRFM’s funders as of July 2022: Google, Microsoft, and the McGovern Foundation.² UC Berkeley’s Jitendra Malik put the objection to the word itself most bluntly, telling Wired the models are “castles in the air; they have no foundation whatsoever.”⁵ Marcus and Davis report him saying the same thing at a CRFM-organized workshop.³ A separate complaint came from Meredith Whittaker of the AI Now Institute. She said the report was renaming an existing category: “They renamed something that already had a name; they’re called large language models. This move constitutes an attempt at erasure.”⁶ Percy Liang, the report’s corresponding author, responded to the criticism directly: “All of these critiques are welcome.”⁵

None of these objections dispute that the models exist or that they matter. They dispute the conclusion the word chooses for the reader: a claim about those models’ reliability that the report’s own naming section says it cannot support.

The term reached federal regulation, then lost it

The objections did not stop the word from spreading. By 2024, “foundation models” appeared in the title of NIST Special Publication 800-218A, a federal technical guidance document on securing generative AI systems. The document uses the term throughout as settled, unremarked vocabulary.⁷ It reached further than a technical standard. Executive Order 14110, signed October 30, 2023, wrote “dual-use foundation model” into a formal legal definition. The order defines it as a model “trained on broad data,” using “self-supervision,” with “at least tens of billions of parameters,” capable of “high levels of performance at tasks that pose a serious risk” to security or public health.⁸ Fifteen months later, Executive Order 14148 rescinded it, along with the rest of the prior administration’s AI order.⁹ The term had reached binding federal text once, and did not stay there.

What the record does not settle is why this word won out over “large language model” and the other names already in circulation. No source traces the mechanism. What is documented is the fact of adoption, not its cause. A term one Stanford center proposed in August 2021, contested within a month by researchers well outside Stanford, sat in a federal publication three years later regardless.