Microsoft's ‘sparks of AGI’ study tested a GPT-4 the public never got

A probing study by the deployer

“Sparks of Artificial General Intelligence: Early experiments with GPT-4” is a 155-page paper posted to arXiv in March 2023 by fourteen researchers, every one of them at Microsoft Research.¹ Its central claim sits in the conclusion, stated plainly: “GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence.”¹ That sentence is why the paper is famous, and it is the sentence people reach for when they say GPT-4 is nearly AGI.

Underneath the title is a specific kind of study. The researchers ran prompts at an early build of GPT-4, kept the exchanges they found impressive, and wrote them up. There was no held-out test set, no scoring protocol, no independent team checking the results. The authors describe their method as “closer to traditional psychology rather than machine learning,” and grant that it “is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation.”¹ This is a qualitative probing study, a report of what a model did on prompts the authors chose. It is not a controlled evaluation, the kind that fixes its questions in advance and reports how often the model gets them right. Everything the paper can and cannot support follows from that.

The model it tested was not the one you got

The GPT-4 in this paper is not the GPT-4 anyone else used. The authors are precise about it. They studied “an early and non-multimodal version of GPT-4,” a text-only build OpenAI handed them before release.¹ The GPT-4 the public received, documented separately by OpenAI, was a later system that also handled images.² They share a name. Little else about them can be pinned down from outside.

The build also did not sit still. The experiments ran while, in the authors' phrase, “GPT-4's development continued,” and one figure shows the model's attempt to draw a unicorn changing across three tries “over the span of a month while the system was being refined.”¹ The thing under study was moving during the study. The authors put the consequence in a footnote on the fourth page.

No one outside the two companies can run the exact model the paper describes. A result that cannot be re-run cannot be checked by anyone else, which leaves the transcripts as a careful record of one lab's private experience.

How the prompts were chosen

A controlled evaluation fixes its questions ahead of time, keeps some of them hidden from the model, and reports a success rate you can reproduce. This study does none of those things. The researchers wrote prompts by hand and adjusted them when they wanted a better answer. They asked for a mathematical proof “in the style of Shakespeare.” They drew a unicorn, erased its horn, and asked the model to put it back.¹ They kept the exchanges that worked. The transcripts are selected to be convincing, which is what makes them vivid and what stops them from being a measurement.

A second problem is one the authors raise themselves. They cannot see GPT-4's training data, so, in their words, they “have to assume that it has potentially seen every existing benchmark.”¹ They offer a concrete tell. GPT-4 reproduced the canary GUID from BIG-bench, a random identifier the benchmark's authors planted so that anyone could detect whether their test had leaked into a model's training set.¹³ If the model has already read the answer key, a correct response shows memory as easily as reasoning, and this method cannot tell the two apart.

What a single transcript can prove

Grant the impressive part first, because it is real. Some transcripts in the paper are genuinely surprising, and anyone who runs its prompts will watch a model do things that look like reasoning.¹ The question is what one good transcript establishes. It establishes that the model produced that output, once, on that prompt. It does not establish how often the model succeeds, on prompts it has not seen, under conditions someone else controls. Showing a capability once and demonstrating it under controls are different claims, and only the second is a measurement.

Melanie Mitchell and David Krakauer, writing about the same debate in 2023, name the trap. A benchmark result can be memorization dressed as reasoning, because “many other, as yet undetected, subtle shortcuts likely exist” in the tasks used to test understanding.⁴ The way to separate reasoning from recall is to change a problem slightly and see whether the skill survives. Tomer Ullman did that to the kind of theory-of-mind puzzle GPT-4 was reported to pass, and found that small, meaning-preserving edits to the wording “turn the results on their head.”⁵ Ullman was answering a different paper, not this one, but the point transfers, because Sparks reports the same kind of success on the same kind of hand-chosen prompt. Perturb the prompt and the display can disappear. This series has met the pattern before, in emergent-abilities, where re-scoring the very same model outputs changed which skills looked like sudden jumps.

Gary Marcus made the gap concrete. He relayed a reader's attempt to run the paper's prompts on the GPT-4 that the public could actually use, in which four of five failed to replicate.⁶ The exchanges held for the authors, on their private build. They did not hold for an outsider on the shipped model. That is the distance between a display and a demonstration.

The citation drops the conditions

The paper is more careful than its reputation. Bubeck and colleagues spend pages on what GPT-4 cannot do. There is a section on failures built into the model's design, a list of missing pieces that includes no long-term memory, no continual learning, and unreliable planning and arithmetic, and a closing admission that the study is “entirely phenomenological,” fixed on surprising behavior and silent on why it happens.¹ The authors did not bury the weaknesses. They wrote them down.

The distortion happens downstream. When “sparks of AGI” is cited as though it settled a question, it carries the four-word claim and drops every condition the authors attached to it: the unreleased model, the hand-picked prompts, the admitted contamination, the phenomenological method. The company's own leadership was more guarded than the citation is. The same month, OpenAI's Sam Altman said GPT-4 “is still flawed, still limited, and it still seems more impressive on first use than it does when you spend more time with it.”⁷ Mitchell, reading the paper closely, took its claim to be about “generality in its ability to deal with different kinds of tasks,” not human-level intelligence across the board.⁸

The authors themselves kept adjusting the frame. The first version on arXiv defined intelligence by borrowing a 1994 public statement, the “Mainstream Science on Intelligence” editorial, which was written to defend the claims of the book The Bell Curve.⁹¹⁰ By the current version, three weeks and four revisions later, that grounding was quietly gone.⁹ A paper that revises the definition of its central term inside a month is not a finished result. It is a work in progress that picked up a permanent citation.