Whisper transcribes audio it never trained on, and silence no one spoke

Whisper's best benchmark score was never the point

Radford, Kim, Xu, Brockman, McLeavey, and Sutskever released Whisper at OpenAI in September 2022, in a paper titled Robust Speech Recognition via Large-Scale Weak Supervision.¹ Its most-cited number is not impressive, and the authors say so. On LibriSpeech test-clean, a standard bank of clearly read audiobook speech, Whisper's best model reaches a word error rate of 2.5, the share of words it gets wrong. The paper calls that relatively unremarkable, roughly what a modern specialized model scores and about where the field stood in mid-2019.¹ A supervised competitor ties it there. What the paper actually defends is robustness: that the gap between how the model does on clean, familiar audio and how it does on messy, unfamiliar audio is unusually small, close to the spread a human shows.¹

Counting substitutions, insertions, and deletions

Word error rate lines a machine's transcript up against a correct one and counts three kinds of mistake: a substitution, one word heard as another; an insertion, a word the transcript adds; and a deletion, a word it drops. Add the three counts and divide by the number of words in the correct transcript.² In a worked example from the standard speech-recognition textbook, a recognizer missed a 13-word reference by six substitutions, three insertions, and one deletion. Its word error rate is (6 + 3 + 1) / 13, or 76.9 percent.² Because insertions count, the rate can pass 100 percent: a transcript can invent more wrong words than the reference had. Whisper's 2.5 on LibriSpeech means fewer than three words wrong per hundred on that clean audio. One caution travels with every such comparison in the paper. The numbers are scored after a text normalizer strips differences of formatting and punctuation. The rate reflects the words a model got right, not whether it wrote out a number or capitalized a name.¹

A single accuracy number like 2.5 says how the model did on one kind of audio. It says nothing about how far the number moves when the audio changes. That movement is what the paper measures, and scale is what buys it.

680,000 hours of whatever text came attached

Whisper was trained on 680,000 hours of audio scraped from the internet, each clip paired with whatever text happened to accompany it online.¹ This is weak supervision: instead of transcripts a linguist checked word by word, the labels are cheap, uneven text of unknown quality, taken as-is and in bulk. The one deliberate filter went the other way from cleaning. The authors built heuristics to detect and remove transcripts that looked machine-generated, so the model would learn from human writing rather than transcript-ese produced by earlier recognizers.¹ Of the total, 117,000 hours cover 96 languages other than English and 125,000 hours are audio in one language paired with English translation; about 65 percent of the training computation went to English transcription.¹

The scale is easier to feel against what came before. A gold-standard academic corpus runs about a thousand hours. LibriSpeech, the benchmark Whisper's 2.5 is measured on, is roughly 1,000 hours of read audiobook English.³ Whisper trained on several hundred times more audio than that, and paid for the volume in label quality. The bet is that hundreds of thousands of noisy hours teach a kind of generalization that a thousand pristine hours cannot.

The gap that opens off the benchmark

The bet is tested zero-shot: Whisper is scored on datasets it never trained on, so it cannot have memorized their quirks. That makes the comparison unusual. A conventional supervised model is normally tested on held-out audio drawn from the same collection it trained on, which flatters it, because test and training audio share the same recording conditions, accents, and style. Score both on someone else's data and the supervised model's home-field advantage disappears.¹

That is what the paper's central table shows. Whisper's largest model and a strong supervised model, wav2vec 2.0, tie on LibriSpeech clean. Averaged across thirteen other datasets Whisper never trained on, the supervised model's error climbs far more steeply than Whisper's.

This is what robustness means here. It is not a lower score on the familiar benchmark, where the two models are identical. It is a much smaller jump when the audio turns unfamiliar. Plotted against a human listener, Whisper's points sit near the line an ideally robust transcriber would follow, and reach into the range a person occupies. The supervised models fall away from that line as soon as the audio leaves their home distribution.¹ Scale of weak supervision bought that. It did not buy the lowest error on any one test.

Always a next word, even through silence

The design that holds up across strange audio has a cost written into it. Whisper is a decoder that reads audio and predicts the next word, one token after another, and it is always predicting a next token. OpenAI's model card states the failure and its cause plainly: predictions may include texts that are not actually spoken in the audio input, because the model combines predicting the next word with transcribing the audio it hears.⁴ The paper lists the same behavior among its known limitations: complete hallucination where the model will output a transcript entirely unrelated to the actual audio.¹ Feed it silence or noise and it does not fall silent. It emits the words that would plausibly come next.

How often depends entirely on the audio, and no single figure is the hallucination rate. In the one controlled measurement, a peer-reviewed study ran about 13,140 short utterance segments from recordings of speakers, many of them with aphasia, through the default Whisper service in 2023. Roughly 1.4 percent of the transcriptions contained an entire hallucinated phrase or sentence that was in no part of the audio.⁵ That number is specific to short clips, and it is not Whisper's rate in general. The same study found hallucination rose with the length of the non-vocal stretches in a clip. It fell hardest on speakers with aphasia, who pause longer. Of the hallucinations that did occur, 38 percent carried explicit harm, such as invented violence or a made-up claim of authority.⁵

On messier, longer, real-world audio, practitioners report far more. The Associated Press collected uncontrolled counts from people running the model on their own recordings: a researcher who found hallucinations in eight of ten public-meeting transcripts, an engineer who found them in about half of more than 100 hours, a developer who found them in nearly all of 26,000.⁶ These are tallies by individuals on their own audio, not measured rates on a defined sample, and they cannot be read as one. What they fix is the setting. Nabla, a medical-scribe product built on Whisper, reports use by more than 30,000 clinicians across 40 health systems on an estimated 7 million visits. It erases the original audio for data-safety reasons, so no recording survives to check the transcript against.⁶

The paper and the deployment do not actually contradict each other. They measure different things. Robustness in the paper is average error across many distributions, a claim about how a large batch of transcripts holds up on average. Verbatim medical and legal transcription needs the opposite guarantee: no invented sentence in any single record, the one place average error says nothing. OpenAI's own model card asked for this restraint at release, cautioning against Whisper in high-risk domains like decision-making contexts.⁴ The product was marketed as near human accuracy and put to exactly that use.