Whisper answers silence with a sentence no one spoke

A sentence the doctor never heard

In a growing number of clinics, a piece of software listens to the visit and drafts the note. One of the most widely used of these tools comes from a company called Nabla and is built on OpenAI's Whisper. The Associated Press reported in 2024 that more than 30,000 clinicians and 40 health systems, including Children's Hospital Los Angeles, had adopted it.¹ To protect patient privacy, the tool deletes the original recording by default once the transcript exists.¹ Nabla says clinicians can opt in to keeping the audio, but deletion is the default.²

That default matters because of what these systems sometimes do. A University of Michigan researcher who examined the output found invented text in eight of ten transcriptions. Another developer found it in nearly all of 26,000.¹ The inventions are not smudged versions of real speech. In one case Whisper took a plain spoken sentence and added a clause about "two other girls and one lady, um, which were Black," people who were never mentioned. In another it prescribed a drug that does not exist, "hyperactivated antibiotics."¹ When the recording is already gone, nothing is left to show the line was never said.

How an invention differs from a mishearing

This is not the mistake people expect from a transcription tool. The expected mistake is a mishearing: the audio says "recognize speech" and the transcript reads "wreck a nice beach." The words come out wrong, but they stay close to the sound. The invented sentence is a different kind of error. Researchers who study it define a transcription hallucination as output that is fluent and grammatical yet unrelated in meaning to what was said. They hold it apart from a mishearing, which stays near the real words in both sound and sense.³ A mishearing is the system getting the sound wrong. A hallucination is the system writing text that no sound produced.

The distinction has a sharp practical edge: Word Error Rate, the standard accuracy score for transcription, cannot see it. It counts the fraction of words a transcript gets wrong against a reference. A model that hallucinates and one that does not can post the same overall error rate, which is why the behavior went unmeasured for so long.³

How often does it happen? There is no single answer.

These figures do not contradict each other. The 1.4% is the average across interviews recorded for aphasia research, audio chosen precisely because aphasia produces the long pauses that were expected to provoke the behavior.⁴ The developer counts come from everyday recordings full of the pauses and background sound that set it off.¹ Nabla runs a version it says it tuned to suppress the behavior, over visits it does not describe in detail.² The rate depends on what the system is fed.

The two machines inside a transcriber

Why nothing in the audio can still become words comes down to how the tool is built. Whisper is an encoder-decoder model, and the two halves do different jobs.⁵ The encoder is the listener. It takes the sound, converted into a picture of which frequencies are present at each instant, and turns it into a compact set of numbers that captures what is in the audio.⁵ The decoder is the writer. It reads what the encoder produced and emits the transcript one word-piece at a time, each choice shaped by the audio and by the words it has already written.⁵

OpenAI describes the decoder in its own paper as "an audio-conditional language model."⁵ A language model is a system trained to predict likely text: give it some words and it produces the word most likely to come next. That is the same next-token prediction that drives a chatbot. Whisper's decoder is one of those, with the audio wired in as an extra input. It was trained on 680,000 hours of recordings paired with their transcripts, and it learned the shape of well-formed language along the way.⁵ Its job is not to report sounds. Its job is to produce probable text, conditioned on the sound when there is sound to condition on.

Why silence comes out as a sentence

When the audio carries clear speech, the decoder has a strong signal from the encoder, and it writes what it hears. When the audio carries silence, a pause, or background noise, the decoder has little or nothing from the encoder to go on. It does not stop. A language model that has run out of input still produces its most probable continuation, and for a system trained on ordinary transcripts, the most probable continuation is an ordinary, grammatical sentence.⁵ Asked to transcribe silence, the decoder does not go quiet. It writes the most probable words, and the most probable words are a sentence.

It helps to compare an older kind of speech recognizer. Systems from before this design worked closer to sound in, word out: they matched stretches of audio to the words those stretches most resembled. They had no way to spin a fluent sentence out of nothing, because they held no model of fluent language to fall back on. Give one silence and it gave silence back. Whisper's decoder cannot answer with silence in that way, because answering with probable text is the only thing it does.

Which probable text? Often something the model saw many times in training over exactly this kind of low-information audio. Whisper learned in part from a large body of video subtitles, and subtitles routinely run credits and boilerplate over the silent end of a clip. So the model came to tie silence to those phrases and reproduces them when it meets silence again.⁶

The trigger is low-information audio in general, not digital silence in particular. A separate study produced the same fluent, unrelated invention on purpose by injecting one second of random noise at the start of an utterance; the same noise spread across the whole utterance instead produced ordinary mishearings.³ Nor is the invention the ordinary randomness that sometimes makes a model creative: Koenecke's team ran Whisper at its least random setting, and it still hallucinated.⁴ That rules out chance as the cause. The invention is the model doing exactly what it was built to do, on an input that gives it nothing to anchor to.

The tendency no decoding rule removes

Two parts of this are settled engineering. The decoder is a generative language model, by OpenAI's own description, and a language model given empty input still produces output.⁵ OpenAI's model card says as much in plain terms. The predictions "may include texts that are not actually spoken in the audio input," a consequence of the model combining language prediction with transcription. The card recommends against using Whisper in high-risk settings such as decision-making.⁷

What is not settled is how far the tendency can be reduced. The fixes that exist work around the decoder rather than change it. Whisper can estimate when a stretch is probably non-speech and decline to transcribe it.⁵ OpenAI's own repository later added an option that skips a silent stretch and re-runs it whenever a hallucination is suspected.⁸ These help. After OpenAI updated Whisper in late 2023, Koenecke's team re-ran the 187 segments that had reliably hallucinated and found only 12 still did.⁴ Yet the same team reports that Whisper still hallucinates, and the silence-skipping option was added precisely because the problem did not go away.⁸ The fixes lower how often it happens without touching why it happens, because the reason is the decoder's design.

Two questions stay open even for the people who build these systems. One is exactly which inputs set off an invention rather than a correct blank. The other is how much of the tendency better training or smarter decoding can ever remove. On the same 187 segments, five competing services from Google, Amazon, Microsoft and others produced no comparable hallucinations.⁴ As of 2023 and 2024, then, the behavior was particular to Whisper and not forced by the encoder-decoder design itself. That is a single comparison at one moment, and a system built the same way could still show it. What is not in doubt is the mechanism: when the stage that writes the words is a language model, an input with nothing to say still comes out as words.