Transcripts that invent sentences

A transcript came back with several sentences of fluent, grammatical text in a language nobody in the video speaks. The confidence scores were perfect. The audio at those timestamps was instrumental music. This is the failure mode nobody warns you about when they tell you speech recognition is solved: not errors you can see, but invented content delivered with total certainty.

What it looks like

A talking-head clip with a music bed. The speech transcribes fine. Then, in a passage where the speaker stops and the music carries, the transcript contains a handful of complete sentences in an unrelated language — well-formed, punctuated, and scored at the top of the confidence range.

Downstream this is worse than a mis-heard word, because everything trusts the transcript: the captions render it, the hook generator reads it, the emphasis picker weighs it. A single hallucinated passage can pull the entire clip’s summary toward a topic that was never discussed.

Why models do this

• The model’s job is to emit text. Given audio, it produces the most likely transcription. “Nothing was said here” is available as an output but competes against every plausible sentence, and plausible sentences are numerous.
• Music is speech-shaped. Rhythm, pitch contour and harmonic structure overlap with the features that indicate speech. A sung phrase or a melodic instrument line can activate the same machinery.
• Multilingual decoding widens the search. The more languages available, the more plausible completions exist for ambiguous audio — and a fluent sentence in some language will usually beat a garbled one in the expected language.
• Confidence is self-assessment. It measures certainty about the decoding path, not whether the input contained speech. A confidently-decoded fabrication scores high by construction.

The filter that looks obvious and is wrong

The tempting rule: if the clip is mostly English and a few segments came back as another language, drop them. It removes the hallucinations. It also removes a large amount of genuinely spoken content, because real speech does this constantly:

• Loanwords and names that the model tags as their language of origin.
• Code-switching — Hinglish, Spanglish, and every other natural mix — where the switch is mid-sentence and completely real. (More on that in the Hindi and Hinglish article.)
• A quoted phrase, a brand name, a piece of terminology.

Deleting those produces a transcript with holes exactly where the interesting words were — trading a visible failure for an invisible one, which is a bad trade.

The signal that works: isolation, not identity

What separates a hallucination from a real code-switch isn’t the language. It’s where it sits in time.

• Real minority-language words are embedded. They sit inside a sentence, surrounded closely by majority-language words. The speaker didn’t pause to say them.
• Hallucinations sit on islands. They appear in passages where the speaker isn’t talking — which is precisely why the model had room to invent — so they’re separated from real speech by silence on both sides.

So the rule is a conjunction: a segment is suspect when it is both a minority language for the clip and temporally isolated by a meaningful gap on either side. Either condition alone produces false positives; together they’re specific. That gap threshold is the tuning knob, and erring toward a longer one is the safer direction — a surviving hallucination is a visible bug someone can report, while a deleted real word is a silent one.

Two related failures worth knowing

• The confident empty result. Language detection guesses a language nobody is speaking, and the service returns a successful response with zero utterances. Treating “HTTP 200” as success means banking an empty transcript and reporting no error at all. Zero words from a clip with audio is a failure and should be handled as one.
• Sending the wrong thing. Uploading an entire video file when the service only needs audio wastes time and can fail on large clips. Extracting the audio first is both faster and, because the encode is chosen for speech rather than for vision, generally more accurate.

The takeaway for anyone building on ASR

1. Never treat confidence as truth. It answers a different question than the one you have.
2. Find a signal the model doesn’t control — timing, energy in the audio, agreement between passes — and filter on that.
3. Prefer conjunctions to single heuristics. One signal deletes real data; two signals in agreement are specific.
4. Make failure loud. A zero-word transcript, a language flip, an isolated island — all should surface, not silently pass through into everything downstream.

Quick answers