Debugging Whisper Hallucinations: Why Your 9-Minute Voice Message Became "12 30" On Repeat

I got a 9-minute Telegram voice message. My self-hosted Whisper instance transcribed it as "12 30. 12 30. 12 30." repeated for several pages. Then it switched to "I gave him a report on the best schools to apply for" on loop. None of that was in the audio.

This is the story of figuring out why, and the fix that made long-form voice transcription actually reliable.

The Setup

Quick context if you haven't read the earlier post: I run Speaches (the maintained fork of faster-whisper-server) on an Alienware R8 with an RTX 2080, served over Tailscale. It exposes an OpenAI-compatible /v1/audio/transcriptions endpoint, and a small bash bridge script on my Mac sends audio files to it whenever a Telegram voice note comes in.

The bridge script was simple:

curl -F "model=Systran/faster-whisper-large-v3" \
     -F "response_format=json" \
     -F "file=@$audio" \
     http://100.64.0.11:9000/v1/audio/transcriptions

No chunking. No VAD. Just throw the whole file at Whisper and read back the text. Worked fine for 30-second voice notes. Worked fine for 2-minute ones. Completely fell apart at 9 minutes and 13 seconds (553 seconds of audio).

Why Whisper Hallucinates on Long Audio

Whisper was trained on 30-second audio clips. That's not a suggestion. It's a hard architectural constraint.

The model processes audio in windows. When you hand it a 553-second file in a single inference pass, the attention mechanism has to work across a sequence that's roughly 18x longer than what it was trained on. After a few minutes, the attention degenerates. The model latches onto high-confidence token patterns and starts repeating them in a loop, each repetition reinforcing the next. It never recovers within the same pass.

The specific hallucination ("12 30") was probably a fragment from somewhere in the actual audio that the model fixated on. The second pattern ("I gave him a report on the best schools to apply for") is a known Whisper training data artifact. People have reported the same phrase showing up in completely unrelated audio. It's baked into the weights.

Attempt 1: VAD Filter

Speaches supports a vad_filter parameter. Voice Activity Detection. I added it:

curl -F "model=Systran/faster-whisper-large-v3" \
     -F "response_format=json" \
     -F "vad_filter=true" \
     -F "file=@$audio" \
     http://100.64.0.11:9000/v1/audio/transcriptions

This helped. The transcription got further into the audio, producing correct output through about 5 minutes and 48 seconds. Then the hallucination loop kicked in again.

The problem: VAD segments silence out of the audio. It finds the gaps between speech and splits on those boundaries. But it doesn't actually chunk the audio into Whisper-sized windows. If someone talks for 6 minutes straight without a long pause, VAD gives Whisper a 6-minute segment and the same attention degradation happens. VAD is necessary but not sufficient.

Attempt 2: Verbose JSON (Finding the Smoking Gun)

I switched the response format to verbose_json to get per-segment metadata:

curl -F "model=Systran/faster-whisper-large-v3" \
     -F "response_format=verbose_json" \
     -F "vad_filter=true" \
     -F "file=@$audio" \
     http://100.64.0.11:9000/v1/audio/transcriptions

This was the breakthrough. verbose_json returns each segment with timestamps, token probabilities, and a compression_ratio field. Here's what I saw:

Segments	Time Range	compression_ratio	Status
1-8	0:00 to 5:47	1.40 - 1.47	Healthy
9+	5:48 onward	12.00	Hallucinating

That compression_ratio jump is the smoking gun.

Normal speech compresses at about 1.3x to 1.8x. That's because real language has variety: different words, different sentence structures, natural variation. When the model hallucinates repetitive text ("12 30. 12 30. 12 30."), the output compresses at 10x to 12x because it's the same tokens over and over. Whisper's own codebase uses a threshold around 2.4 internally to detect this. The metric was sitting right there in the API response. I just wasn't asking for it.

Attempt 3: Brute-Force Chunking

Before building anything smart, I tested the dumb version. Split the audio into 30-second chunks with ffmpeg and transcribe each one independently:

ffmpeg -i input.ogg -f segment -segment_time 30 \
       -c copy chunk_%03d.ogg

Then loop through the chunks, POST each one, concatenate the text.

Hallucination: gone. Every chunk is short enough for Whisper to handle cleanly. But two new problems showed up:

Boundary artifacts. Sentences get cut mid-word at the 30-second mark. "I was saying that the configura-" in one chunk, "-tion file was wrong" in the next. Readable but ugly.
Silence hallucinations. Chunks that landed on a natural pause sometimes contained mostly silence. Whisper doesn't like silence. Instead of returning an empty string, it hallucinates a polite "Thank you." or "Thanks for watching." Another training data artifact.

Still, this confirmed the core theory. Chunking fixes the attention degradation. The remaining issues are edge cases.

The Fix: Hybrid Chunking with Hallucination Filtering

The final version of the bridge script does this:

Short audio (35 seconds or less): single-shot transcription with VAD enabled. Original behavior. No chunking needed.
Long audio (over 35 seconds): split into 30-second chunks with ffmpeg. Transcribe each chunk individually using verbose_json format. Filter out any segment where compression_ratio > 2.4. Concatenate the surviving segments.

The compression ratio filter catches two failure modes at once. It catches the big hallucination loops (compression_ratio of 10+), and it catches the smaller "Thank you." artifacts on silent chunks (those also compress high because they're short repetitive phrases against a backdrop of nothing).

The 9-minute voice message that was producing pages of "12 30" garbage now transcribes cleanly in about 45 seconds.

What I Learned

Whisper's 30-second window is architectural, not optional. The model was trained on 30-second clips. Sending it longer audio doesn't make it try harder. It makes it break in a specific, deterministic way. Once the attention mechanism starts looping, it won't recover in the same inference pass.

VAD is not a replacement for chunking. vad_filter=true in Speaches (and faster-whisper generally) detects silence for segmentation purposes. It doesn't enforce a maximum segment length. If there's continuous speech for 6 minutes, VAD will hand Whisper a 6-minute segment and the same problem happens.

compression_ratio is the best hallucination detector. It's right there in the verbose_json output. Healthy speech: 1.3 to 1.8. Hallucinated loops: 10+. Whisper's own codebase uses 2.4 as the threshold. If you're doing any kind of automated transcription, check this field.

The json format hides the diagnostics. The default response_format=json gives you a text string and nothing else. Switch to verbose_json and you get per-segment timestamps, token probabilities, and the compression ratio. If I'd started with verbose output I would have found the problem in 10 minutes instead of an hour.

Self-hosted Whisper gives you these knobs. Cloud transcription APIs (OpenAI's, Google's, etc.) abstract away the per-segment metrics. You get text back and that's it. If the text is wrong, you have no way to diagnose why. Running faster-whisper locally means you can see exactly where the model broke and build around it.

The hallucination pattern is deterministic, which is the one nice thing about it. Same audio, same garbage, every time. That makes it testable. And testable means fixable.

Enjoyed this post?

Get notified when I publish something new. No spam, unsubscribe anytime.

← dxFeed vs Schwab Market Data Latency The AI That Ate 223 GB →