Whisper ASR: single-chunk and long-file

Whisper was my introduction to non-LLM inference, and it taught me something important: the inference challenges for encoder-decoder models are fundamentally different from decoder-only LLMs. There is no KV cache that grows unboundedly. There is no continuous batching over variable-length generations. Instead, you get a fixed-size encoder bottleneck and a chunking problem that sounds simple but has subtle failure modes.

The architecture in 60 seconds

Whisper is an encoder-decoder transformer trained on 680,000 hours of multilingual audio. The pipeline has three stages:

• Audio preprocessing. Raw audio is resampled to 16 kHz, converted to an 80-channel log-mel spectrogram with a 25 ms window and 10 ms hop. This produces a 2D feature map: 80 frequency bins by T time frames, where T = audio_seconds * 100.
• Encoder. The spectrogram is processed by a stack of transformer layers with two initial convolution layers that downsample by 2x. The encoder expects exactly 3000 time frames, corresponding to 30 seconds of audio. Output: a sequence of 1500 encoder hidden states.
• Decoder. An autoregressive transformer that generates text tokens conditioned on the encoder output. It uses cross-attention to attend to the encoder states and produces tokens until it emits an end-of-text token.

The 30-second constraint is baked into the architecture. The encoder's positional embeddings are learned for exactly 3000 frames. You cannot feed it 60 seconds of audio and hope for the best. This is the root of the long-file problem.

Single-chunk inference: the happy path

For audio under 30 seconds, inference is straightforward:

import whisper
import torch

model = whisper.load_model("large-v3")

# Load and preprocess audio
audio = whisper.load_audio("short_clip.wav")
audio = whisper.pad_or_trim(audio)  # Pad/trim to exactly 30s

# Compute mel spectrogram
mel = whisper.log_mel_spectrogram(audio).to(model.device)

# Decode with greedy search
options = whisper.DecodingOptions(language="en", fp16=True)
result = whisper.decode(model, mel, options)
print(result.text)

The pad_or_trim function is doing the critical work: it ensures the audio is exactly 480,000 samples (30 seconds at 16 kHz). Shorter clips get zero-padded. The encoder processes the full 30-second window regardless, which means a 2-second clip wastes about 93% of the encoder compute on silence.

The long-file problem

Real-world audio is rarely 30 seconds. Meeting recordings run for an hour. Podcasts go for two. Customer service calls can be anywhere from 30 seconds to 45 minutes. Whisper needs a chunking strategy, and there are two main approaches.

Approach 1: sequential chunking with timestamps

The simplest method splits audio into 30-second chunks and transcribes each one sequentially. Whisper's official transcribe() function does this, but with a clever twist: it uses the model's predicted timestamps to find the actual end of speech within each chunk, then seeks forward to that point rather than blindly advancing by 30 seconds.

# Whisper's transcribe() uses a seek-based approach:
# 1. Feed 30s chunk starting at current position
# 2. Decode with timestamp tokens enabled
# 3. Find the last timestamp token in the output
# 4. Advance seek position to that timestamp
# 5. Repeat until end of audio

# The timestamp tokens are special tokens like <|0.00|>, <|0.02|>, ...
# They indicate when each word occurs within the 30s window
# This lets the model "consume" only the speech it transcribed

This works well but has two failure modes:

• Hallucination on silence. If a chunk is mostly silence (like a pause in a meeting), Whisper sometimes hallucinates repeated phrases or garbage text. The model was trained on audio that always contains speech, so it tries to find speech even when there is none.
• Boundary artifacts. When speech is split mid-word or mid-sentence at a chunk boundary, the decoder may produce garbled output at the transition. The timestamp-based seeking helps, but does not eliminate this entirely.

Approach 2: sliding window with overlap

A more robust approach uses overlapping windows and merges the results. HuggingFace Transformers implements this as the "chunked" long-form algorithm:

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

model_id = "openai/whisper-large-v3"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch.float16
).to("cuda")
processor = AutoProcessor.from_pretrained(model_id)

# Chunked transcription with overlap
result = model.generate(
    input_features,
    return_timestamps=True,
    chunk_length_s=30,        # 30-second chunks
    stride_length_s=[4, 2],   # 4s left overlap, 2s right overlap
)

# The overlapping regions are used to align and merge
# transcriptions across chunk boundaries

The overlap regions serve as anchors: the model transcribes the same audio twice (once at the end of one chunk, once at the start of the next), and a merging algorithm uses the overlapping text to find the best splice point. This dramatically reduces boundary artifacts.

Batched inference for throughput

When you need to transcribe many files, the key optimization is batching across the encoder. The encoder processes fixed-size inputs (3000 frames), so batching is trivial compared to LLMs: stack N spectrograms into a tensor of shape [N, 80, 3000] and run one forward pass.

The decoder is harder to batch because different chunks produce different-length transcripts. The approaches mirror LLM batching:

• Pad to max length. Simple but wasteful if one chunk in the batch produces 5 tokens and another produces 200.
• Greedy with early stopping. Remove sequences from the batch as they emit the end token, compacting the batch at each step.
• Static batching with bucketing. Group chunks that are likely to produce similar-length outputs (for example, by audio energy or voice activity detection).

In practice, the encoder is the throughput bottleneck for batch processing. On an A100, the encoder processes about 50 to 80 chunks per second for Whisper large-v3 in FP16 with a batch size of 32. The decoder adds roughly 20 to 40% more time on top.

Optimization techniques

Several optimizations are particularly effective for Whisper:

• FP16 and BF16. The encoder is compute-bound, so half precision gives nearly 2x speedup on Tensor Cores with negligible quality impact.
• Faster Whisper (CTranslate2). The CTranslate2 engine compiles the model into an optimized graph with INT8 quantization, often achieving 4x speedup over the original PyTorch implementation. The faster-whisper library wraps this into a clean API.
• Speculative decoding. Use a smaller Whisper model (tiny or base) as a draft model and the large model as the verifier. This works because most of the transcript is straightforward, and the small model gets it right 80 to 90% of the time.
• Voice activity detection (VAD) preprocessing. Run Silero VAD or a similar detector before Whisper to identify speech segments. Skip silent regions entirely instead of feeding them to the model. This saves encoder compute and eliminates hallucination on silence.

# VAD + Whisper pipeline (pseudocode)
from silero_vad import get_speech_timestamps, load_silero_vad

vad_model = load_silero_vad()
speech_segments = get_speech_timestamps(audio, vad_model)

# Only transcribe segments with actual speech
for segment in speech_segments:
    chunk = audio[segment['start']:segment['end']]
    chunk = whisper.pad_or_trim(chunk)
    mel = whisper.log_mel_spectrogram(chunk)
    result = whisper.decode(model, mel, options)

Where Whisper fits in the modalities picture

Whisper's architecture is an interesting bridge between the pure encoder models we looked at yesterday with embeddings and the pure decoder models that dominate LLM inference. The encoder half behaves like an embedding model: fixed input size, compute-bound, easy to batch. The decoder half behaves like a small LLM: autoregressive, variable-length output, memory-bound at small batch sizes.

This duality means the optimization toolkit is broader. You can apply encoder tricks (sequence packing, INT8 compute) and decoder tricks (speculative decoding, KV cache optimization) to different parts of the same model. It is a good exercise in thinking about where the bottleneck actually is, rather than applying optimizations uniformly.

Tomorrow: TTS and streaming speech synthesis, where we flip the direction and generate audio from text, with real-time latency constraints that make LLM TTFT look generous.