Whisper is the model that made speech recognition boring, in the best way. Before it, ASR was a pile of fragile pipelines: VAD, then a frame-level acoustic model, then a language model, then a decoder, each a separate moving part. Whisper is one model, end to end, and it just works.
Understanding its architecture is the key to serving it well.
The input: Mel spectrograms
Whisper doesn't take raw audio. It converts audio to a Mel spectrogram: a 2D representation where the x-axis is time, the y-axis is frequency (on the Mel scale, which matches human pitch perception), and the color is energy. Whisper uses 80 Mel bins, 30 seconds of audio at a time, which is a 3000×80 grid.
This is the input to the encoder. It's a vision-style input, and Whisper treats it like one: a convolutional stem (2 conv layers) patchifies the spectrogram, then a positional embedding is added, and the result feeds a standard transformer encoder.
The encoder: listen
The encoder is a stack of transformer blocks that process the entire spectrogram at once. This is the "listening" phase: it produces a sequence of hidden states that summarize what was heard. For a 30-second clip, that's 1500 tokens worth of encoder states.
Critically, the encoder runs once per audio chunk, not once per output token. That's the asymmetry that matters for serving: the encoder is a prefill-like operation, compute-bound, and it happens before any text comes out.
The decoder: transcribe
The decoder is a standard autoregressive transformer, but with cross-attention into the encoder states. It generates text one token at a time, and at each step it attends to the encoder's representation of the audio.
This is where the KV cache comes in. The decoder's self-attention KV cache grows with each generated token, and the cross-attention keys and values (from the encoder) are fixed. The decoder is memory-bound, just like any LLM decode, which means the same serving tricks apply: continuous batching, KV cache management, speculative decoding.
What this means for serving
- Batching matters. Whisper is a transformer; it batches. A batch of 8 clips processes almost as fast as 1, because the encoder is compute-bound and the GPU is underutilized at batch 1.
- Long audio is chunked. Whisper processes 30 seconds at a time. Longer files get split, transcribed, and stitched, with a language model pass to clean up the seams.
- The decoder is the bottleneck. The encoder is fast; the decoder is where the seconds go. Optimizing the decoder is optimizing the whole pipeline.
Whisper is a translator. The encoder reads the whole page (the audio) at once. The decoder writes the translation one word at a time, looking back at the page for reference.
The production numbers
In production you don't run the reference PyTorch implementation. You run faster-whisper (CTranslate2) or whisper.cpp. The wins: 4-8x speedup from INT8 quantization and fused kernels, ~1.5GB VRAM for large-v3 INT8 (vs ~3GB FP16), and ~150-200x realtime throughput on an A100. That means a minute of audio transcribes in about a third of a second. The decoder is the bottleneck, so batched beam search across segments is where the throughput lives.
One production trap: Whisper hallucinates on silence and music. The fix is a VAD frontend (Silero VAD) that only feeds speech segments to the model. Every serious deployment has this.
Sources
- Robust Speech Recognition via Large-Scale Weak Supervision: the Whisper paper.
- OpenAI Whisper repo: reference implementation, decoding strategy, temperature fallback.
- faster-whisper: CTranslate2-based, 4-8x speedup, INT8, batched beam search, integrated VAD.
The takeaway
Whisper made ASR boring by making it one model. The serving lesson is that it's still a transformer, with all the same levers: batch it, cache it, and optimize the decoder.