I reach for Whisper by default, and every time I have had to serve speech with a latency budget I have ended up somewhere else. It took me longer than it should have to see that this is not a tuning problem. The two architectures answer different questions, and the question they answer is baked into their shape.
What each one is
Whisper is an encoder-decoder. A mel spectrogram goes in, the encoder processes it, and an autoregressive decoder attends over the encoder output to produce text. It was trained on an enormous, weakly supervised, multilingual corpus, which is why it is so robust to accent, noise and domain, and why it remains the sensible default for batch work.
Conformer interleaves convolution with self-attention: convolution captures local acoustic structure, attention carries the long range. Paired with a transducer decoder it becomes Conformer-Transducer, and FastConformer adds aggressive downsampling on top to cut the cost.
The thing that actually separates them
Whisper processes fixed thirty-second windows. That is not an implementation choice you can flag your way out of, it is what the model was trained to consume.
The consequence is brutal for streaming. To emit a partial transcript every second you pad the audio out to thirty seconds and run the encoder again. A minute of speech that costs one encoder pass offline costs on the order of sixty passes when streamed at one-second granularity. You are not paying for more audio, you are paying to look at the same audio repeatedly.
A transducer has no such window. It consumes frames as they arrive and emits tokens when it is ready, so partial results are a property of the architecture rather than a trick played on top of it. That is the whole difference, and it is why the streaming benchmarks are populated by transducer models rather than by Whisper variants.
How I would choose
- Offline transcription, files, batch. Whisper, without much hesitation. The robustness is real and the thirty-second window costs you nothing when nobody is waiting.
- Interactive, sub-second, someone is listening. A transducer. Conformer-Transducer or one of the streaming-specialised models, quantised, on whatever hardware is nearest the user.
- Speech-to-speech. The ASR latency is only the first third of the loop, and it is the third with the cheapest fix. Getting Whisper out of the streaming path is usually the largest single win available.
What surprised me most reading the current benchmarks is where the good streaming numbers now live. The strongest real-time English results are coming from compact models running quantised on CPU, faster than real time, with sub-second algorithmic latency. For a class of product that removes the GPU from the speech path entirely, which changes the deployment argument more than any accuracy delta would.
Whisper's thirty-second window is not a limitation to engineer around. It is the model telling you it was built for a different job.
Sources
- Robust speech recognition via large-scale weak supervision: the Whisper paper, including the thirty-second window and the training corpus.
- Conformer: convolution-augmented transformers for speech recognition.
- Turning Whisper into a real-time transcription system: what it costs to stream a model that was not designed to stream.
- Pushing the limits of on-device streaming ASR: the compact, low-latency end of the current benchmarks.
- Whisper, dissected: the encoder-decoder in more detail.