Skand Tandon
Modalities

Speech-to-speech end-to-end latency

A voice assistant must respond within 500 ms to feel natural. That budget spans ASR, LLM inference, and TTS, each with its own latency profile. Breaking down where the milliseconds go reveals what to optimize and what to pipeline.

2025-12-26 modalities 7 min read

Human conversation has a natural turn-taking latency of about 200 to 300 ms. Anything above 500 ms feels sluggish. Anything above 1 second feels broken. When you are building a speech-to-speech system, that 500 ms budget is your entire world, and it is shockingly easy to blow through it.

After working through Whisper ASR and streaming TTS individually, today I am stitching the full pipeline together and measuring where every millisecond goes.

The cascaded pipeline

The traditional architecture for speech-to-speech is a three-stage cascade:

Each stage adds latency, and the stages are sequential: the LLM cannot start until ASR produces text, and TTS cannot start until the LLM produces text. Or can it? The trick to hitting the latency budget is finding every opportunity to overlap these stages.

The latency breakdown

Let me walk through a concrete example. The user speaks a 3-second utterance and we need to start playing audio back as quickly as possible.

# Stage 1: ASR
audio_duration = 3.0  # seconds
# Whisper processes in 30-second chunks, but we can use streaming:
# - Endpointer detects end of speech: ~200 ms after silence
# - Whisper encoder (30s mel spectrogram -> features): ~30 ms on A100
# - Whisper decoder (autoregressive text generation): ~50 ms for short utterance
asr_latency = 200 + 30 + 50  # ~280 ms after user stops speaking

# Stage 2: LLM
# - Prefill the ASR output + system prompt (~200 tokens): ~15 ms
# - Generate first token (TTFT): ~15 ms
# - We only need enough tokens to start TTS (~20 tokens): 20 * 20ms = ~400 ms
llm_latency_to_first_chunk = 15 + 15 + 400  # ~430 ms (for 20 tokens)

# Stage 3: TTS
# - Encode 20 text tokens to audio: ~50-100 ms for a streaming TTS
tts_latency = 75  # ms for first audio chunk

# Total without pipelining:
total_sequential = 280 + 430 + 75  # ~785 ms -- too slow

785 ms. Already above our 500 ms target, and this assumes fast models on good hardware. We need to pipeline.

Pipelining: overlap everything

The key insight is that the LLM generates text token by token, and TTS can start synthesizing audio as soon as it has a chunk of text (a phrase or even a few words). You do not need the complete LLM response before starting TTS.

Similarly, streaming ASR models can emit partial transcripts before the user finishes speaking. With a streaming Conformer or Whisper with chunked inference, you can start LLM prefill while the user is still talking.

# Pipelined timing:
# t=0: User starts speaking
# t=2800ms: User stops (endpointer fires at t=3200ms)
# t=3200ms: ASR emits final transcript
# t=3215ms: LLM prefill complete, first token generated
# t=3615ms: LLM has generated ~20 tokens (enough for TTS)
# t=3690ms: TTS emits first audio chunk

# But with streaming ASR, we can do better:
# t=2000ms: Streaming ASR emits partial transcript of first clause
# t=2015ms: LLM starts prefilling partial transcript
# t=2030ms: LLM starts generating tokens
# t=2430ms: LLM has 20 tokens, TTS starts
# t=2505ms: First audio plays

# Perceived latency (from user stops speaking to audio starts):
# Sequential: 785 ms
# Pipelined: ~505 ms (from end of speech at t=3000ms to audio at t=3505ms)
# With streaming ASR: ~305 ms (LLM got a head start)

Streaming ASR is the single biggest latency win. By feeding partial transcripts to the LLM before the user finishes speaking, you can overlap ASR and LLM inference and save 200+ ms.

The 200 ms rule

Human perception research shows that response latency below 200 ms feels instantaneous, 200 to 500 ms feels responsive, and above 500 ms feels slow. Your target should be first audio byte within 300 to 500 ms of the user stopping speech. Every component in the chain needs to be optimized for latency, not throughput.

End-to-end models: skipping the cascade

An emerging alternative is to skip the cascade entirely. Models like Moshi (Kyutai), GPT-4o's voice mode, and Gazelle process audio tokens directly and output audio tokens, with no intermediate text representation. The architecture looks like:

The latency advantage is significant: no ASR decoder, no text tokenization, no TTS text encoder. The model goes directly from audio features to audio features. Moshi achieves a theoretical latency of 160 ms (one frame of audio input to one frame of audio output) plus the codec encoding/decoding overhead of about 80 ms.

The tradeoff is that these models are harder to debug (no intermediate text to inspect), harder to control (no system prompt in the traditional sense), and currently less capable than cascaded systems with large LLMs.

Where the milliseconds hide

In my experience profiling speech-to-speech systems, the biggest latency surprises come from:

Optimization priorities

If I were building a speech-to-speech system from scratch, my priority order would be:

  1. Streaming ASR to overlap with LLM inference. Biggest single win.
  2. Streaming TTS with small chunk sizes (5 to 10 words per chunk).
  3. LLM TTFT optimization because time-to-first-token directly adds to perceived latency. Use a small, fast model (7B or smaller) rather than a large one.
  4. Co-location of all three models, ideally on the same GPU or same machine.
  5. Aggressive endpointing with a learned model rather than a fixed silence threshold.

In speech-to-speech, latency is the product, not a metric. Users do not notice throughput. They notice the pause before the assistant speaks. Every millisecond you save is directly perceptible.

Next up: long context inference with RoPE scaling and context parallelism, where we tackle the challenge of serving models with 128K+ context windows without blowing up memory or latency.