Skand Tandon
Modalities

TTS: streaming real-time speech

Text-to-speech feels like the inverse of Whisper, but the inference constraints are entirely different. When a voice agent needs to respond in under 500 ms, every component of the TTS pipeline becomes latency-critical: text normalization, acoustic model, vocoder, and the streaming protocol.

2025-12-21 modalities 8 min read

TTS inference humbled me. I came from LLM serving, where a time-to-first-token of 200 ms is considered good and 500 ms is acceptable. In voice agents, the user is waiting for the AI to "speak," and the entire pipeline from receiving text to the first audio chunk hitting the speaker needs to be under 300 to 500 ms. That is the time-to-first-audio (TTFA), and it is the metric that defines whether a voice agent feels responsive or awkward.

The challenge is that TTS is not one model. It is a pipeline, and every stage adds latency.

The modern TTS pipeline

Most production TTS systems have three stages, though some newer models collapse these:

For streaming, each stage needs to operate incrementally, producing output before it has consumed all its input. This is where things get architecturally interesting.

Streaming the acoustic model

The acoustic model is typically the latency bottleneck. Non-autoregressive models like VITS and Matcha-TTS are fast but traditionally require the full input text before producing any audio. Autoregressive models like VALL-E and XTTS generate audio tokens left-to-right, which naturally supports streaming but is slower overall.

The practical approach for streaming with non-autoregressive models is sentence-level chunking:

# Sentence-level streaming TTS
import re

def stream_tts(text, model, vocoder):
    # Split on sentence boundaries
    sentences = re.split(r'(?<=[.!?])\s+', text)

    for sentence in sentences:
        # Generate mel spectrogram for this sentence
        mel = model.synthesize(sentence)

        # Convert to waveform
        audio_chunk = vocoder(mel)

        # Yield immediately, do not wait for next sentence
        yield audio_chunk

This is simple but effective. The first sentence is typically short (5 to 15 words), so the model can produce it quickly. While the client plays the first chunk, the server generates the second. As long as generation is faster than playback (faster than real-time), the listener hears continuous speech.

The real-time factor

TTS performance is measured by the real-time factor (RTF): the ratio of generation time to audio duration. An RTF of 0.5 means you generate audio twice as fast as it plays back. For streaming, you need RTF well below 1.0 to build up a buffer. An RTF of 0.1 to 0.3 is typical for optimized GPU inference with modern models.

The vocoder bottleneck

Vocoders are surprisingly compute-intensive. HiFi-GAN generates audio at 22,050 Hz sample by sample (or in small frames). For one second of audio, that is 22,050 samples. Each sample requires a forward pass through a stack of transposed convolutions. On a GPU this is fast, but on a CPU it can be the slowest part of the pipeline.

Key optimizations for vocoders:

# Chunked vocoding with crossfade
import numpy as np

def chunked_vocoder(mel, vocoder, chunk_size=64, overlap=8):
    """Generate audio in chunks with crossfade for seamless streaming."""
    audio_chunks = []
    hop = chunk_size - overlap

    for i in range(0, mel.shape[-1], hop):
        mel_chunk = mel[:, :, i:i + chunk_size]
        if mel_chunk.shape[-1] < chunk_size:
            # Pad the last chunk
            mel_chunk = F.pad(mel_chunk, (0, chunk_size - mel_chunk.shape[-1]))

        audio = vocoder(mel_chunk)

        if audio_chunks and overlap > 0:
            # Crossfade with previous chunk
            fade_len = overlap * vocoder.hop_length
            fade_out = np.linspace(1, 0, fade_len)
            fade_in = np.linspace(0, 1, fade_len)
            audio[:fade_len] = (audio[:fade_len] * fade_in +
                                audio_chunks[-1][-fade_len:] * fade_out)
            audio_chunks[-1] = audio_chunks[-1][:-fade_len]

        audio_chunks.append(audio)
        yield audio  # Stream each chunk as it is ready

End-to-end latency breakdown

For a voice agent responding to a user, the full chain looks like this:

Total: 270 to 1000 ms. To stay under 500 ms end-to-end, you need every component to be at its best. This is why many voice agent systems pipeline aggressively: start TTS as soon as the first LLM sentence is complete, not after the full response is generated.

# Pipelined voice agent (conceptual)
async def voice_respond(user_audio):
    # ASR and LLM can partially overlap
    text = await asr.transcribe(user_audio)
    llm_stream = llm.generate_stream(text)

    async for sentence in extract_sentences(llm_stream):
        # Start TTS immediately on each sentence
        audio_chunks = tts.stream_synthesize(sentence)
        async for chunk in audio_chunks:
            await send_audio_to_client(chunk)

Audio codec models: the new paradigm

A newer approach bypasses the mel spectrogram entirely. Models like EnCodec (Meta), SoundStream (Google), and DAC learn to compress audio into discrete tokens, similar to how LLM tokenizers work for text. The TTS model then generates these audio tokens autoregressively, and a lightweight decoder converts tokens back to waveform.

This is the architecture behind models like VALL-E, Bark, and MusicGen. The inference pattern looks more like an LLM:

The advantage is simplicity and quality. The disadvantage is speed: autoregressive generation of 75 audio tokens per second across 8 codebook levels means generating 600 tokens per second of audio. That is significantly more tokens per second than a typical LLM needs to produce, and it must all happen faster than real-time.

Serving considerations

TTS serving differs from LLM serving in a few important ways:

In LLM serving, you optimize for tokens per second. In TTS serving, you optimize for time-to-first-audio and real-time factor. The metrics are different, the bottlenecks are different, and the streaming semantics are different. But the underlying principle is the same: find the bottleneck, measure it, and attack it specifically.

Tomorrow we go visual: image generation with diffusion models, where the inference loop is iterative denoising and the kernel optimization story is completely different from anything we have seen so far.