Deploy vLLM, benchmark TTFT and throughput

Everything I have done so far, roofline benchmarks, profiling traces, custom kernels, has been about understanding individual pieces. Today I put the pieces together: deploy a real serving engine, send real requests, and measure end-to-end performance. vLLM is the obvious starting point. It is open source, widely deployed, and implements most of the optimizations I have been studying: PagedAttention, continuous batching, and efficient KV cache management.

Standing up the server

vLLM ships with an OpenAI-compatible API server. Getting it running is straightforward:

# Install
pip install vllm

# Launch the server with Llama-3.1-8B-Instruct
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --dtype float16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --port 8000

A few flags worth explaining:

• --gpu-memory-utilization 0.90: vLLM pre-allocates GPU memory for the KV cache at startup. 0.90 means it reserves 90% of available VRAM. The remaining 10% is headroom for CUDA context and temporary buffers. On an 80GB A100, this gives roughly 72GB for model weights plus KV cache blocks.
• --max-model-len 4096: Caps the maximum sequence length. Longer sequences need more KV cache blocks per request, which reduces the number of concurrent requests the server can handle.
• --dtype float16: Llama-3.1-8B in FP16 uses about 16GB of VRAM for weights. The rest goes to KV cache, which on this configuration supports roughly 200+ concurrent sequences of length 4096.

On startup, vLLM logs the number of KV cache blocks allocated. This is the single most important number for capacity planning. Each block holds a fixed number of tokens across all layers. More blocks means more concurrent requests before the server starts queueing.

The two metrics that matter

In LLM serving, two metrics capture most of the user experience:

• TTFT (Time to First Token): How long from request arrival to the first token of the response. This is the latency users feel while staring at a blank screen. It is dominated by prefill time (processing the input prompt) plus any queueing delay.
• Throughput (tokens/second): The total token generation rate across all concurrent requests. This determines cost efficiency. More tokens per second per GPU means lower cost per token.

These two metrics are in tension. Batching more requests improves throughput but can increase TTFT because new requests must wait for a scheduling slot and prefill competes with decode for GPU time.

The benchmarking script

I wrote a simple benchmark client that sends concurrent requests and measures per-request latency:

import asyncio
import aiohttp
import time
import json

async def send_request(session, prompt, max_tokens=128):
    payload = {
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True,
    }
    t_start = time.perf_counter()
    t_first_token = None
    token_count = 0

    async with session.post(
        "http://localhost:8000/v1/chat/completions",
        json=payload
    ) as resp:
        async for line in resp.content:
            decoded = line.decode().strip()
            if decoded.startswith("data: ") and decoded != "data: [DONE]":
                if t_first_token is None:
                    t_first_token = time.perf_counter()
                token_count += 1

    t_end = time.perf_counter()
    return {
        "ttft": (t_first_token - t_start) if t_first_token else None,
        "total_time": t_end - t_start,
        "tokens": token_count,
        "tps": token_count / (t_end - t_start) if token_count else 0,
    }

async def benchmark(concurrency, num_requests, prompt):
    connector = aiohttp.TCPConnector(limit=concurrency)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [send_request(session, prompt) for _ in range(num_requests)]
        results = await asyncio.gather(*tasks)
    return results

I ran this with a fixed prompt of about 200 tokens ("Explain the attention mechanism in transformers in detail") and 128 output tokens, sweeping concurrency from 1 to 64.

Results and what they mean

The pattern was clear and matched the theory:

• Concurrency=1: TTFT around 45ms (pure prefill latency, no queueing). Per-request decode speed about 85 tokens/sec. Total throughput: 85 tok/s. The GPU is severely underutilized because decode at batch=1 is memory-bound.
• Concurrency=8: TTFT rose to about 60ms (slight queueing). But total throughput jumped to 520 tok/s. Continuous batching is working: vLLM processes all 8 requests' decode steps in a single batched forward pass, amortizing weight loads across 8 tokens.
• Concurrency=32: TTFT around 120ms. Throughput at 1,400 tok/s. The GPU is now much better utilized. Each decode step produces 32 tokens for the memory cost of loading weights once.
• Concurrency=64: TTFT spiked to 280ms. Throughput plateaued at about 1,600 tok/s. At this point, KV cache memory is becoming the constraint. vLLM starts preempting (evicting) some requests to make room for others, which causes retries and inflates tail latency.

Understanding continuous batching in practice

The key insight from watching vLLM's logs during the benchmark: continuous batching is not static. The batch composition changes every iteration. When a request finishes generating, its slot opens immediately for a new request. When a new request arrives, vLLM runs its prefill in the next iteration, interleaved with ongoing decode steps from other requests.

This is fundamentally different from static batching, where all requests in a batch must finish before the batch is released. With static batching, a batch of 32 requests runs at the speed of the slowest (longest output). With continuous batching, fast requests exit early, and their capacity is recycled immediately.

vLLM exposes metrics via its /metrics endpoint in Prometheus format. The most useful ones:

• vllm:num_requests_running: How many requests are actively generating. Watch this to see the effective batch size.
• vllm:num_requests_waiting: How many are queued waiting for KV cache capacity. If this is consistently above zero, you are over-subscribed.
• vllm:gpu_cache_usage_perc: KV cache utilization. When this hits 100%, preemptions start.
• vllm:avg_prompt_throughput_toks_per_s and vllm:avg_generation_throughput_toks_per_s: Server-side throughput counters.

What I would do differently in production

This benchmark was simple on purpose. In a real deployment, I would add:

1. Variable prompt lengths: Real traffic has a distribution of input lengths. Short prompts prefill fast; long prompts can block the scheduler. A good benchmark uses a realistic distribution, not a single fixed prompt.
2. Variable output lengths: Same reasoning. Requests that generate 10 tokens exit quickly and free KV cache. Requests that generate 2,000 tokens hold resources much longer.
3. Quantization: Running with AWQ or GPTQ INT4 would halve the model's weight memory, freeing more space for KV cache blocks and significantly increasing the concurrency sweet spot.
4. Percentile latencies: Averages hide problems. The p99 TTFT is what determines user experience in a production system.

Benchmarking a serving engine is not about finding the peak number. It is about mapping the trade-off curve between latency and throughput, and finding the operating point that matches your SLA.

Tomorrow on day 47, I will do the same exercise with SGLang, which takes a different approach to scheduling and adds native support for structured output, a feature that changes the performance profile in interesting ways.