Every LLM serving system faces the same fundamental tension: throughput wants large batches, but latency wants small ones. Today I want to trace this tradeoff carefully, because understanding where the knee in the curve falls is what separates a well-tuned deployment from one that is either wasting money or frustrating users.
Why bigger batches improve throughput
During decode, the model weights are loaded from HBM once per step regardless of how many requests are in the batch. A single decode step for batch size 1 loads the full model (say, 14 GB for a 7B FP16 model) to produce one token. A decode step for batch size 64 loads the same 14 GB but produces 64 tokens. The memory bandwidth cost is amortized across the batch.
This is the roofline model in action. Decode is memory-bandwidth-bound, and batching raises the arithmetic intensity by doing more compute per byte loaded. As batch size increases, you move up the roofline diagonal toward the compute ceiling.
# Simplified throughput model
#
# H100 bandwidth: 3.35 TB/s
# Model size (7B FP16): 14 GB
# Time to load weights once: 14 GB / 3350 GB/s = 4.18 ms
#
# Batch size 1: 1 token / 4.18 ms = 239 tok/s
# Batch size 8: 8 tokens / 4.18 ms = 1,914 tok/s
# Batch size 32: 32 tokens / 4.18 ms = 7,655 tok/s
# Batch size 128: 128 tokens / 4.18 ms = 30,622 tok/s
#
# In practice, compute cost grows with batch size too,
# but until you saturate the Tensor Cores, it is nearly free.In practice the scaling is not perfectly linear because attention computation grows with the total number of cached tokens across the batch, and there are cache effects and other overheads. But the general shape holds: throughput scales roughly linearly with batch size until you hit a ceiling.
Why bigger batches hurt TTFT
Time-to-first-token (TTFT) measures how long a user waits before the first token appears. It has two components:
- Queuing delay: Time spent waiting for a batch slot to open up.
- Prefill time: Time to process the prompt and produce the first token.
Larger batches hurt both. Queuing delay increases because the scheduler waits longer to fill a bigger batch (in static batching) or because existing requests occupy all slots (in continuous batching). Prefill time increases because the prefill computation for new requests competes with ongoing decode steps for GPU resources.
In continuous batching, the prefill of a new request interrupts the decode batch. While the GPU is running prefill for request N+1, all N existing requests in the decode batch are stalled. The larger the batch, the more requests are impacted by each prefill event.
Measuring the tradeoff
To make this concrete, let me walk through what happens at different batch sizes for a 7B model on an H100, with 512-token prompts and 256-token outputs:
# At batch size 1:
# Prefill (512 tokens): ~10 ms
# Per-token decode: ~4.2 ms
# TTFT: ~10 ms
# Total generation: 10 + 256 * 4.2 = 1,085 ms
# Throughput: 256 / 1.085 = 236 tok/s
# At batch size 16:
# Prefill (512 tokens): ~12 ms (slightly slower due to batch context)
# Per-token decode: ~4.5 ms (weights loaded once, 16 tokens produced)
# TTFT: ~12 ms + queuing delay
# Per-request decode: 256 * 4.5 = 1,152 ms
# Throughput: 16 * 256 / 1.164 = 3,519 tok/s
# At batch size 128:
# Prefill (512 tokens): ~25 ms (attention over large total context)
# Per-token decode: ~8 ms (nearing compute bound)
# TTFT: ~25 ms + significant queuing delay
# Throughput: 128 * 256 / 2.073 = 15,800 tok/s
# At batch size 256:
# Prefill: ~50 ms or more
# Per-token decode: ~16 ms (compute saturated)
# TTFT: ~50 ms + large queuing delay
# Throughput: 256 * 256 / 4.146 = 15,810 tok/s
# Note: throughput stops scaling! You have hit the compute ceiling.The saturation point
There is a batch size beyond which throughput stops improving. This is the point where decode transitions from memory-bound to compute-bound. On an H100 with a 7B FP16 model, this happens around batch size 128 to 192 for typical sequence lengths. Beyond this point, you are paying more TTFT and inter-token latency for no throughput benefit.
The exact saturation point depends on:
- Model size: Larger models hit the bandwidth ceiling at lower batch sizes because there is more weight data to load.
- Sequence length: Longer sequences increase the attention computation, which scales with the total number of KV cache tokens across the batch.
- Quantization: INT8 or FP8 models halve the weight bytes, so the bandwidth ceiling is hit at roughly 2x the batch size compared to FP16.
- GPU type: GPUs with higher bandwidth-to-compute ratios (like the A100 vs H100) hit the ceiling at different batch sizes.
Operational strategies
In practice, you rarely pick a single batch size and stick with it. Instead, you set a maximum batch size and let the continuous batching scheduler fill up to that limit. The question becomes: what should that maximum be?
Here is how I think about it:
- Latency-sensitive (chatbots, interactive): Cap batch size well below saturation. Aim for TTFT under 200 ms and inter-token latency under 50 ms. For a 7B model on H100, this typically means max batch size of 16 to 32.
- Throughput-sensitive (batch processing, offline): Push batch size to the saturation point. TTFT does not matter because no user is waiting. For a 7B on H100, batch size 128 to 192.
- Mixed workloads: Use disaggregated serving to decouple the two. Prefill workers handle TTFT, decode workers handle throughput, and they are scaled independently.
Never run past the throughput saturation point. You are trading latency for nothing. Find the knee in your throughput curve by benchmarking your specific model, GPU, and workload, and set your max batch size just below it.
The TTFT vs throughput tradeoff is not a problem to solve. It is a dial to set. The right setting depends on whether you are building a chatbot or a data pipeline, and the only way to find it is to measure your specific system.
Next up: moving from measurement to deployment. On day 53 I will build a production Dockerfile for vLLM that handles all the packaging details that benchmarking scripts ignore.