Tensor parallelism is one of those things that sounds simple on paper. You split each weight matrix across N GPUs, each GPU computes its shard of the matmul, you all-reduce the partial results, and you get N times the memory and (theoretically) N times the throughput. In practice, I wanted to measure what actually happens to latency and throughput as you scale from one GPU to eight.
This post walks through the benchmark I ran, the methodology, and the results. The short version: tensor parallelism is essential for serving large models, but the scaling is sublinear, and the crossover points matter more than the peak numbers.
Why tensor parallelism for inference
There are several parallelism strategies for LLMs: tensor parallelism (TP), pipeline parallelism (PP), and expert parallelism (EP). For inference, TP is the default because it reduces latency per token. Pipeline parallelism adds latency proportional to the number of pipeline stages, which is fine for training but painful for interactive serving where every millisecond of TTFT counts.
With tensor parallelism, every GPU participates in every layer. The weight matrices for each linear layer are sharded column-wise or row-wise across devices. After each sharded matmul, the GPUs synchronize via an all-reduce. The result is that a single token passes through the full model in roughly 1/N the compute time, plus communication overhead.
- TP=1: single GPU, no communication overhead, limited by one GPU's memory and compute.
- TP=2: model split across 2 GPUs. Each all-reduce touches 2 devices.
- TP=4: 4-way split. All-reduce spans 4 devices.
- TP=8: full 8-GPU node. All-reduce spans the entire NVLink mesh.
The setup
I used vLLM 0.6.x on a single DGX node with 8x H100 80GB GPUs connected via NVLink (900 GB/s bidirectional per GPU pair). The model was Llama 2 70B in FP16. At FP16, 70B parameters need roughly 140 GB of weight memory, so TP=1 is impossible on a single 80GB H100. The minimum is TP=2. For comparison, I also benchmarked Llama 2 13B, which fits on a single GPU.
The benchmark script uses vLLM's built-in benchmarking with a fixed set of 500 ShareGPT prompts. I measured three things:
# Launch vLLM with tensor parallelism
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-2-70b-hf \
--tensor-parallel-size 4 \
--dtype float16 \
--max-model-len 4096
# Run the benchmark
python benchmark_serving.py \
--backend vllm \
--model meta-llama/Llama-2-70b-hf \
--dataset-name sharegpt \
--num-prompts 500 \
--request-rate inf- Time to first token (TTFT): how long until the first token is generated.
- Inter-token latency (ITL): the time between consecutive generated tokens.
- Throughput: total output tokens per second across all concurrent requests.
Results: Llama 2 70B
For the 70B model, here is what I observed across TP=2, 4, and 8 with a concurrency of 32 requests:
TTFT dropped significantly going from TP=2 to TP=4. Prefill is compute-bound, and doubling the available FLOPs nearly halves prefill time. Going from TP=4 to TP=8 still helped, but the improvement was smaller because the all-reduce overhead across 8 GPUs starts to bite. At TP=8, each all-reduce synchronization involves more data movement across the NVLink fabric.
Inter-token latency showed a similar pattern. Decode is memory-bandwidth-bound, and splitting the model across more GPUs means each GPU reads fewer weight bytes. TP=2 to TP=4 cut ITL nearly in half. TP=4 to TP=8 improved it by about 30%, not 50%, because the all-reduce for each transformer layer adds a fixed communication cost per step.
Throughput at high concurrency is where tensor parallelism really shines. With enough requests in flight, the compute and bandwidth of all GPUs are utilized. At TP=4, throughput was roughly 3.2x the TP=2 baseline, not 2x. At TP=8, it reached about 5.5x TP=2. The gains are real but sublinear.
The scaling efficiency of tensor parallelism depends on the ratio of compute time to communication time. For large batch sizes and long sequences, compute dominates and scaling is near-linear. For small batches and short sequences, communication overhead is proportionally larger, and scaling efficiency drops.
Results: Llama 2 13B
For the 13B model (which fits on a single GPU), tensor parallelism tells a different story. At TP=1, there is zero communication overhead. Going to TP=2, TTFT improved by about 40% (not 50%), and ITL improved by about 35%. Going to TP=4, ITL only improved by another 20% relative to TP=2. The communication overhead of all-reduce across 4 GPUs is a larger fraction of the total time because the per-GPU compute is smaller.
The throughput story was also less compelling. At batch size 1, TP=4 was actually slower than TP=2 for the 13B model because each GPU's matmul was so small that the NVLink all-reduce dominated. Only at batch sizes above 16 did TP=4 start to outperform TP=2.
Rule of thumb: use the minimum tensor parallelism degree that fits your model in memory. Over-sharding small models across many GPUs wastes NVLink bandwidth on communication that outweighs the compute savings.
Where the overhead lives
I profiled a single forward pass with torch.profiler to see exactly where time goes at different TP degrees. The breakdown for a single decode step of Llama 2 70B at TP=4:
# Approximate time breakdown per decode step, TP=4
# Linear layers (sharded matmuls): 62%
# All-reduce (NCCL): 24%
# Attention (FlashAttention): 9%
# Other (layernorm, RoPE, etc.): 5%At TP=2, the all-reduce was about 15% of each step. At TP=8, it grew to about 31%. This is the fundamental tradeoff: more GPUs means each matmul is smaller (faster), but each all-reduce is larger (slower). The optimal TP degree depends on model size, batch size, and your interconnect bandwidth.
NVLink vs. PCIe
All the numbers above assume NVLink interconnect. On a system where GPUs communicate over PCIe Gen4 x16 (about 32 GB/s per direction), tensor parallelism scaling is dramatically worse. The all-reduce overhead at TP=4 over PCIe was roughly 4x what it was over NVLink. For a 70B model at TP=4 over PCIe, the all-reduce consumed nearly 50% of each decode step, making TP=8 essentially pointless on PCIe-only systems.
This is why NVIDIA sells NVLink and why DGX systems exist. The interconnect is not a nice-to-have; it is the bottleneck that determines whether tensor parallelism scales or stalls.
Practical takeaways
- Use the minimum TP degree that fits your model. For a 70B FP16 model on 80GB GPUs, that is TP=2.
- If you need lower latency, increase TP, but expect diminishing returns beyond TP=4 for most models.
- If you need higher throughput, increase batch size before increasing TP. Batching is free parallelism with zero communication cost.
- Never run tensor parallelism over PCIe if you can avoid it. NVLink is 10x+ faster for all-reduce.
- Profile your specific model and workload. The crossover points depend on sequence length, batch size, and model architecture.
Tomorrow I will look at MIG (Multi-Instance GPU), which goes in the opposite direction: instead of spreading one model across many GPUs, you split one GPU into many isolated instances.