Tensor parallel scaling curves

Tensor parallelism is the first tool you reach for when a model does not fit on one GPU. Split each weight matrix across N GPUs, compute a shard of every layer in parallel, then all-reduce the partial results. Simple in concept, tricky in practice, because the communication overhead grows with the number of GPUs and eventually dominates the compute savings.

I have been running scaling experiments for a while now, and the curves tell a consistent story: TP is incredibly effective up to a point, and then it falls off a cliff. The point where it falls off depends on model size, sequence length, batch size, and your interconnect. Let me walk through the mechanics and the numbers.

What tensor parallelism actually splits

In Megatron-style tensor parallelism (which vLLM, TensorRT-LLM, and SGLang all use), each transformer layer's linear projections are split across GPUs. For the attention block, the Q, K, V projections are split column-wise so each GPU handles a subset of attention heads. The output projection is split row-wise. For the MLP, the gate and up projections are split column-wise, and the down projection is split row-wise.

The row-wise splits require an all-reduce after computation to sum the partial results. In a standard transformer block, this means two all-reduce operations per layer: one after the attention output projection and one after the MLP down projection.

# Tensor parallel attention (simplified)
# Each GPU holds heads[rank*h_per_gpu : (rank+1)*h_per_gpu]
q = x @ W_q_shard  # local matmul, no communication
k = x @ W_k_shard
v = x @ W_v_shard
attn_out = attention(q, k, v)  # local attention on owned heads
partial = attn_out @ W_o_shard  # row-split, partial sum
output = all_reduce(partial)    # communication!

For a model with L layers, the total number of all-reduce calls per forward pass is 2L. A 70B model with 80 layers means 160 all-reduce operations. Each all-reduce on N GPUs exchanges roughly 2 * (N-1)/N * message_size bytes using the ring algorithm. The cost is dominated by the number of operations and the latency of each one, not just the bandwidth.

The scaling math

In a perfect world, TP=N would give you an N-times speedup for compute and N-times more memory bandwidth. Each GPU does 1/N of the matmul work and reads 1/N of the weights. For decode (which is memory-bandwidth bound), this means N-times the effective bandwidth, which directly translates to N-times the tokens per second.

In reality, you have to subtract the communication cost. The time for each token becomes:

T(TP=N) = T_compute/N + T_memory/N + 2*L * T_allreduce(N)

Where T_allreduce(N) includes both the latency component (which is roughly constant per call, around 5 to 10 microseconds on NVLink) and the bandwidth component (which scales with message size). For a 70B model, each all-reduce message during decode is small, roughly hidden_dim * 2 bytes = 8192 * 2 = 16 KB per layer. The latency dominates at this message size.

This gives us 160 calls * ~8 microseconds = ~1.3 milliseconds of communication overhead per token. At TP=1 on a single H100, a 70B model generates a token in roughly 30 ms (limited by memory bandwidth reading 140 GB of FP16 weights at 3.35 TB/s). At TP=2, the compute and memory time halve to 15 ms, but you add ~1.3 ms of communication, giving 16.3 ms. That is a 1.84x speedup, not 2x.

Where the curve bends

The pattern I see consistently is:

• TP=1 to TP=2: Near-linear scaling, typically 1.8x to 1.9x speedup. The communication overhead is small relative to the compute/memory savings.
• TP=2 to TP=4: Good scaling, 3.2x to 3.5x over baseline. Still clearly worth it. Communication cost is noticeable but not dominant.
• TP=4 to TP=8: This is where it gets interesting. For large models (70B+), the scaling is still decent: 5x to 6x over baseline. For smaller models (7B to 13B), the scaling can be as low as 4x to 5x because the compute per layer is smaller, making the fixed communication latency relatively more expensive.

The critical insight is that the communication cost is partly fixed (latency) and partly proportional (bandwidth). For small models with small hidden dimensions, the fixed latency dominates, and scaling falls off earlier. For large models, the per-GPU compute is substantial enough to absorb the communication overhead.

Prefill vs decode: different curves

The scaling curves look different for prefill and decode because their bottlenecks differ. Prefill is compute-bound: splitting the large matrix multiplications across GPUs gives near-linear compute speedup, and the all-reduce messages are larger (proportional to sequence length * hidden_dim), which amortizes the per-call latency better.

Decode is memory-bandwidth bound: the speedup comes from splitting weight reads across more GPUs, each with its own HBM bandwidth. But the all-reduce messages are small (just one token's worth), so the fixed latency per call hurts more.

In practice, I see prefill scaling close to 0.9x linear up to TP=8 for long sequences (2048+ tokens). Decode scaling is more like 0.7x linear at TP=8. This means the optimal TP degree can differ depending on whether your workload is TTFT-sensitive (optimize prefill) or throughput-sensitive (optimize decode).

NVLink vs PCIe: a cliff, not a slope

Everything I have described assumes NVLink interconnect, which provides 900 GB/s bidirectional bandwidth on H100 (450 GB/s per direction) with sub-10 microsecond latency. On PCIe Gen5, you get roughly 64 GB/s per direction with higher latency.

The difference is not gradual. Going from NVLink to PCIe at TP=4 can turn a 3.5x speedup into a 2x speedup because every all-reduce now takes 5x to 10x longer. I have seen cases where TP=4 on PCIe is slower than TP=2 on NVLink for decode workloads. If you are doing tensor parallelism across PCIe-connected GPUs, stop at TP=2 unless you have measured and confirmed it helps.

Batch size interaction

Larger batch sizes improve TP scaling because they increase the compute-to-communication ratio. With a batch of 32 tokens decoding simultaneously, the matmul work per layer increases 32x while the all-reduce message size also increases 32x. But the all-reduce bandwidth now dominates over latency, and NVLink bandwidth is enormous, so the overhead stays manageable.

This is why high-throughput serving with continuous batching and large batches works well at TP=8, while interactive single-request serving sees diminishing returns beyond TP=4. The batch size determines which regime you are in.

Tensor parallelism is not a knob you turn up to make things faster. It is a tradeoff between splitting compute and paying communication. Measure your actual scaling curve at your actual batch sizes before committing to a TP degree. The right answer for a 7B model at batch 1 is very different from a 70B model at batch 64.

For background on model parallelism strategies, see model parallelism: TP, EP, PP. For how this interacts with GPU architecture, see GPU architecture: SMs, HBM, caches. And for a hands-on simulation, check simulate tensor parallelism.