Skand Tandon
Runtime

Model parallelism: TP, EP, PP

When a model does not fit on one GPU, you split it. But how you split it changes everything: latency, throughput, communication cost, and scaling behavior. Tensor, expert, and pipeline parallelism each solve different problems.

2025-10-18 runtime 9 min read

A Llama 3 70B model in BF16 weighs 140 GB. An H100 has 80 GB of HBM. The arithmetic is clear: one GPU is not enough. You need at least two, and probably more once you account for KV cache, activations, and the optimizer states if you are fine-tuning. The question is not whether to split the model, but how.

There are three main strategies, and each one makes a fundamentally different trade-off. Understanding these trade-offs is the difference between a system that scales and one that stalls.

Tensor parallelism (TP)

Tensor parallelism splits individual layers across GPUs. Each GPU holds a slice of every weight matrix, computes its portion, and then the results are combined with an all-reduce collective.

For a linear layer Y = XW, TP splits the weight matrix W column-wise across N GPUs. Each GPU computes Y_i = X @ W_i (its slice of the output), then the partial results are combined. For the MLP with its two linear layers, one is split column-wise and the other row-wise, so the all-reduce only happens once per MLP block.

# TP=4 for a linear layer with shape [4096, 4096]
# GPU 0: W[:, 0:1024]    -> Y_0 = X @ W_0   (shape: [B, 1024])
# GPU 1: W[:, 1024:2048]  -> Y_1 = X @ W_1
# GPU 2: W[:, 2048:3072]  -> Y_2 = X @ W_2
# GPU 3: W[:, 3072:4096]  -> Y_3 = X @ W_3
# All-gather: Y = concat(Y_0, Y_1, Y_2, Y_3)

The beauty of TP is that it reduces latency. All GPUs work on the same token simultaneously, and the result is ready as soon as the all-reduce completes. The cost is that all-reduce is a synchronous collective: every GPU must wait for every other GPU before proceeding. This means TP is only practical over high-bandwidth interconnects like NVLink (900 GB/s on H100 NVSwitch) or NVLink on Blackwell (1.8 TB/s). Over PCIe (64 GB/s) or network (100-400 Gbps), the communication overhead dominates.

Rule of thumb

Use tensor parallelism within a single node (across GPUs connected by NVLink). Never use TP across nodes unless you have InfiniBand with RDMA and can tolerate the latency hit.

Pipeline parallelism (PP)

Pipeline parallelism splits the model by layers. GPU 0 gets layers 0 through 19, GPU 1 gets layers 20 through 39, and so on. Each GPU processes its layers and passes the activations to the next GPU.

The advantage: communication is point-to-point, not collective. You send one activation tensor to the next GPU, not an all-reduce across all GPUs. This makes PP work over slower interconnects, including across nodes over network.

The disadvantage: pipeline bubbles. In the simplest case, GPU 1 sits idle while GPU 0 processes its layers. The pipeline only becomes efficient when multiple micro-batches are in flight, filling the bubble. But for inference (especially decode, where batch size per sequence is 1 token), it is hard to fill the pipeline. This makes PP less attractive for latency-sensitive inference than for training.

# PP=4 for a 32-layer model
# GPU 0: layers 0-7    (processes first, passes activation to GPU 1)
# GPU 1: layers 8-15   (waits for GPU 0, then processes)
# GPU 2: layers 16-23  (waits for GPU 1)
# GPU 3: layers 24-31  (waits for GPU 2, produces output)
#
# Latency = sum of all stage latencies (serial)
# vs TP where latency ≈ max stage latency + all-reduce

PP is most useful in inference when combined with TP. A common pattern for serving a 405B model: TP=8 within each node (8 GPUs connected by NVSwitch), PP=2 across two nodes (connected by InfiniBand). Each node holds half the layers, and within each node, every layer is tensor-parallelized across 8 GPUs.

Expert parallelism (EP)

Expert parallelism is specific to Mixture-of-Experts (MoE) models like Mixtral, DeepSeek, and Grok. In a MoE layer, only a subset of "experts" (typically 2 out of 8 or 2 out of 64) are activated for each token. Expert parallelism places different experts on different GPUs.

The communication pattern is an all-to-all: each GPU sends its tokens to the GPUs that hold the selected experts, and the results are sent back. Unlike TP's all-reduce (which communicates the full tensor), EP's all-to-all only moves the tokens routed to each expert, which is a fraction of the total.

EP is compelling for MoE because it maps naturally to the model's sparsity. Each GPU only needs to hold and compute the experts assigned to it. The challenge is load balancing: if the router sends most tokens to the same expert, that GPU becomes a hotspot while others idle. Good MoE training includes auxiliary losses to encourage balanced routing, but in practice, some imbalance is inevitable.

Combining them

Production systems for large models almost always combine multiple parallelism strategies:

The choice depends on your hardware topology, model architecture, and optimization target (latency vs throughput):

Parallelism is not one strategy. It is a toolbox. The art is matching each tool to the right layer, the right interconnect, and the right performance target. Get it wrong and you leave half your GPUs idle. Get it right and you unlock linear scaling.

We will see these parallelism strategies show up again when we discuss GPU architecture (why NVLink bandwidth matters so much for TP) and multi-GPU instances and MIG (how cloud providers expose these topologies).