Skand Tandon
Infrastructure

GPU architecture: SMs, HBM, caches

To optimize inference, you need to understand the machine you are optimizing for. This is a tour of the GPU from the transistor level up: streaming multiprocessors, Tensor Cores, HBM, the cache hierarchy, and how data flows.

2025-10-20 infrastructure 9 min read

I used to think of the GPU as a black box: data goes in, tensor math happens, results come out. That mental model works until you need to debug why your kernel is hitting 40% of theoretical throughput instead of 80%. Then you need to understand the actual hardware. Here is the architecture from the bottom up, with a focus on why each piece matters for LLM inference.

Streaming Multiprocessors (SMs)

The SM is the fundamental compute unit of an NVIDIA GPU. Think of it as a small processor with its own register file, shared memory, and execution units. A GPU is made of many SMs running in parallel:

Each SM contains:

Why SMs matter for inference

More SMs means more parallel compute capacity. But for memory-bound decode, the SM count barely matters. What matters is how fast data gets to the SMs. The compute units are fast enough; they are starved for data.

Tensor Cores: the matrix engines

Tensor Cores are the reason modern GPUs are so fast at neural network workloads. They perform fused matrix-multiply-accumulate (MMA) operations: D = A * B + C where A, B, C, D are small matrices (typically 16x16 or 16x8 tiles).

Each generation adds lower-precision support:

For LLM inference, Tensor Cores are fully utilized during prefill (large matrix multiplies) but underutilized during decode (small vector-matrix multiplies where the batch dimension is 1). This is the hardware manifestation of the compute-vs-bandwidth bottleneck from the roofline post.

HBM: High Bandwidth Memory

HBM is the main memory of the GPU. It is where model weights, KV cache, and activations live. HBM is built as stacked DRAM dies connected to the GPU through silicon interposers, providing much higher bandwidth than traditional GDDR memory:

Two numbers define HBM's role in inference:

# Back-of-envelope decode speed
model_bytes = 70e9 * 2  # 70B params, 2 bytes each (BF16)
bandwidth = 3.35e12     # H100 HBM3 bandwidth in bytes/sec
time_per_token = model_bytes / bandwidth  # ~0.042 seconds
tokens_per_second = 1 / time_per_token    # ~24 tok/s (single sequence)

This is why the H200's jump to 4.8 TB/s and Blackwell's 8.0 TB/s matter so much. For memory-bound decode, bandwidth is throughput.

The cache hierarchy

Between HBM and the compute units sits a cache hierarchy that can dramatically affect performance:

The bandwidth at each level is dramatically different:

The ratio between L2 bandwidth and HBM bandwidth (roughly 3.5x on H100) is why L2-resident KV caches are such a big deal. If your per-request KV cache fits in L2, attention computation is 3.5x faster than if it spills to HBM. This is one reason why GQA (grouped-query attention, which shrinks the KV cache) had such an outsized impact on inference performance.

NVLink and NVSwitch

For multi-GPU serving, the interconnect between GPUs matters as much as the GPUs themselves. NVLink is NVIDIA's proprietary high-bandwidth interconnect:

Compare this to PCIe Gen5 at 64 GB/s or InfiniBand at 50-100 GB/s. Tensor parallelism requires all-reduce operations after every layer. Over NVLink, an all-reduce on a 16 MB tensor takes microseconds. Over PCIe, it takes milliseconds. That difference, multiplied by 80 layers, is the difference between a fast model and a slow one.

The GPU is not a monolithic accelerator. It is a hierarchy of memories connected by buses of decreasing bandwidth, with compute units at the leaves. Understanding this hierarchy is understanding why certain optimizations work and others do not.

Tomorrow: how this architecture has evolved from Hopper through Blackwell to Rubin, and what each generation changed for inference.