A100 is the GPU that trained GPT-3 and the first wave of frontier models. It's Ampere, the architecture that made tensor cores the defining feature of a datacenter GPU. Understanding A100 means understanding where the modern era started.
I include the A100 because a lot of the intuition I use daily was formed on it, and because plenty of production traffic still runs on these. Knowing what it could not do is how I recognise what the newer parts actually bought.
What changed from V100
- TF32. A100 introduced TF32, a 19-bit format (8 exponent, 10 mantissa) that runs on tensor cores at FP32-ish throughput. It made mixed-precision training the default.
- MIG. Multi-Instance GPU: partition one A100 into up to 7 isolated instances, each with its own memory and compute. This is the origin of GPU sharing for inference.
- Structured sparsity. 2:4 sparsity (hardware-enforced 50% zero pattern) doubles effective throughput.
- cp.async. Async copy from global to shared memory, bypassing registers. The precursor to Hopper's TMA.
- Bigger memory. 40GB or 80GB HBM2e at 2TB/s (80GB variant), up from V100's 32GB.
- NVLink 3. 600GB/s per GPU, up from 300GB/s.
- More SMs. 108 SMs vs V100's 80, and L2 cache grew from 6MB to 40MB, critical for memory-bound inference.
The numbers
- FP16: 312 TFLOPS dense (80GB SXM), 624 with sparsity.
- TF32: 156 TFLOPS dense.
- INT8: 624 TOPS dense, 1248 with sparsity.
- Memory: 80GB HBM2e, 2TB/s.
- NVLink 3: 600GB/s per GPU.
- Power: 400W TDP (SXM), 300W (PCIe).
Where the serving story begins
A100 is where the serving story begins. Its 80GB was the first time a serious model could fit on one GPU: a 13B model in FP16 is ~26GB, leaving room for KV cache. Its MIG feature is why "GPU sharing" exists as a concept, and why cloud providers could slice GPUs for smaller workloads.
It's also the baseline every comparison uses. When NVIDIA says "B200 is 30x faster than A100 for inference", the A100 is the yardstick. And for serving, MIG means you can run 7 small models on one A100, each isolated.
Sources
- NVIDIA A100 product page: official specs.
- NVIDIA Ampere architecture whitepaper: TF32, MIG, the full design.