T4 is the GPU almost nobody talks about but almost everybody uses. It's the default on every cloud, the edge inference card, the budget option. And it's the reason "small model, big throughput" is a real strategy.
What T4 is
- Turing architecture. The same generation as the RTX 20-series gaming cards, but repackaged for datacenter.
- 16GB GDDR6. Not HBM, GDDR6. Cheaper, slower, but plenty for small models.
- 70W TDP. Passive-cooled, no power connector, fits in any server.
- No NVLink. T4 is strictly single-GPU. No multi-GPU scaling.
- INT8 and FP16 tensor cores. Turing introduced the first tensor cores, and T4 has them.
The numbers
- FP16: 65 TFLOPS dense.
- INT8: 130 TOPS dense, 260 with sparsity.
- Memory: 16GB GDDR6, 320GB/s.
- Power: 70W.
Why it matters for inference
T4 is the GPU that made "serve a small model cheaply" a real business. A 7B model in INT8 is ~7GB, which fits on a T4 with room for KV cache. The throughput is modest, but the cost per hour is tiny, and for high-volume low-complexity workloads (classification, extraction, embeddings), T4 is often the most cost-effective GPU per dollar.
It's also the GPU that runs Whisper and other ASR at scale: a batch of audio clips on a T4 is cheap and fast enough for most production needs.
Sources
- NVIDIA T4 product page: official specs.
- NVIDIA T4 architecture whitepaper: Turing datacenter design.