L4 is T4's successor, built on the Ada Lovelace architecture. It's the modern answer to the question T4 answered: how do you make a small, efficient GPU that's still useful for AI?
What changed from T4
- More memory. 24GB GDDR6, up from 16GB. And 300GB/s bandwidth, close to T4's 320GB/s but with more capacity.
- AV1 encode. L4 has dedicated AV1 encoders, making it the choice for video AI (transcoding, vision pipelines).
- RTX IO. GPU-accelerated storage decompression, which speeds up loading large assets.
- Ada tensor cores. 4th-gen tensor cores with FP8 support (though L4's FP8 is limited vs datacenter parts).
- Still no NVLink. L4 is single-GPU, like T4.
The numbers
- FP16: 121 TFLOPS dense.
- INT8: 242 TOPS dense.
- Memory: 24GB GDDR6, 300GB/s.
- Power: 72W TDP.
Why it matters for inference
L4 is the modern edge card. Its 24GB fits a 7B model in FP16 (14GB) or a 13B model in INT8 (13GB), with room for KV cache. The AV1 encoders make it the default for video AI, and the low power means it slots into existing servers without power upgrades.
For inference, L4 is the "good enough, cheap" option: not a B200, but for a 7B model serving high volume, it's often the most cost-effective per token.
Sources
- NVIDIA L4 product page: official specs.
- NVIDIA Ada Lovelace architecture: the family L4 belongs to.