Before vLLM, serving an LLM was a memory-management nightmare. The KV cache grows and shrinks per request, and the old engines handled it the naive way: preallocate a big contiguous block per request, waste whatever isn't used, and let fragmentation eat the rest. GPU utilization hovered around 50%. It was fine for demos, brutal for production.
vLLM's thesis, from the paper: an LLM serving system is an operating system for memory, so let's build it like one. Two ideas made it work.
Idea 1: PagedAttention
Virtual memory solved exactly this problem for CPUs decades ago: don't allocate contiguous physical memory, allocate fixed-size pages and map them with a page table. vLLM does the same for the KV cache. The cache is split into fixed-size blocks (typically 16 tokens per block), and a block table maps logical blocks to physical GPU memory blocks.
Why this wins:
- Near-zero fragmentation. Blocks can be scattered anywhere in memory; the block table keeps track.
- Sharing. Two requests with a common prefix can share the same physical blocks. That's the foundation of prefix caching.
- Copy-on-write. When a shared block needs to diverge, only the changed block is copied, not the whole cache.
In the paper's numbers: PagedAttention gets 2-4x throughput over the naive baselines, and up to 90% of the theoretical optimum. The memory was there all along; it was just fragmented and wasted.
Before vLLM, every request rented a whole warehouse and used a corner of it. vLLM rents shelves. Same building, way more tenants.
Idea 2: continuous batching
The old way to batch: wait for N requests, run them together, wait again. The new way: a running batch is never closed. When a request finishes, a new one joins the batch immediately. When a request hits a long generation, it doesn't block everyone else, the scheduler just runs the others around it.
This is the difference between "GPU idle half the time" and "GPU busy". Continuous batching keeps the pipeline full, and it's why vLLM and its descendants (SGLang, TensorRT-LLM) dominate serving.
What this means for you
If you're serving models, these two ideas are why you reach for vLLM, SGLang, or TensorRT-LLM instead of rolling your own. The memory manager and the scheduler are hard, and they've been solved well. Your job is to configure them, not rebuild them.
But understanding the internals matters, because the knobs you'll turn, max_num_seqs, gpu_memory_utilization, enable_prefix_caching, all of them map to these two ideas. Turn them blind and you're just guessing.
The takeaway
vLLM didn't make the GPU faster. It made the GPU busier.
Next: SGLang and RadixAttention, the other big serving engine, and what it does differently.