vLLM was the last serving engine. This is its friendly rival: SGLang. Same goal, different bet on where the wins are.
vLLM's bet was memory management (PagedAttention). SGLang's bet is prefix reuse (RadixAttention) plus structured output. Both are right, which is why both are thriving.
RadixAttention
Prefix caching in detail: when prompts share a prefix, you can reuse the KV cache instead of recomputing it. SGLang makes this the centerpiece of the whole engine.
Instead of a flat cache, SGLang organizes the KV cache as a radix tree. Each node is a prefix, and the tree finds the longest shared prefix in O(log n) time. When a request arrives, it walks the tree, finds the deepest matching node, and reuses everything above it.
The radix tree is the difference between "cache some prefixes" and "cache every prefix, optimally". It's the data structure that makes prefix caching a first-class feature rather than an afterthought.
Structured output
SGLang's other bet: make the model produce valid JSON, guaranteed. It does this with constrained decoding, where the sampler is restricted to tokens that keep the output valid. The grammar is compiled to a state machine, and at each step only the tokens that lead to a valid completion are considered.
Why this matters for inference: when you need JSON, you have two options. Generate free-form and pray, or constrain the output and know it's right. Constrained decoding means no retries, no parsing errors, no "the model added a trailing comma again". For production APIs, that's huge.
vLLM is a memory manager that serves models. SGLang is a prefix tree that serves models. Same job, different superpower.
Which one do I use?
Both, honestly. vLLM when I need raw throughput and mature memory management. SGLang when I have heavy prefix reuse (RAG, chat) or need guaranteed structured output. They've converged a lot, and the choice is workload-dependent, not religion.
The deeper lesson: serving engines are differentiated by one or two bets. Understand the bet, and you understand the engine.
The takeaway
vLLM bet on memory. SGLang bet on reuse. Both won, because both were betting on the same underlying truth: inference is a memory problem.
Next: TensorRT-LLM, the compile-everything approach.