Deploy SGLang, benchmark structured output

Yesterday on day 46, I deployed vLLM and mapped the TTFT-throughput tradeoff curve. Today I wanted to do the same with SGLang, but with a twist: structured output. In production, many LLM calls need to return valid JSON matching a schema. The question is how much performance that constraint costs, and whether SGLang's approach to constrained decoding changes the answer.

What makes SGLang different

SGLang (from the LMSYS team at UC Berkeley) is built on a few ideas that differentiate it from vLLM:

• RadixAttention: A radix tree-based KV cache that enables automatic prefix sharing across requests. If ten requests share the same system prompt, SGLang stores that prefix's KV cache once and reuses it. vLLM has prefix caching too (I wrote about the concept on day 40), but SGLang's radix tree approach handles arbitrary prefix patterns more naturally.
• Compressed Finite State Machine (FSM): For structured output, SGLang compiles the JSON schema into a finite state machine that tracks which tokens are valid at each generation step. The "compressed" part is key: instead of checking one token at a time against the FSM, SGLang pre-computes multi-token jumps, allowing the model to advance several states in a single decode step when the grammar forces a deterministic sequence (like the characters in a JSON key name).
• Constrained decoding with overlap: SGLang overlaps FSM computation with GPU execution. While the GPU generates the next token, the CPU prepares the FSM mask for the subsequent step. This hides most of the FSM overhead behind GPU compute.

Deploying SGLang

The setup is similar to vLLM. SGLang also exposes an OpenAI-compatible API:

# Install
pip install "sglang[all]"

# Launch the server
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --dtype float16 \
    --context-length 4096 \
    --mem-fraction-static 0.88 \
    --port 8001

The --mem-fraction-static flag is SGLang's equivalent of vLLM's --gpu-memory-utilization. It controls how much VRAM to pre-allocate for the KV cache pool. I used 0.88 to leave similar headroom as my vLLM deployment.

On startup, SGLang logs the radix cache configuration and the number of available token slots. It also reports the maximum number of concurrent requests, which depends on the model size and available KV cache memory.

Baseline: unconstrained generation

First, I ran the same benchmark as yesterday against SGLang to establish a baseline. Same prompt, same output length, same concurrency sweep:

• Concurrency=1: TTFT about 42ms, decode at 90 tok/s. Comparable to vLLM.
• Concurrency=16: TTFT about 75ms, throughput at 1,100 tok/s. Slightly better than vLLM at the same concurrency, likely due to RadixAttention reusing the shared system prompt prefix across requests.
• Concurrency=32: TTFT about 110ms, throughput at 1,500 tok/s. Again competitive with vLLM.

For unconstrained generation, SGLang and vLLM are in the same ballpark. The differences are within measurement noise for most configurations. Both engines are mature and well-optimized for the standard serving workload.

Structured output: the JSON schema benchmark

Now the interesting part. I defined a JSON schema for a structured extraction task:

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {
            "type": "array",
            "items": {"type": "string"}
        },
        "experience_years": {"type": "number"},
        "is_senior": {"type": "boolean"}
    },
    "required": ["name", "age", "skills", "experience_years", "is_senior"]
}

The request uses the response_format parameter to enforce JSON output:

payload = {
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
        {"role": "system", "content": "Extract structured data from the text."},
        {"role": "user", "content": "John is a 35-year-old senior engineer ..."}
    ],
    "response_format": {
        "type": "json_schema",
        "json_schema": {"name": "person", "schema": schema}
    },
    "max_tokens": 256,
}

Measuring the structured output overhead

I ran 200 requests with and without the JSON schema constraint at concurrency=16 and compared:

• Without schema: Median TTFT 72ms, median decode speed 68 tok/s per request, median total time 1.9s for roughly 130 output tokens.
• With schema: Median TTFT 78ms, median decode speed 62 tok/s per request, median total time 1.1s for roughly 70 output tokens (the JSON is more compact than free-form text).

The per-token decode speed dropped by about 9% with constrained decoding. But total request time was actually lower because the structured output produces fewer tokens. JSON is dense: a structured response has less fluff than a free-form answer with explanation.

Where constrained decoding gets expensive

The overhead is not constant. It depends on the schema complexity:

• Simple schemas (flat objects with string/number fields): Minimal overhead. The FSM has few states, and the token mask is cheap to compute. I measured less than 5% decode speed reduction.
• Nested schemas (arrays of objects, recursive structures): More FSM states, larger token masks. Overhead around 10-15%.
• Enum-heavy schemas (fields restricted to specific string values): The FSM must intersect the enum values with the model's vocabulary at each step. This can be expensive for large enums. I saw up to 20% overhead with an enum of 50+ string values.

The key factor is how many tokens in the vocabulary are valid at each decode step. For unconstrained generation, all tokens are valid (or at least all tokens above the sampling threshold). For constrained generation, the FSM mask can eliminate 99%+ of the vocabulary. Computing and applying this mask takes CPU time, and if it is not overlapped with GPU computation, it becomes the bottleneck.

RadixAttention and prefix caching

One benefit of SGLang's RadixAttention became clear during the structured output benchmark: system prompt reuse. All 200 requests shared the same system prompt ("Extract structured data from the text."). SGLang cached the KV values for this prefix and reused them across all requests.

I verified this by checking the server logs, which report cache hit rates. With the shared system prompt, the radix cache hit rate was about 40% of input tokens across the benchmark. This translates directly to lower TTFT because those tokens do not need to be re-prefilled.

For workloads with long, shared system prompts (common in production, where system prompts can be hundreds of tokens), this is a significant advantage. The longer the shared prefix relative to the unique user message, the bigger the win.

SGLang vs. vLLM: when to use which

After benchmarking both, here is my working mental model:

• Use vLLM when you need the broadest model support, battle-tested stability, and a large community. vLLM supports more model architectures, more quantization formats, and has more production deployments. It is the safe choice.
• Use SGLang when structured output is a core use case, when prefix caching matters (shared system prompts, multi-turn conversations), or when you want to use its frontend language for complex LLM programs with branching and control flow.
• Both are excellent for standard serving. The throughput difference is in the single-digit percentages. Pick based on features, not raw speed.

Structured output is not free, but it is cheaper than most people think. The real cost is not the constraint itself. It is the tokens you waste when you do not use constraints and the model rambles, apologizes, or wraps your JSON in markdown code fences.

With both serving engines benchmarked, the next step is to compare them against compiled approaches. On day 48, I will put TensorRT-LLM up against eager PyTorch and see what compilation buys you.