What I read, and why.
The high-quality resources behind this journey. Papers, docs, and books I actually read, not a link dump. Curated from the field's best.
Architecture
Attention is All You Need
The paper that started it all. Transformer architecture, self-attention, and why it beat RNNs. Still the foundation of every model I serve.
arxiv.org/abs/1706.03762FlashAttention: Fast and Memory-Efficient Exact Attention with IO Awareness
The kernel that changed inference. IO-aware tiling, online softmax, and why memory traffic beats FLOPs. I built a simplified version of it.
arxiv.org/abs/2205.14135RoFormer: Rotary Position Embedding
RoPE is how modern models know token order. It's in every model I serve, and it's the reason long context works.
arxiv.org/abs/2104.09864Inference Optimization Research
Efficient Memory Management for LLM Serving with PagedAttention
The vLLM paper. Virtual memory for the KV cache, near-zero fragmentation, and 2-4x throughput. The reason I use vLLM today.
arxiv.org/abs/2309.06180Fast Inference from Transformers via Speculative Decoding
The original speculative decoding paper. Draft and target, acceptance sampling, and the math I simulated.
arxiv.org/abs/2211.17192GPTQ: Accurate Post-Training Quantization
Round-to-nearest with Hessian weighting. The quantization algorithm I built, and the reason 4-bit models work.
arxiv.org/abs/2210.17323EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
Feature-level drafting. The successor to plain speculative decoding, and what I covered in the advanced phase.
arxiv.org/abs/2401.15077Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
The honest look at splitting prefill and decode. When it helps, when it doesn't, and the real tradeoffs. From 2025.
arxiv.org/abs/2506.05508Developer Tools
vLLM
The serving engine I use daily. PagedAttention, continuous batching, prefix caching. The default choice for a reason.
github.com/vllm-project/vllmSGLang
RadixAttention and structured output. The engine that made prefix caching a first-class feature.
github.com/sgl-project/sglangNVIDIA Dynamo
Disaggregated serving made practical. Prefill and decode on separate machines, orchestrated by a scheduler.
docs.nvidia.com/dynamoLMCache
KV cache management as a service. The layer that makes prefix caching work across replicas.
github.com/LMCache/LMCacheGPU Infrastructure
Programming Massively Parallel Processors
The CUDA bible. From thread blocks to memory coalescing. If you want to understand why kernels are fast, this is where you start.
goodreadsSite Reliability Engineering
Google's book on running production services. The SLOs, error budgets, and postmortems that apply directly to inference.
sre.google/booksGPU Glossary
Modal's excellent glossary of GPU terms. The reference I keep open when the acronyms get thick.
modal.com/gpu-glossaryNVIDIA Blackwell Architecture Technical Brief
What the next generation of GPUs actually does. FP4, NVLink 5, and the reasoning-era design choices.
resources.nvidia.comAWQ: Activation-aware Weight Quantization
Only ~1% of weight channels are salient, and they're found by looking at activations, not weights. Scale them up before rounding and you keep uniform INT4. The default in vLLM and TensorRT-LLM.
arxiv.org/abs/2306.00978Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
The draft model was inside the target all along. Multiple MLP heads, tree-based attention, verified in a single pass. 2.2-3.6x speedup, no second model to deploy.
arxiv.org/abs/2401.10774SGLang: Efficient Execution of Structured Language Model Programs
The RadixAttention paper. A radix tree over the KV cache, co-designed with a structured generation DSL. Up to 5x on multi-call programs.
arxiv.org/abs/2312.07104TensorRT-LLM: In-flight batching with chunked context
Three queues, chunked prefill interleaved with decode, and the whole transformer compiled into fused kernels. Scheduling outside the graph, speed inside it.
nvidia.github.io/TensorRT-LLMObservability
OpenTelemetry GenAI Semantic Conventions
The emerging standard for LLM telemetry. gen_ai.request.model, gen_ai.usage.input_tokens, and first-class TTFT and time-per-output-token histograms. Instrument once, export anywhere.
opentelemetry.io/docs/specs/semconv/gen-aivLLM Production Metrics
The /metrics endpoint that powers real dashboards. gpu_cache_usage_perc, num_preemptions_total, request_queue_size. Alert on preemptions, correlate cache pressure with TTFT spikes.
docs.vllm.ai/en/latest/serving/metricsLangfuse: trace-level LLM observability
Infrastructure metrics tell you that latency is high. Langfuse tells you why, which prompts are slow, which model regressed, and which users are affected. Cost per trace, quality scores, open source.
langfuse.com/docs/analyticsProduction Systems
Baseten Delivery Network: fast cold starts for big models
Cut cold starts 2-3x for models with billions or trillions of parameters, even when burst scaling to 1,000s of replicas. The cold-start problem, solved at the network layer.
baseten.co/blogBaseten multi-cloud capacity management (MCM)
Unify GPU capacity across clouds, run inference in ours, yours, or both. 99.99% uptime, low latency, compliance-ready. The multi-cloud GPU strategy in practice.
baseten.co/blogUber: Scaling AI/ML Infrastructure
How one of the world's biggest real-time systems scales AI/ML infrastructure. The reliability lessons that apply to inference at scale.
uber.com/blogFrom the labs (2026)
The inference-engineering posts actually worth reading from the teams building the engines. Collected May-Aug 2026.
Pushing the Limits of Serving DeepSeek-V4-Pro
SGLang on a 1.6T-parameter MoE with FP8 and FP4 weights. The masterclass in serving a frontier model at extreme scale, with the Blackwell optimizations that make it work.
lmsys.org · Aug 19, 2026Advanced CUDA Graph Techniques in SGLang
Graphing as much of the workload as possible to kill kernel-launch overhead, without breaking compatibility. Breakable CUDA graphs and their production tradeoffs.
lmsys.org · Aug 17, 2026Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
A single radix tree managing KV cache reuse across full attention and MLA/sliding-window heads. The cache structure for hybrid-attention models like Qwen3.8.
lmsys.org · Aug 11, 2026SpecForge v0.3.0: Unified Disaggregated & Colocated Speculative Decoding
Speculative decoding training framework with SpecBundle draft models, now supporting both disaggregated and colocated EAGLE3 deployment. Directly integrated with SGLang.
lmsys.org · Aug 4, 2026HPC-Ops × SGLang: Attention, Router GEMM, and MoE Kernels from Tencent Hunyuan
Battle-tested production kernels from Tencent's large-scale inference: Dynamic Attention and Fused MoE. Real optimizations now in SGLang.
lmsys.org · Aug 7, 2026Towards Blackwell-Native 8-bit and 4-bit RL: End-to-End MXFP8 and NVFP4 RL in Miles
First implementation of Blackwell-native MXFP8 and NVFP4 RL. The models it produces are natively quantized for serving, which is why it matters for inference.
lmsys.org · Jul 29, 2026RadixArk × Google: Bringing Full SGLang Features to TPUs
The full SGLang inference stack now runs on Google TPUs with the same feature set as NVIDIA GPUs. Hardware-flexible serving, in practice.
lmsys.org · Jul 30, 2026MoonEP: Perfectly Balanced Expert Parallelism via Dynamic Redundant Experts
Moonshot AI's open-source expert parallelism library. Dynamic redundant experts for near-perfect load balancing on MoE inference. The Kimi K2/K3 serving stack.
github.com/MoonshotAI/MoonEP · Jul 2026DeepSpec: Full-Stack Speculative Decoding from DeepSeek
DeepSeek's complete codebase for training and evaluating speculative decoding algorithms, from draft model training to deployment. Spec-decode as a first-class optimization.
github.com/deepseek-ai/DeepSpec · Jun 2026FlashKDA: High-Performance Kimi Delta Attention Kernels
Moonshot's custom Delta Attention used in Kimi-K2/K3, with flash-attention-style GPU kernels. More KV-cache-efficient than standard MHA, and the key to their long-context serving.
github.com/MoonshotAI/FlashKDA · 2026MiniMax H3: Open Model Breaking Boundaries Between Tasks and Modalities
MiniMax's new open multimodal model, released with MSA (their custom attention library, open-sourced June 2026). Attention designed for efficient inference across text, image, and video.
minimax.io · Jul 31, 2026Full-Stack Performance Optimization of AR+DiT in SGL-Diffusion
Replacing the HuggingFace backend with SRT to accelerate autoregressive + diffusion image models. Dedicated TP for AR, sequence-parallelism for DiT, dynamic batching. SGLang beyond text.
lmsys.org · Aug 5, 2026SGLang Day-0 Support for Qwen3.8-2.4T-A95B
2.4T total parameters, 95B active per token, hybrid attention (full + sliding window). Day-0 support in SGLang for the largest open model. This is what "ready at launch" looks like.
lmsys.org · Aug 12, 2026How GPT-5.6 fuses frontier intelligence with frontier efficiency
OpenAI's engineering post: load balancing, kernel optimization, speculative decoding, and workload-specific KV config, all driven by GPT-5.6 Sol in Codex. Sol rewrote production kernels in Triton/Gluon for a 20% cost cut, and designed its own draft model for +15% token efficiency. The agentic harness keeps prompt-cache hit rates high with append-only history.
openai.com · Jul 29, 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
THUDM/Zhipu's optimization for sparse attention that reuses index patterns across transformer layers. Relevant for GLM-4.5-class models using sparse attention for long-context efficiency.
github.com/THUDM/IndexCache · 2026Books
AI Engineering: Building Applications with Foundation Models
Chip Huyen's book. The breadth-first companion to this journey, from evals to deployment.
O'ReillyBuild a Large Language Model (From Scratch)
Sebastian Raschka's hands-on LLM architecture book. The code companion to the deep-implementation phase.
ManningAI Systems Performance Engineering
Chris Fregly's new book on building for performance with GPUs, CUDA, and PyTorch. The performance-focused companion.
O'Reilly