Design the full inference stack

This is the architecture post. After 97 days of individual components, attention mechanisms, quantization schemes, scheduling algorithms, deployment patterns, it is time to draw the full picture. Not a theoretical reference architecture, but the system I would actually build if someone handed me a set of GPUs and said "serve this model in production."

Every choice here traces back to a specific post in this series. If you have been following along, this should feel like a puzzle where all the pieces finally click together.

The request lifecycle

A user sends a request. Here is every layer it touches:

Client
  → API Gateway (auth, rate limiting, routing)
    → Load Balancer (least-connections, KV-cache-aware)
      → Inference Server (vLLM / SGLang)
        → Scheduler (continuous batching + chunked prefill)
          → Model (quantized weights on GPU)
            → Attention (FlashAttention + PagedAttention)
              → KV Cache (paged blocks in HBM)
            → Output sampling
          → Response tokens (streamed via SSE)
        → Scheduler releases KV blocks
      → Load Balancer records latency
    → API Gateway logs usage
  → Client receives stream

Each layer has a job, and each job maps to a post. Let me walk through the design decisions at every layer.

Layer 1: API gateway

The gateway sits in front of everything and handles concerns that should never touch the GPU:

Authentication and rate limiting: API keys, per-user quotas, token budgets. This is standard web infrastructure. I use something like Kong, Envoy, or a lightweight FastAPI middleware.
Request validation: Check that the request has valid parameters, the model name exists, the max_tokens is within bounds. Reject bad requests before they consume GPU cycles.
Routing: Direct requests to the right model pool. If you serve multiple models, the gateway decides which cluster handles each request. For the small/large model routing pattern, the gateway or a lightweight classifier decides which model to use.
Observability: Log every request with a trace ID, timestamp, token counts, and latency. This feeds your eval harness and cost tracking.

Layer 2: Load balancer

The load balancer distributes requests across inference server replicas. This is where routing and queueing decisions happen:

Least-connections is the baseline: send to the replica with the fewest in-flight requests. This naturally balances load when request durations vary.
KV-cache-aware routing: If you have prefix caching enabled, route requests with similar prefixes to the same replica. A hash of the system prompt or first N tokens works as the routing key. This improves cache hit rates dramatically for workloads with shared context (e.g., same system prompt across all requests).
Health checks: The load balancer needs to know which replicas are healthy. Not just "is the process alive" but "is the model loaded and ready to serve." More on this in the implementation post.

Layer 3: Inference server

This is the core. I would use vLLM or SGLang as the inference engine, wrapped in a thin API layer. The inference server handles:

Continuous batching: New requests join the batch at every iteration. No waiting for a batch to fill. This is table stakes for production serving.
Chunked prefill: As we covered in day 94, long-context requests get their prefill broken into chunks so decode latency stays bounded.
PagedAttention: The KV cache is managed in pages, allocated on demand, freed when requests complete. No wasted memory from pre-allocation.
Streaming: Tokens are sent to the client as they are generated via server-sent events (SSE). The client sees the first token as soon as prefill finishes.

For the model itself, I would use INT8 or INT4 quantization depending on the quality/cost tradeoff. On Hopper GPUs, FP8 with vLLM's built-in support is the sweet spot: nearly lossless and 2x faster than FP16 for compute-bound operations.

Layer 4: GPU and memory

The GPU layer is where the roofline model determines everything:

Prefill: Compute-bound. Benefits from FP8/INT8 compute and large batch sizes.
Decode: Memory-bound. Benefits from weight quantization (fewer bytes to move) and speculative decoding (more tokens per weight load).
KV cache: The biggest memory consumer after the model weights. Block management and eviction policies determine how many concurrent requests you can serve.

For multi-GPU serving, tensor parallelism splits the model across GPUs within a node. For very large models (400B+), pipeline parallelism across nodes. For MoE models, expert parallelism.

Architecture decision

Should you disaggregate prefill and decode onto separate GPU pools? For most workloads, no. The complexity is not worth it. Disaggregation (day 18) makes sense when your traffic has a bimodal distribution: lots of long-context prefills mixed with latency-sensitive decode. Chunked prefill handles most of these cases without the operational overhead.

The complete stack diagram

┌─────────────────────────────────────────────┐
│  Clients (SDK / curl / browser)             │
└────────────────────┬────────────────────────┘
                     │ HTTPS
┌────────────────────▼────────────────────────┐
│  API Gateway                                │
│  - Auth, rate limits, request validation    │
│  - Model routing, usage logging             │
└────────────────────┬────────────────────────┘
                     │
┌────────────────────▼────────────────────────┐
│  Load Balancer                              │
│  - Least-connections + prefix-hash routing  │
│  - Health checks (liveness + readiness)     │
└──────┬─────────────┬──────────────┬─────────┘
       │             │              │
┌──────▼──────┐ ┌────▼────┐  ┌─────▼─────┐
│  vLLM       │ │  vLLM   │  │  vLLM     │
│  Replica 1  │ │  Rep. 2 │  │  Rep. 3   │
│             │ │         │  │           │
│  ┌────────┐ │ │ ┌─────┐ │  │ ┌───────┐ │
│  │Scheduler│ │ │ │Sched│ │  │ │ Sched │ │
│  │Chunked │ │ │ │     │ │  │ │       │ │
│  │Prefill │ │ │ │     │ │  │ │       │ │
│  └────────┘ │ │ └─────┘ │  │ └───────┘ │
│  ┌────────┐ │ │ ┌─────┐ │  │ ┌───────┐ │
│  │Model   │ │ │ │Model│ │  │ │ Model │ │
│  │INT4/FP8│ │ │ │     │ │  │ │       │ │
│  └────────┘ │ │ └─────┘ │  │ └───────┘ │
│  GPU 0..N   │ │ GPU 0..N│  │ GPU 0..N  │
└─────────────┘ └─────────┘  └───────────┘

Sizing the system

How do you decide how many replicas, how many GPUs per replica, and what quantization level? Start from the SLOs and work backward:

Define your SLOs: P99 TTFT under 2 seconds. P99 inter-token latency under 50ms. Throughput of 1000 requests per minute. These numbers come from product requirements.
Benchmark a single replica: Deploy one instance with your chosen model and quantization. Run a load test to find the saturation point: the request rate at which P99 latency exceeds your SLO.
Compute replicas needed: Target request rate divided by per-replica capacity, with 30% headroom for traffic spikes and maintenance. If one replica saturates at 50 req/min and you need 1000 req/min, you need 26 replicas (1000 / 50 * 1.3).
Check the cost: Multiply replicas by GPU cost. If it is too high, consider a smaller model, more aggressive quantization, or distillation.

Operational concerns

A production stack is not just the serving path. You also need:

Zero-downtime deploys: Blue-green or rolling deployments so model updates do not cause downtime. The new model needs to warm up (load weights, compile kernels) before receiving traffic.
Autoscaling: Scale replicas based on queue depth or GPU utilization, not CPU. GPU utilization is the scarce resource.
Monitoring: Track TTFT, inter-token latency, throughput, GPU memory utilization, KV cache hit rate, and request queue depth. Alert on SLO breaches. End-to-end latency breakdowns help you find the bottleneck.
Eval pipeline: Continuous evaluation against production to catch quality regressions.
Cost tracking: Track cost per request, cost per token, and cost per correct answer. Optimize for the business metric, not the engineering metric.

What I would actually deploy today

If I were starting a production inference stack today for a general-purpose LLM application, here is the specific stack:

Model: Llama 3.1 70B in FP8 on H100 GPUs, or Llama 3.1 8B fine-tuned if I have task-specific data.
Engine: vLLM with chunked prefill, prefix caching, and PagedAttention.
API layer: FastAPI with SSE streaming, health checks, and Prometheus metrics. (We build this in the next post.)
Infrastructure: Kubernetes with GPU node pools. Each pod runs one vLLM instance with tensor parallelism across the GPUs in the node.
Load balancing: Envoy or Nginx with least-connections routing and health check probes.
Observability: Prometheus + Grafana for metrics, structured logging for request traces.

The best inference stack is not the one with the most features. It is the one where every component earns its complexity. Start simple, measure everything, and add complexity only when measurements justify it.

Tomorrow: we build it. FastAPI, vLLM, health checks, and a Dockerfile. Real code, real configs, ready to deploy.