Blue-green deployment for inference

Deploying a new model version in production is one of those tasks that sounds routine until you realize that loading a 70B parameter model onto GPUs takes minutes, not seconds. During that loading time, you either drop requests or serve stale results. Blue-green deployment is the pattern I have found most practical for avoiding this entirely.

The core idea

Blue-green deployment maintains two identical environments: "blue" (the current live version) and "green" (the new version being deployed). Traffic flows to blue while green loads the new model, warms up, and passes health checks. Once green is ready, a load balancer or router flips traffic from blue to green. If something goes wrong, you flip back. Blue sits idle as a rollback target until the next deployment.

For a standard web service, this is straightforward. For GPU inference, the "warm up" step is the challenge. Loading model weights into GPU memory, compiling CUDA graphs, and warming the KV cache all take significant time.

Why cold starts matter here

A typical cold start sequence for a vLLM instance serving a 13B FP16 model on an H100:

• Download model weights (if not cached): 2 to 10 minutes depending on network and model size.
• Load weights to GPU: 15 to 30 seconds for 26 GB of weights over PCIe Gen5.
• CUDA graph capture (if enabled): 20 to 60 seconds. vLLM pre-captures graphs for common batch sizes.
• First request warmup: the first request triggers JIT compilation of any remaining kernels. This request is slow.

Total cold start: anywhere from 1 to 12 minutes. During a rolling deployment without blue-green, some fraction of your replicas are in this cold start phase, either not serving or serving with degraded latency. I have seen P99 latency spike by 10x during rolling updates because a newly started replica gets routed traffic before it has finished warming up.

Implementation with Kubernetes

The simplest Kubernetes implementation uses two Deployments and a Service. The Service selector determines which Deployment receives traffic:

# blue deployment (currently live)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving-blue
  labels:
    app: llm-serving
    slot: blue
spec:
  replicas: 4
  selector:
    matchLabels:
      app: llm-serving
      slot: blue
  template:
    metadata:
      labels:
        app: llm-serving
        slot: blue
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.0
        args:
          - --model=meta-llama/Llama-2-13b-hf
          - --dtype=float16
        resources:
          limits:
            nvidia.com/gpu: 1
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10

# Service pointing to blue
apiVersion: v1
kind: Service
metadata:
  name: llm-serving
spec:
  selector:
    app: llm-serving
    slot: blue   # flip this to "green" to cut over
  ports:
  - port: 80
    targetPort: 8000

The deployment process:

1. Deploy the green Deployment with the new model version.
2. Wait for all green pods to pass readiness probes (which should include a test inference request, not just a TCP check).
3. Run a smoke test against the green pods directly (bypassing the Service).
4. Patch the Service selector from slot: blue to slot: green.
5. Monitor error rates and latency. If anything looks wrong, patch back to slot: blue.
6. Once confident, scale down the blue Deployment to free GPUs.

The readiness probe trap

The default readiness probe for most inference servers is a simple health endpoint that returns 200 once the HTTP server is up. This is not sufficient. The HTTP server starts before model weights are fully loaded and before CUDA graphs are captured. I have seen pods marked "ready" that returned 503s or 30-second latencies on their first requests.

A proper readiness probe for inference should actually run inference:

# A readiness check that actually runs inference
#!/bin/bash
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
  -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-2-13b-hf","prompt":"test","max_tokens":1}')

if [ "$RESPONSE" = "200" ]; then
  exit 0
else
  exit 1
fi

This is slower and heavier than a simple GET /health, but it guarantees the pod is actually warm before it receives real traffic.

Cost considerations

Blue-green deployment doubles your GPU footprint during the transition period. For a fleet of 8 H100s, that means you need 16 H100s available during deployment. At on-demand cloud pricing, H100s cost roughly $3/hour each. An 8-GPU blue-green deployment that takes 15 minutes to complete costs about $6 extra in GPU time. That is a rounding error compared to the cost of a botched rolling update that spikes latency for your users.

Some teams keep the blue fleet warm permanently as a hot standby for rollback. This is expensive but justified for critical services. Others scale blue to zero after a bake period (say, 30 minutes of clean green metrics) and accept the cold start penalty for rollback.

Canary as an extension

Blue-green is an all-or-nothing cutover. For higher-stakes changes (new model architecture, major quantization change), a canary deployment is safer. Route 5% of traffic to the green fleet, monitor quality metrics and latency, then gradually increase. This requires a smarter load balancer (like Istio or Envoy with weighted routing) but the infrastructure pattern is the same: green runs alongside blue, you just control the traffic split.

# Istio VirtualService for canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: llm-serving
spec:
  hosts:
  - llm-serving
  http:
  - route:
    - destination:
        host: llm-serving-blue
      weight: 95
    - destination:
        host: llm-serving-green
      weight: 5

Start with blue-green for model version bumps. Graduate to canary when you are changing model architecture or quantization, where output quality might shift in ways that latency metrics alone will not catch.

Next up I will jump to load testing with Locust to find the saturation point where the model starts queuing and latency degrades.