Zero-downtime deploys and cost

In traditional web services, a deploy takes seconds: pull the new container, health check passes, swap traffic. The old pod drains and dies. Total overlap time is maybe 30 seconds per pod. The GPU cost of that overlap is effectively zero because CPUs are cheap.

LLM inference deploys are different in every dimension. Loading a 70B model into GPU memory takes 2 to 5 minutes. CUDA context initialization adds more time. Warmup (running a few dummy requests to trigger JIT compilation in TensorRT-LLM or torch.compile) adds another 30 to 60 seconds. During all of this, the new replica is occupying a GPU but not serving traffic. And GPUs cost $2-4 per hour per card. That overlap is expensive.

Blue-green deployment

Blue-green is the simplest zero-downtime strategy: run two complete environments, switch traffic from one to the other atomically.

• Blue is the current production fleet serving traffic.
• Green is an identical fleet running the new model version, fully loaded and warmed up.
• Once green passes health checks and smoke tests, you flip the load balancer to point at green.
• Blue drains its in-flight requests, then shuts down.

The advantage is simplicity and instant rollback: if green has a problem, flip back to blue. The disadvantage is cost: you need double the GPU capacity during the transition. For a fleet of 8 nodes, each with 8 H100s (64 GPUs total), blue-green means provisioning another 64 GPUs for the duration of the deploy. At $3/GPU/hr, a 30-minute deploy costs an extra $96 just in GPU overlap.

# Blue-green deploy sequence (pseudocode)
def blue_green_deploy(new_version):
    green = provision_fleet(new_version, size=FLEET_SIZE)
    for replica in green:
        wait_for_model_load(replica)        # 2-5 min per replica
        wait_for_warmup(replica)            # 30-60s
        assert health_check(replica)

    # Atomic traffic switch
    load_balancer.set_backend(green)

    # Drain blue
    blue = get_current_fleet()
    for replica in blue:
        replica.stop_accepting()
        wait_for_drain(replica, timeout=120)
        replica.shutdown()

Rolling deployment

Rolling deploys update replicas one (or a few) at a time. You remove a replica from the pool, update it, reload the model, warm it up, add it back, then move to the next.

The cost advantage is significant: you only need one extra replica's worth of GPU capacity at any time (or zero extra if you're willing to temporarily reduce capacity). The tradeoff is that the deploy takes much longer, and you're running a mixed fleet of old and new versions during the rollout.

• Capacity dip. While a replica is reloading, your serving capacity drops. If you have 8 replicas and update one at a time, you're at 87.5% capacity during each step. This is fine if you have headroom, dangerous if you're already near saturation.
• Version skew. During a rolling deploy, some replicas serve v1 and others serve v2. If the model versions produce meaningfully different outputs, this can cause inconsistent user experiences. For most LLM updates this is acceptable, but for safety-critical changes it might not be.
• Duration. With 8 replicas and 5 minutes per reload, a serial rolling deploy takes 40 minutes. You can parallelize (update 2 at a time) to cut that in half, at the cost of a deeper capacity dip.

The cold start problem

The dominant cost in both strategies is the cold start: the time between "GPU allocated" and "replica serving traffic." For LLM inference, cold start includes:

• Weight download: pulling the model from object storage. For a 70B model in safetensors format, this is about 140 GB. On a 25 Gbps network link, that's roughly 45 seconds. From local NVMe cache, seconds.
• Weight loading: deserializing weights and copying them to GPU memory via model.load_state_dict() or the serving engine's loader. This is CPU and PCIe bandwidth bound, typically 60 to 180 seconds.
• CUDA warmup: the first inference request triggers kernel compilation and memory allocation. Running a few warmup requests ensures the first real user doesn't pay this cost.
• TensorRT compilation (if applicable): TensorRT-LLM builds optimized engines at startup. This can take 5 to 15 minutes for large models unless you pre-compile and cache the engine files.

Cost comparison

Let's compare strategies for a fleet of 8 replicas, each using 1 H100 at $3/hr:

• Blue-green: 8 extra GPUs for 30 minutes = $12 per deploy. Fast, safe, expensive.
• Rolling (1 at a time): 1 extra GPU for 40 minutes = $2 per deploy. Slow, cheap, capacity-sensitive.
• Rolling (2 at a time): 2 extra GPUs for 20 minutes = $2 per deploy. Moderate speed and cost.
• In-place (no extra GPUs): $0 extra, but you lose 12.5% capacity per step and risk failed loads.

For teams deploying daily, the cost difference between blue-green ($360/month) and rolling ($60/month) is real but not enormous relative to the base fleet cost ($17,280/month for 8 H100s running 24/7). Most teams optimize deploys for speed and safety, not cost.

Graceful drain and request handling

The detail that makes or breaks zero-downtime: how you handle in-flight requests during the switch. LLM requests can take 10 to 60 seconds for long generations. You cannot just kill the old replica.

# Graceful drain pattern
async def drain_replica(replica, timeout_seconds=120):
    replica.stop_accepting_new_requests()

    start = time.time()
    while replica.in_flight_count() > 0:
        if time.time() - start > timeout_seconds:
            # Force-cancel remaining requests, client retries
            replica.cancel_all()
            break
        await asyncio.sleep(1)

    replica.shutdown()

The load balancer should remove the draining replica from rotation before it starts draining. The replica finishes its in-flight requests, then exits. If a request is still running after the drain timeout (say, 2 minutes), you have a choice: force-cancel it (the client retries on a new replica) or extend the timeout. Most teams force-cancel with a generous timeout.

My recommendation

For most inference teams, rolling deploys with one or two replicas updating at a time are the right default. Blue-green is worth the cost if you need instant rollback or cannot tolerate any version skew. Pre-cache your weights. Pre-compile your engines. And always, always test your drain logic under load before you need it in production.

Next: benchmarking tooling and profiling, because you can't improve what you don't measure.