Multi-cloud capacity

The GPU shortage of 2023-2024 taught every inference team the same lesson: if your capacity plan depends on a single cloud provider, you will get burned. Reserved instances sell out. Spot capacity disappears during training runs by large labs. Entire regions run dry of H100s for weeks. The solution is multi-cloud capacity, and it's less about ideology ("avoid vendor lock-in!") and more about survival ("we need GPUs, anywhere, right now").

Why multi-cloud for inference specifically

Multi-cloud for web services has been debated for years and often dismissed as unnecessary complexity. Inference is different for a few concrete reasons:

• GPU scarcity. Unlike CPUs, GPU capacity is genuinely constrained. AWS, GCP, Azure, OCI, CoreWeave, Lambda Labs, and others each have different inventory at different times. Spreading across providers is portfolio diversification against stockouts.
• Pricing variance. An A100-80GB on-demand ranges from roughly $2/hr to $4/hr depending on provider and region. Spot and preemptible pricing varies even more. Multi-cloud lets you chase the cheapest available capacity.
• Latency geography. If you serve users globally, you want inference endpoints near them. No single cloud has GPU instances in every region. GCP might have H100s in asia-southeast1 when AWS doesn't have them in ap-southeast-1.

The architecture: a unified control plane

The key architectural decision is where to put the routing layer. You need a single control plane that knows about all your GPU pools across clouds and can route requests intelligently. This looks like:

# Simplified multi-cloud routing
clouds = {
    "aws-us-east-1":    {"gpus": "8xH100", "replicas": 4, "healthy": 4, "cost_per_hr": 25.0},
    "gcp-us-central1":  {"gpus": "8xH100", "replicas": 2, "healthy": 2, "cost_per_hr": 22.0},
    "coreweave-us-east": {"gpus": "8xH100", "replicas": 6, "healthy": 5, "cost_per_hr": 18.0},
}

def route_request(request, strategy="cheapest"):
    available = {k: v for k, v in clouds.items() if v["healthy"] > 0}
    if strategy == "cheapest":
        return min(available, key=lambda k: available[k]["cost_per_hr"])
    elif strategy == "nearest":
        return nearest_by_latency(request.source_region, available)
    elif strategy == "least_loaded":
        return min(available, key=lambda k: load_metric(k))

In practice, frameworks like SkyPilot, Anyscale, or custom Kubernetes federation layers handle this. The control plane needs to track health, queue depth, and cost across all providers and make routing decisions in single-digit milliseconds.

Challenges that actually bite

Multi-cloud sounds clean on a whiteboard. Here's where it gets messy:

• Model weight distribution. A 70B parameter model in FP16 is roughly 140 GB. Every time you spin up a new replica on a new cloud, you need to pull those weights. If your weights live in an S3 bucket in us-east-1 and your new replica is on GCP in europe-west4, that download takes minutes and costs real egress fees. Solution: replicate weights to each cloud's object storage ahead of time. Keep checksums. Automate the sync.
• Inconsistent GPU types. An "H100" on AWS (p5 instances) and an "H100" on GCP (a3 instances) have the same GPU silicon but different networking, CPU ratios, and NVLink topologies. You need to validate that your serving engine performs equivalently on each cloud's instance type. I've seen 15% throughput differences on "identical" GPUs across providers due to CPU bottlenecks in tokenization or different PCIe configurations.
• Networking and egress. Cross-cloud traffic is expensive ($0.08-0.12/GB) and adds latency. You don't want inference requests crossing cloud boundaries unless there's no capacity locally. The routing layer should strongly prefer same-cloud responses.
• Observability fragmentation. Metrics from CloudWatch, Cloud Monitoring, and a bare-metal provider's Prometheus don't naturally join together. You need a unified observability stack (Grafana with multiple data sources, or Datadog) to see your global fleet as one system.

Spot and preemptible: the cost lever

The biggest cost savings in multi-cloud inference come from using spot/preemptible GPU instances for non-latency-critical workloads. Spot H100s can be 60-70% cheaper than on-demand. The catch: they can be reclaimed with 30 seconds to 2 minutes of notice.

For inference, this works if:

• Your system can drain requests from a spot instance gracefully (stop accepting new requests, let in-flight requests complete, then release).
• You have enough on-demand "base" capacity to absorb the load during spot reclamation.
• Your autoscaler can provision replacement spot instances quickly, potentially on a different cloud.

Container portability

Multi-cloud only works if your inference workload is portable. This means containerized serving with no cloud-specific dependencies baked in. NIMs and container packaging help here: a single container image with the model weights, serving engine, and health checks can run on any cloud with NVIDIA GPU support.

The stack I've seen work best:

• Model weights in a cloud-agnostic format (safetensors or GGUF), synced to each provider's object storage.
• A single Dockerfile with the serving runtime (vLLM, TensorRT-LLM, SGLang).
• Kubernetes on each cloud, with a federation layer or a meta-scheduler like SkyPilot.
• A global load balancer (Cloudflare, Fastly, or a custom DNS-based solution) in front.

When not to go multi-cloud

Multi-cloud adds real operational overhead. If you can get reliable reserved capacity on a single provider, that's simpler and often cheaper per GPU-hour than the engineering cost of running across clouds. Multi-cloud makes sense when:

• You need more GPUs than any single provider can guarantee.
• You serve users in regions where your primary cloud has no GPU presence.
• You want to use spot pricing aggressively and need fallback capacity elsewhere.
• Regulatory requirements mandate data residency in specific countries.

For everyone else, a single cloud with reserved instances and a good autoscaling setup is the right starting point. Add clouds when the pain justifies the complexity.

Next: zero-downtime deploys and cost, because getting a model update out without dropping requests is its own challenge.