Skand Tandon
Production Systems

Multi-cloud geo-aware routing

GPU capacity is scarce and unevenly distributed. When you run inference across AWS, GCP, and Azure in multiple regions, routing decisions determine whether users get 50ms or 500ms latency. Here is how I think about geo-aware routing for LLM inference.

2025-12-10 production systems 8 min read

After building the async batch client, I started running benchmarks against inference endpoints in different regions. The results were striking: the same model, same hardware, same prompt, but TTFT varied by 10x depending on which region handled the request. Most of that variance was pure network latency. That is when I started thinking seriously about routing.

In traditional web services, geo-routing is a solved problem: put your app behind a CDN, use DNS-based routing, done. LLM inference is different because the backend is not stateless. GPU instances are expensive, capacity is limited, and you cannot just spin up a new replica in 200ms like you can with a container running a REST API. This post covers the routing strategies I have found useful when running inference across multiple clouds and regions.

Why multi-cloud in the first place?

Three reasons come up repeatedly:

The routing layer

The simplest architecture that works is a lightweight proxy at the edge that makes routing decisions based on three signals: client location, backend health, and backend load.

# Simplified routing logic
def select_backend(client_region: str, backends: list[Backend]) -> Backend:
    """Pick the best backend for this request."""
    # Filter to healthy backends only
    healthy = [b for b in backends if b.is_healthy]
    if not healthy:
        raise NoCapacityError("All backends down")

    # Sort by composite score: latency + load penalty
    def score(b: Backend) -> float:
        latency_ms = REGION_LATENCY[client_region][b.region]
        load_penalty = b.queue_depth * 10  # ms per queued request
        return latency_ms + load_penalty

    return min(healthy, key=score)

The REGION_LATENCY table is a static matrix of measured RTT between regions. I populate it by running periodic pings between the router locations and the backends. It does not change often. The queue_depth comes from the inference server's metrics endpoint, which vLLM and SGLang both expose.

Client location detection

There are two practical approaches:

I prefer the edge PoP approach. A Cloudflare Worker that inspects cf.colo and cf.country headers knows exactly where the request entered the network, with no database lookup required.

Health checks for GPU backends

Standard HTTP health checks are necessary but not sufficient for inference backends. A vLLM server can return 200 on /health while its KV cache is full and every new request would be queued for minutes. You need deeper health signals:

# Backend health check
async def check_backend_health(backend: Backend) -> HealthStatus:
    try:
        async with aiohttp.ClientSession() as session:
            # Check basic health
            async with session.get(
                f"{backend.url}/health", timeout=5
            ) as resp:
                if resp.status != 200:
                    return HealthStatus.DOWN

            # Check metrics for load
            async with session.get(
                f"{backend.url}/metrics", timeout=5
            ) as resp:
                metrics = parse_prometheus(await resp.text())

            cache_usage = metrics["vllm:gpu_cache_usage_perc"]
            queue_depth = metrics["vllm:num_requests_waiting"]

            if cache_usage > 0.95:
                return HealthStatus.DEGRADED
            if queue_depth > 100:
                return HealthStatus.DEGRADED

            return HealthStatus.HEALTHY
    except Exception:
        return HealthStatus.DOWN

Latency vs. cost: the tradeoff knob

Pure latency-based routing sends everything to the nearest region. That is correct for user-facing chat, but wasteful for batch workloads. If you are running 5,000 evaluation prompts, you do not care about 150ms of network latency. You care about total throughput and cost.

I use a simple flag in the request to signal intent:

{
  "model": "meta-llama/Llama-3.1-8B",
  "prompt": "...",
  "routing_hint": "lowest_cost"  // or "lowest_latency"
}

The router interprets this: lowest_latency picks the nearest healthy backend. lowest_cost picks the backend on the cheapest spot instance, regardless of region. For batch jobs, routing to the cheapest available region can save 20-40% on GPU costs.

Practical note

Spot pricing data is available via each cloud's API (AWS describe-spot-price-history, GCP compute.machineTypes.list with preemptible pricing). I poll this every 5 minutes and cache it in the router. Prices shift slowly enough that stale data by a few minutes is fine.

Failover and draining

When a backend goes down, the router needs to redirect in-flight requests without losing them. For non-streaming requests, this is straightforward: retry on a different backend. For streaming responses, it is harder because you cannot seamlessly resume a partially-generated response from a different server.

My approach:

DNS vs. application-layer routing

DNS-based routing (Route53 latency routing, Cloud DNS geolocation policies) is simple to set up but coarse. DNS TTLs mean failover takes 30-60 seconds minimum. You cannot route based on server load because DNS has no visibility into backend metrics.

Application-layer routing (a reverse proxy like Envoy, Nginx, or a custom service) gives you full control. You can route per-request, inspect headers, check backend metrics in real time, and fail over instantly. The cost is that you need to run and maintain the proxy.

For LLM inference, application-layer routing wins. The requests are long-lived (streaming can last seconds), the backends are expensive (you cannot afford to waste capacity due to stale DNS), and the routing decisions are complex (latency + cost + load + model availability).

The best routing decision is the one you do not have to make. If you can afford to run the same model in three regions with enough capacity in each, simple nearest-region routing handles 90% of cases. Complexity only pays off at scale, when GPU costs are high enough that a 20% savings justifies the engineering.

Tomorrow I will look inside the GPU itself and explore memory profiling to understand where the bytes actually go during inference.