Autoscaling: concurrency, batching, cold starts

Autoscaling a web application is straightforward: watch CPU utilization, add instances when it climbs above 70%, remove them when it drops. Autoscaling LLM inference is nothing like that. The fundamental challenge is that GPU utilization is a terrible signal, model loading takes minutes instead of seconds, and the relationship between concurrency and latency is nonlinear in ways that surprise you.

Why GPU utilization lies

GPU utilization as reported by nvidia-smi measures what fraction of time at least one kernel is running on the GPU. This number is almost always above 90% during inference, even when the GPU has plenty of spare capacity. The reason: continuous batching keeps the GPU busy with a steady stream of decode iterations, even if the batch is small and throughput is well below maximum.

A GPU at "99% utilization" might be serving 10 requests with a batch size of 10, generating 100 tokens per second. The same GPU could serve 50 requests with a batch size of 50, generating 400 tokens per second, and still report 99% utilization. The metric does not distinguish between these states.

Better signals for autoscaling LLM inference:

• Queue depth: how many requests are waiting to be processed. This directly measures demand that is not being met.
• KV cache utilization: what fraction of the KV cache memory is in use. When this approaches 100%, new requests will be rejected or preempted. vLLM exposes this via /metrics.
• Time-to-first-token (TTFT) at p95: when TTFT starts climbing, the prefill queue is backing up. This is a user-facing latency metric that directly maps to experience.
• Active batch size / max batch size: the ratio of the current continuous batch size to the configured maximum. When this approaches 1.0, you are at capacity.

# vLLM Prometheus metrics useful for autoscaling
vllm:num_requests_waiting        # queue depth
vllm:gpu_cache_usage_perc        # KV cache utilization (0-1)
vllm:num_requests_running        # active concurrent requests
vllm:e2e_request_latency_seconds # end-to-end latency histogram

Concurrency and the batching curve

LLM inference has a unique throughput-latency relationship driven by the roofline. At low concurrency (few in-flight requests), the GPU is memory-bandwidth-bound during decode. Each token takes the same time regardless of batch size because you are loading the same model weights either way. Adding more requests to the batch is essentially free throughput, because the weight-loading cost is amortized.

This holds until you hit a knee in the curve. At some batch size, the decode step transitions from memory-bound to compute-bound. Beyond this point, adding more requests increases both throughput and latency. The knee depends on your model size, GPU, and precision:

• Llama 3 8B FP8 on H100: the knee is around batch size 64 to 128. Below this, adding requests is nearly free.
• Llama 3 70B FP8 on 8xH100: the knee is around batch size 16 to 32, because the model is larger and the per-token compute is higher.

A good autoscaler understands this curve. Before the knee, absorb more requests on existing replicas. After the knee, scale out to new replicas. Scaling out too early wastes GPUs. Scaling out too late spikes latency.

Cold starts: the GPU tax

Cold start is the time from "new replica requested" to "first request served." For LLM inference, this includes:

• Container image pull: 30 seconds to 5 minutes depending on image size, registry proximity, and network bandwidth. A 20 GB image over 1 Gbps takes ~160 seconds.
• Model loading: loading weights from disk (or network) into GPU memory. For a 70B FP16 model (140 GB), loading from a local NVMe SSD at 7 GB/s takes 20 seconds. From a network file system, it could be minutes.
• Engine compilation (TensorRT-LLM only): if you are building TRT engines at startup, add 10 to 60 minutes. Pre-compile and cache engines to avoid this.
• Warmup: the first few requests may be slower due to CUDA context initialization, JIT compilation, and cache population. Some deployments run a synthetic warmup request before marking the replica as ready.

Total cold start time for LLM inference is typically 1 to 5 minutes, compared to seconds for a typical web service. This changes how you think about scaling:

• Predictive scaling: do not wait for the traffic spike to arrive. Use time-of-day patterns or upstream signals (like a batch of documents entering a RAG pipeline) to pre-scale.
• Warm pools: keep a small number of idle replicas with models already loaded. They burn GPU-hours, but they respond instantly. Think of it as insurance.
• Model weight caching: cache model weights on local NVMe at each node. Even if the container is pulled fresh, loading from local disk is 10x faster than from S3 or a network file system.

Scale-to-zero

For development, staging, and low-traffic models, scale-to-zero saves significant cost. But the cold start penalty makes it impractical for latency-sensitive production workloads. A compromise: scale to a minimum of 1 replica during business hours and zero overnight. The first request of the day takes the cold start hit, but the rest of the day runs at warm latency.

# KEDA ScaledObject for LLM inference
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference
spec:
  scaleTargetRef:
    name: llm-deployment
  minReplicaCount: 1    # never fully cold during business hours
  maxReplicaCount: 8
  cooldownPeriod: 300   # 5 min cooldown before scaling down
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_queue_depth
      query: sum(vllm:num_requests_waiting{deployment="llm"})
      threshold: "10"   # scale up when queue > 10

Putting it together

A practical autoscaling strategy for LLM inference:

1. Primary signal: KV cache utilization. Scale up at 80%, scale down at 30%.
2. Secondary signal: request queue depth. Scale up when requests are waiting longer than your TTFT SLA.
3. Cooldown: 5 minutes minimum. The cost of an idle GPU for 5 minutes is far less than the cost of a cold start oscillation cycle.
4. Minimum replicas: at least 1 during expected traffic hours. Scale-to-zero only for non-production.
5. Model weight caching: always cache on local NVMe. The model should be loadable from local disk in under 30 seconds.

Autoscaling LLM inference is less about reacting to load and more about predicting it. The cold start penalty is so high that reactive scaling always arrives late. The best autoscaler is the one that adds capacity five minutes before you need it.

This wraps up the infrastructure phase. From here, we move into tooling: benchmarking and profiling, the tools you need to measure everything we have discussed.