Autoscaling is the discipline of answering one question: how many replicas do I need right now? Get it wrong in one direction and you're paying for idle GPUs. Get it wrong in the other and users wait, or worse, time out.
I built a simulator to stop guessing. This is what it taught me.
The inputs
To size replicas you need three numbers:
- Requests per second (RPS) hitting the service.
- Throughput per replica - how many requests one GPU-backed replica can serve per second. For LLMs this is usually a function of batch size and sequence length, not a fixed number.
- Cold start time - how long a new replica takes to become ready. For LLMs, minutes, because the model weights must load into GPU memory.
Plus the policy: how much headroom you keep, and how fast you react.
The naive answer (wrong)
Most people start with: replicas = ceil(RPS / per_replica). That's wrong for two reasons:
- It ignores the spike. Traffic isn't flat. If RPS doubles for a minute, you need double the replicas for that minute, and by the time they boot, the spike is over.
- It ignores the cold start. A replica that takes 3 minutes to boot can't help with a spike that lasts 90 seconds. You needed it 3 minutes ago.
The real question isn't "how many", it's "how many, and how fast can I get them".
What the simulation showed
I simulated a service with 20 RPS average, 100 RPS spikes, 2 RPS per replica, and a 3-minute cold start. The results:
- Scaling on CPU utilization or GPU utilization is too slow. By the time the metric crosses the threshold, the spike is already here. You're always chasing.
- Scaling on queue depth works better. The queue is the leading indicator. When it grows, add replicas. When it shrinks, remove them.
- You need a floor. Always keep at least one warm replica, because the first request of a spike can't wait 3 minutes. This is the "buffer" that saves you.
- The per-replica throughput isn't constant. It rises with batch size, so a replica under load serves more per second than an idle one. Your sizing math has to account for that or you over-provision.
Autoscaling is like staffing a kitchen. You can't hire a chef in the middle of dinner rush. You keep one on standby, watch the line out the door, and hire before the rush, not during it.
The rules I extracted
- Scale on queue depth, not utilization. Utilization is a lagging indicator; the queue is leading.
- Keep a warm floor. One idle replica is cheap insurance against the first seconds of a spike.
- Model the cold start. If boot takes 3 minutes, your scale-up decision must be made 3 minutes early.
- Know your per-replica throughput curve. It's not a constant. Measure it across batch sizes.
In production, these rules become the autoscaler config: min_replicas, max_replicas, a queue-depth target, and a cooldown. But the config only makes sense if you've seen the simulation, because the simulation is where you learn what the config is fighting.
The takeaway
You can't autoscale against a spike you can't see coming. But you can keep a warm seat, watch the queue, and never let the first request of a rush wait for a cold boot.
Next: measuring cold start latency, the number that makes all of this concrete.