You can benchmark an inference endpoint at a fixed concurrency and get nice stable numbers. But those numbers mean nothing unless you know where the system breaks. The saturation point is the load level at which latency starts growing faster than throughput, meaning the server is queueing requests internally. Finding this point is the single most useful thing you can do before going to production.
I used Locust for this because it is Python-native, easy to customize for LLM-style requests, and has a built-in web UI for watching latency curves in real time.
Why Locust for LLM endpoints
Locust is a load testing framework where you define user behavior in Python. Unlike tools designed for simple HTTP benchmarks (like wrk or ab), Locust lets you write complex request patterns: variable prompt lengths, streaming responses, different endpoints. For LLM inference, this matters because a single prompt length does not represent real traffic.
The key concepts:
- User: a simulated client that sends requests in a loop with configurable wait times between them.
- Task: a function decorated with
@taskthat defines what each user does. - Shape: a class that controls how many users are active over time (ramp up, hold, ramp down).
The Locust file
Here is the Locustfile I wrote for testing a vLLM OpenAI-compatible endpoint. It samples prompts from a list of varying lengths to simulate real traffic:
from locust import HttpUser, task, between
import json
import random
# Prompts of varying lengths to simulate real traffic
PROMPTS = [
"Explain what a transformer is in one paragraph.",
"Write a Python function that implements binary search. Include docstring and type hints.",
"Summarize the key differences between TCP and UDP. " * 5,
"What are the trade-offs between FP16 and INT8 quantization for LLM inference? " * 3,
"Tell me a joke.",
]
class InferenceUser(HttpUser):
wait_time = between(0.1, 0.5) # short wait to stress the server
@task
def generate(self):
prompt = random.choice(PROMPTS)
payload = {
"model": "meta-llama/Llama-2-13b-hf",
"prompt": prompt,
"max_tokens": 128,
"temperature": 0.7,
}
with self.client.post(
"/v1/completions",
json=payload,
catch_response=True,
timeout=30,
) as response:
if response.status_code == 200:
response.success()
elif response.status_code == 429:
response.failure("Rate limited")
else:
response.failure(f"Status {response.status_code}")Launch it with:
# Start Locust pointing at your inference server
locust -f locustfile.py --host http://gpu-server:8000 \
--headless -u 100 -r 10 --run-time 5m \
--csv results/load_testThe flags: -u 100 sets the max users, -r 10 ramps up 10 users per second, and --csv saves results for analysis.
Finding the saturation point
The saturation point is where throughput stops growing linearly with load. Below saturation, adding more concurrent users increases throughput proportionally. At saturation, the GPU's compute or memory bandwidth is fully utilized, and additional requests queue up. Above saturation, throughput plateaus while latency climbs steeply.
I ran the test with a step-load pattern, holding at each concurrency level for 2 minutes before stepping up:
from locust import LoadTestShape
class StepLoadShape(LoadTestShape):
"""Step load: 10 users, then 20, 30, ... up to 100."""
step_time = 120 # seconds per step
step_load = 10 # users added per step
max_users = 100
spawn_rate = 10
def tick(self):
run_time = self.get_run_time()
current_step = int(run_time // self.step_time) + 1
target_users = min(current_step * self.step_load, self.max_users)
if run_time > self.step_time * (self.max_users // self.step_load):
return None # stop
return (target_users, self.spawn_rate)Looking at the results for a Llama 2 13B on a single H100:
- 10 users: median latency 180ms, P99 250ms, throughput 55 req/s.
- 20 users: median 200ms, P99 310ms, throughput 98 req/s. Nearly linear scaling.
- 30 users: median 230ms, P99 420ms, throughput 128 req/s. Still roughly linear.
- 40 users: median 310ms, P99 780ms, throughput 140 req/s. Throughput gain is flattening.
- 50 users: median 520ms, P99 1,800ms, throughput 142 req/s. Saturation.
- 60+ users: median 900ms+, P99 3,000ms+, throughput stuck at 143 req/s.
Latency always follows a hockey stick curve under load. It is flat, flat, flat, then suddenly vertical. The bend in the stick is your saturation point. For this setup, it was around 35 to 40 concurrent users. Set your autoscaler to trigger well below that.
What happens at saturation
When the GPU is fully utilized, vLLM's continuous batching scheduler queues incoming requests. The queue itself adds latency. But there is a subtler effect: as the running batch grows, the KV cache fills up, and vLLM may need to preempt (swap out) some requests to make room for new ones. Preemption causes re-computation and makes P99 latency much worse than P50.
This is why the P99 blows up before the median does. The median request gets processed in a normal batch. The P99 request arrives when the batch is full, waits in the queue, and then may get preempted partway through generation.
Using the results
The load test gives you three actionable numbers:
- Max throughput: about 143 req/s for this setup. This is the ceiling; no amount of traffic shaping will push past it on one GPU.
- Saturation concurrency: about 35 to 40 users. Set your autoscaler's target concurrency to 30 (with headroom) and scale out before you hit 40.
- SLO-safe concurrency: if your SLO is P99 < 500ms, you cannot run above 30 concurrent users on this replica.
These numbers feed directly into capacity planning. If you expect peak traffic of 400 req/s, you need at least 3 replicas (400/143 = 2.8, round up), but for SLO compliance at P99 < 500ms, you probably need 5 replicas (to keep each replica at 25 concurrent users).
Capacity planning is not "peak traffic / max throughput." It is "peak traffic / SLO-safe throughput." The gap between those two numbers is the cost of reliability.
Tomorrow I will look at profiling with Nsight Systems to understand exactly where the time goes when we hit saturation.