Benchmarking LLM inference on a SLURM cluster is a rite of passage that involves shell scripts, 50+ CLI flags, and a prayer. NVIDIA's srtctl replaces the whole mess with declarative YAML configuration, and it's the kind of tool you don't know you need until you've done it the hard way.
The problem
Distributed LLM inference benchmarks on SLURM clusters are painful. You need to launch TensorRT-LLM, SGLang, or vLLM across multiple nodes, coordinate the load, and collect the numbers. The traditional approach: complex shell scripts with dozens of flags, fragile across nodes, impossible to reproduce.
What srtctl does
srtctl is a command-line tool for distributed LLM inference benchmarks on SLURM clusters using TensorRT-LLM, SGLang, and vLLM. It replaces complex shell scripts and 50+ CLI flags with declarative YAML configuration.
- Declarative YAML - describe the benchmark, not the shell commands.
- Multiple runtimes - TensorRT-LLM, SGLang, and vLLM from one tool.
- Reproducible - the YAML is the record; anyone can re-run it.
srtctl is to LLM benchmarks what Kubernetes is to deployments: you declare the desired state, and the tool makes it happen.
Why it matters
Benchmarking is how you decide what to deploy. If the benchmark harness is fragile, the decisions are suspect. Declarative config makes benchmarks reproducible, which makes the numbers trustworthy, which makes the deployment decision sound.
- Reproducibility - the YAML is the record; anyone can re-run it.
- Multi-runtime - compare TensorRT-LLM against SGLang against vLLM fairly.
- Cluster-native - built for SLURM, the scheduler most HPC clusters run.
The takeaway
A benchmark you can't reproduce is a guess. Declarative config turns benchmarking into a repeatable experiment.
Sources
- srt-slurm repo: the srtctl tool for distributed LLM benchmarks.
- SLURM docs: the scheduler srtctl targets.
- TensorRT-LLM repo: one of the runtimes srtctl drives.
- vLLM repo: another runtime srtctl drives.