This started as seven separate notes, one per tool, and every one of them was a README summary with an opinion bolted on the end. That is not writing, it is a bookmark folder. So here they are together, judged on the only axis I care about: what does adopting this actually cost me.
aicr, the AI Cluster Runtime
Operating a GPU cluster is a different job from operating a CPU cluster. The failure modes are stranger, the hardware is more expensive, and a bad node has a wider blast radius. aicr is the attempt to make that first-class: GPU health rather than node health, fabric state as a real signal, and the safety layer that stops one sick machine ruining a job.
The tax. It is another control plane below your scheduler, and it wants to be authoritative about node state. If it disagrees with Kubernetes about whether a node is healthy, you now have two sources of truth and an argument to arbitrate at three in the morning.
nvsentinel, for the fabric
The failure I find most unsettling is a link that degrades rather than dies: throughput drops, a job silently stretches, and nothing pages anyone. Sentinel makes that visible, and its most useful metric is not link health but agent coverage, which is the meta-question of whether your monitoring is actually monitoring.
The tax. Coverage is a number you have to chase, and a fabric monitor that covers eighty percent of nodes gives false confidence over the other twenty. Also worth knowing that the first-party material here is thin: the repo is real, the vendor docs I went looking for did not exist.
DALI, for the data path
Every cluster carries a hidden tax where the CPU spends longer decoding and reshaping images than the GPU spends computing on them. DALI moves decode and augmentation onto the GPU and pipelines the prefetch.
The tax. It only matters if you serve vision or multimodal. For text-only LLM inference the data path is not your bottleneck and this is a dependency you are carrying for nothing. It also takes GPU cycles from the model to feed the model, which is the right trade only when the CPU really was the constraint. Measure before you assume it was.
cudf-spark, on the data side
Swap the Spark executor for a GPU one and the data-heavy stages get an order of magnitude. The API does not change, which is the entire appeal.
The tax. This sits on the data side rather than the serving side and I would normally skip it. I stopped skipping it when I noticed how much of a GPU budget goes to jobs that are not inference at all. The honest catch is that not every operator has a GPU implementation, so a job can silently fall back to CPU and you get none of the speedup while paying all of the GPU rental.
NVCF, serverless GPU functions
Write a function, deploy it, scale to zero, pay per invocation. For genuinely spiky work, a nightly batch that wants a hundred GPUs for an hour and none for the other twenty-three, this is the right shape.
The tax. Cold starts. Loading a large model into GPU memory is measured in minutes, not milliseconds, so scale-to-zero means scale-to-a-very-slow-first-request. For steady-state serving this is the wrong tool and the idle GPU you were trying to avoid was cheaper than the latency you just bought.
srt-slurm, for benchmarking
Distributed inference benchmarking on SLURM is normally a pile of shell scripts and fifty command-line flags. srtctl replaces that with declarative YAML, so the config is the record and anyone can re-run it.
The tax. Very little, and this is the one I would adopt fastest. Benchmarking is how I decide what to deploy, so a fragile harness does not produce a bad number, it produces a bad decision. The caveat is that reproducible and correct are different properties, and a declarative config makes it easy to reproduce a measurement that was wrong to begin with. See the harness page for what I would pin.
security-workflows and SkillEvaluator
Two variations on the same idea: put the check in CI or it does not happen. security-workflows packages compliance scanning as reusable GitHub Actions so every repo runs the same thing with the same thresholds. SkillEvaluator does the equivalent for agent skills, where you cannot unit-test a return value and have to measure whether behaviour improved.
The tax. Both are only as good as their thresholds, and a gate everyone has learned to bypass is worse than no gate, because it launders the decision. I have a low tolerance for tooling that cannot show it helps, and to their credit both are at least trying to be measurable about it.
The pattern across all seven: each solves a real problem, each adds a component that can fail in a way the thing it replaced could not, and none of the documentation leads with that. Adoption cost is the number I want and never get.
A tool that makes the good case faster and the bad case invisible has not helped you. It has moved the failure somewhere you are not looking.
Sources
- NVIDIA/aicr: the cluster runtime.
- NVIDIA/nvsentinel: fabric health monitoring.
- NVIDIA/DALI: the GPU data loading library.
- rapidsai/cudf: the dataframe library the Spark plugin sits on.
- NVIDIA/srt-slurm: declarative benchmarks on SLURM.
- NVIDIA/security-workflows: compliance scanning as reusable Actions.
- SLURM documentation: the scheduler underneath the benchmark harness.