Every GPU cluster has a moment where you realize you don't actually know what's running on it. Not "what pods exist" - that's easy. But who owns the GPUs, which tenant's workload is eating the fleet, and whether the GitOps state matches reality. That's the gap a tenant inventory fills.
The numbers that matter
The inventory is four counts, and I would want all four before I made any capacity decision:
- Nodes and allocatable GPUs. Not the number you bought. The number the scheduler can actually place work on, which is smaller once taints and broken device plugins are accounted for.
- Requested against used. These diverge, and the gap is the most interesting line in the report. Requested but not used is a tenant holding capacity it is not burning.
- Namespaces with GPU requests. Usually fewer than people expect, and the distribution is heavily skewed toward two or three of them.
- Public ingress rules routing to it. The one people forget. It is how you work out the blast radius of a routing change before you make it.
The shape I keep finding is the same one: utilisation sits somewhere in the seventies to eighties, and a small number of large language-model deployments account for most of the fleet, with the disaggregated graphs the biggest single line items. The absolute numbers matter less than the ratio, because the ratio is what tells you whether you have a capacity problem or a placement problem.
Per-tenant, not per-pod
The insight that makes the inventory useful: roll up by tenant, not by pod. A table of 100 deployments is noise. A table of 10 tenants, each with their GPU request, their product line, and their public routes, is a decision tool.
It answers questions like:
- Who do I call when the fleet is at 95%? (The tenant using 25% of it.)
- Which product line is growing? (Watch the per-tenant GPU request trend.)
- What's the blast radius of a routing change? (The tenant's ALB paths.)
Dynamo graphs as first-class citizens
Disaggregated serving breaks the naive "one deployment = one GPU count" model. A Dynamo graph spans multiple roles: prefill and decode, each with its own replica count and GPU-per-replica. The inventory has to roll those up too:
<model>-prod: decode 10x8 + prefill 10x8 = 160 GPUs.<model>-canary: decode 2x8 + prefill 2x8 = 32 GPUs.<model>-enterprise: decode 1x8 + prefill 1x8 = 16 GPUs.
These are the real units of capacity, and they're invisible to a naive `kubectl get deploy`.
GitOps drift as a first-class signal
The inventory's most valuable output is the drift flag: live GPU workloads without a GitOps model match, and GitOps models with no live workload. Both directions matter.
- Live, not in GitOps → someone deployed outside the pipeline. Either intentional (a loadtest) or a process violation.
- In GitOps, not live → parked or not deployed. Either intentional (a staged rollout) or drift.
The drift list turned out to be a who's-who of the biggest deployments: the largest Dynamo graphs, a speculative-decoding pair, a 64-GPU decode worker. The workloads that mattered most were the ones outside the GitOps model registry, which is the opposite of what you would hope and, on reflection, exactly what you should expect. The things under the most pressure get changed by hand.
Regenerate, don't remember
The rule that keeps the inventory honest: no living doc states a counted fact. The inventory is regenerated from live queries, not hand-maintained. Hand-maintained counts drift by 20+ nodes in a month; live queries don't.
This is the same discipline as the render-integrity check: trust the invariant, not the memory.
You can't manage capacity you can't see. Roll up by tenant, treat Dynamo graphs as first-class units, and let drift be a signal instead of a surprise.
Sources
- Kubernetes docs: managing resources: how pod requests map to allocatable GPUs.
- NVIDIA Dynamo repo: the disaggregated serving framework that makes graphs first-class.
- Argo CD auto-sync docs: how GitOps drift is detected and reconciled.
- NVIDIA GPU Operator docs: the device plugin that makes GPUs schedulable.