Every GPU cluster has a moment where you realize you don't actually know what's running on it. Not "what pods exist" - that's easy. But who owns the GPUs, which tenant's workload is eating the fleet, and whether the GitOps state matches reality. That's the gap a tenant inventory fills.
The numbers that matter
From a live inventory of the fleet:
- 129 B200 nodes, 1032 allocatable GPUs.
- 798 GPUs requested by pods, 798 used. 234 free.
- 10 namespaces with GPU pod requests.
- 53 public ALB ingress rules routing to all of it.
The headline: 798 of 1032 GPUs are in use, and the biggest tenants are the GLM deployments (256 GPUs for panja, 208 for code) and the Dynamo disaggregated graphs.
Per-tenant, not per-pod
The insight that makes the inventory useful: roll up by tenant, not by pod. A table of 100 deployments is noise. A table of 10 tenants, each with their GPU request, their product line, and their public routes, is a decision tool.
It answers questions like:
- Who do I call when the fleet is at 95%? (The tenant using 25% of it.)
- Which product line is growing? (Watch the per-tenant GPU request trend.)
- What's the blast radius of a routing change? (The tenant's ALB paths.)
Dynamo graphs as first-class citizens
Disaggregated serving breaks the naive "one deployment = one GPU count" model. A Dynamo graph spans multiple roles: prefill and decode, each with its own replica count and GPU-per-replica. The inventory has to roll those up too:
glm-52-fp8-prod-290726: decode 12×8 + prefill 14×8 = 208 GPUs.glm-52-fp8-prod2-300726: decode 16×8 + prefill 16×8 = 256 GPUs.glm-52-fp8-enterprise-110826: decode 3×8 + prefill 3×8 = 48 GPUs.
These are the real units of capacity, and they're invisible to a naive `kubectl get deploy`.
GitOps drift as a first-class signal
The inventory's most valuable output is the drift flag: live GPU workloads without a GitOps model match, and GitOps models with no live workload. Both directions matter.
- Live, not in GitOps → someone deployed outside the pipeline. Either intentional (a loadtest) or a process violation.
- In GitOps, not live → parked or not deployed. Either intentional (a staged rollout) or drift.
On this fleet, the drift list was a who's-who of the biggest deployments: all the GLM-52 Dynamo graphs, the DSpark DeepSeek V4 pair, the 64-GPU gemma4 decode worker. These were the workloads that mattered most, and they were all outside the GitOps model registry.
Regenerate, don't remember
The rule that keeps the inventory honest: no living doc states a counted fact. The inventory is regenerated from live queries, not hand-maintained. Hand-maintained counts drift by 20+ nodes in a month; live queries don't.
This is the same discipline as the render-integrity check: trust the invariant, not the memory.
The takeaway
You can't manage capacity you can't see. Roll up by tenant, treat Dynamo graphs as first-class units, and let drift be a signal instead of a surprise.
Sources
- Kubernetes docs: managing resources: how pod requests map to allocatable GPUs.
- NVIDIA Dynamo repo: the disaggregated serving framework that makes graphs first-class.
- Argo CD auto-sync docs: how GitOps drift is detected and reconciled.
- NVIDIA GPU Operator docs: the device plugin that makes GPUs schedulable.