Every GPU cluster has a failure mode that never pages anyone: a link that doesn't break, but bends. Throughput drops 20%, training silently stretches from 3 days to 4, and no alert fires because nothing is down. The fabric is the least-instrumented part of the stack, and the most expensive to get wrong.
What NVLink Sentinel is
NVLink Sentinel is NVIDIA's telemetry and health-monitoring layer for NVLink fabrics. It watches every link in the domain, reports utilization and errors, and exposes the health signals that let you treat the fabric as a first-class citizen instead of a black box.
It ships as a DaemonSet-style agent on every GPU node, plus a collector that aggregates the per-node view. The data lands in Prometheus via a metrics endpoint, so it plugs straight into the dashboards you already have.
One bad link slows the whole domain. NVLink domains are all-to-all: every GPU talks to every other GPU through the switch fabric. A single degraded link becomes the bottleneck for every collective that touches it. Sentinel is what tells you which link, where, and how bad.
What it exposes
- Link utilization per direction, per link. The fabric's equivalent of a network interface's bytes/sec.
- Error counters - retransmissions, unresponsive links, impaired paths. The early-warning signs of a link going bad.
- Health events - XID errors, SXID, GPU fell-off-bus, fabric manager status. The GPU-level signals that correlate with fabric health.
- Agent coverage - which nodes have an agent, which don't. The meta-metric that tells you whether your monitoring is actually monitoring.
The 88% problem
Here's the trap I hit: the agent DaemonSet tolerated only the `nvidia.com/gpu` taint. My fleet is tainted three different ways, so the agent could only schedule on 15 of 130 nodes. Roughly 920 B200s had no GPU health monitor, no syslog monitor, no metadata collector. The labels were correct, the pods simply couldn't be scheduled.
The fix was additive and low-risk: add tolerations for the other two taints, exactly like the EFA exporter already did. But the lesson is general: agent coverage is a metric, not an assumption. If you don't measure it, your "fleet monitoring" is monitoring a sample that may not represent the fleet.
Dry-run first, always
Sentinel's remediation pipeline (fault-remediation, fault-quarantine, node-drainer) can run in dry-run mode: it detects faults, analyzes them, decides what to do, and then does nothing. Cordon, drain and reboot are all simulated.
This is the right way to earn trust. The circuit breaker that caps how much of the fleet can be quarantined at once is the safety mechanism that makes arming the pipeline defensible. My staging ladder:
- Fix coverage to 130/130. Still dry-run.
- Build the dashboards, watch dry-run decisions for 1-2 weeks.
- Arm quarantine (cordon only, no drain). Circuit breaker already on.
- Arm node-drainer for empty/low-priority nodes first.
- Arm remediation (reboot / GPU reset / RMA flow). Only with confidence from 1-4.
Stage 0 is the prerequisite for everything: arming remediation while blind to 88% of the fleet would act on a badly unrepresentative sample.
The remediation funnel
Once the pipeline is live, the dashboard that matters is the funnel: detected → published → received → processed → cordoned → drained → remediated → returned to service. Drop-off at any stage is a pipeline failure, not a GPU failure.
- Detected: `dcgm_health_active_events`
- Published: `dcgm_health_events_publish_time_to_grpc_channel_*`
- Received: `fault_quarantine_events_received_total`
- Backlog: `fault_quarantine_event_backlog_count`
- Processed: `fault_quarantine_events_successfully_processed_total`
- Acted: `fault_quarantine_cordons_applied_total`
- State: `fault_quarantine_current_quarantined_nodes`
- Safety: `fault_quarantine_breaker_state` / `breaker_utilization`
MTTR per fault class is the number that tells you whether the system is working: XID vs ECC vs thermal vs fabric each have their own remediation path and their own time-to-recover.
The takeaway
The fabric is the least-instrumented part of the stack, and the most expensive to get wrong. Measure coverage before you measure links, and dry-run before you arm.
Sources
- NVIDIA NVLink Sentinel GitHub: the open-source repo, agent + collector + metrics.
- NVIDIA docs: NVLink Sentinel: deployment, tolerations, and the metrics endpoint.
- NVIDIA GPU Operator docs: how Sentinel integrates with the operator's DaemonSets.
- NVIDIA developer blog: NVLink Sentinel: the design story and the "one bad link" framing.