On a 130-node B200 cluster with ~8200 EFA hardware counters live, the fabric is not a black box. It's a set of signals you can learn to read, and the difference between a healthy cluster and a slowly-dying one is usually a handful of counters.
The counters that matter
EFA exposes a rich set of hardware counters via the `efa_hw_counter` metric family. The ones that matter for fabric health, in order of severity:
- retrans - packets retransmitted. The earliest sign of a flaky path.
- unresponsive - a peer that isn't answering. Usually a node or NIC that's wedged.
- impaired - a path that's degraded. The NIC is up but the link is bad.
All three are cheap to scrape (low cardinality subsets) and correlate directly with NCCL collective stalls and training job failures.
topk(10, sum by (instance, counter) (rate(efa_hw_counter{counter=~".*retrans.*|.*unresponsive.*|.*impaired.*"}[5m])))
This is the "who's sick" panel. It ranks nodes by fabric distress over the last 5 minutes.
Flap delta: the job-scoped signal
A single retrans spike is noise. Sustained retransmissions during a training job are a signal. The `efa:flap_delta:with_job` recording rule captures exactly this: flap activity scoped to a job, so you can tell "the fabric is flapping" from "a job is hammering a bad path".
This is the difference between alerting on burn rate and alerting on blips. Page on sustained, job-scoped fabric distress. Don't page on a single counter increment.
The ops decision tree
- Transient spike, jobs healthy → observe. No cordon.
- Sustained + jobs failing → cordon the node so schedulers skip it, notify network/GPU infra.
- Exporter down only → restart the exporter pod. Do NOT cordon for scrape-only failures.
- Reboot / EC2 replace → only with written approval and the janitor graduated out of manual mode.
The discipline: cordon is reversible, drain is not. Cordon a node and schedulers skip it. Draining training pods without team OK is how you turn a fabric problem into a data-loss incident.
After node recycle
New GPU nodes may lack the `driver.installed` label, which silently drops them from the syslog health-monitor's desired state. The fix is a label job or a one-liner:
kubectl label nodes -l nvidia.com/gpu.present=true \
nvsentinel.dgxc.nvidia.com/driver.installed=true --overwriteThis is the same class of bug as the 88% blind spot: a label drift that makes your monitoring sample unrepresentative, invisible until you measure coverage.
The takeaway
Read the counters before the jobs fail. The fabric tells you it's sick before NCCL does, if you're listening.
Sources
- AWS EFA on EKS docs: the efa-exporter and how counters surface in Prometheus.
- AWS EC2 EFA docs: EFA hardware, SRD protocol, and the "no TCP/IP stack" design.
- AWS EFA driver repo: the kernel module and the counters it exposes.
- NVIDIA GPU Operator docs: how EFA and NVLink Sentinel coexist in the observability stack.