I went looking through a benchmarking harness for how it placed processes on a second machine, because I wanted to run a two-node configuration. There was a field for it in the config. The arithmetic downstream of that field was careful and correct. There was no placement anywhere in the system.
The field was read in four places and obeyed in none of them:
- Feasibility arithmetic, deciding whether a model fits in the aggregate memory of n nodes.
- An interconnect penalty applied to predicted throughput when a layout crosses machines.
- Scheduler concurrency, sizing how many trials could run at once.
- Track selection, deciding whether a multi-node branch was offered at all.
All four are legitimate uses. Together they produce a system that reasons fluently about two nodes, plans for two nodes, reports on two nodes, and runs everything on one.
The two mechanisms that were missing
It came down to two small things, both of which look like they work if you read the code quickly.
The pool handed out ids nobody used. A node pool tracked free node indices and its acquire call returned one. Every call site discarded the return value and used the pool purely as a counting semaphore for concurrency. The information needed to place a process was being computed and thrown away.
The placeholders were never substituted. Launch
templates contained $HEAD_ADDR and $NODE_RANK. The
launcher used a shell-less process spawn, so no shell ever expanded them and
no code did it by hand. They reached the engine as literal dollar-sign
strings.
Meanwhile every URL in the configuration was 127.0.0.1, which
is the part that should have been obvious and was not, because a loopback
address in a config that is mostly used single-node does not look wrong.
Kubernetes would not have fixed it
My first instinct was that this is what an orchestrator is for, and that running the thing as a two-pod job would make the problem go away. It would not. In a two-pod job the code still addresses loopback and still emits the placeholder as text. The orchestrator would place two pods perfectly and the second one would sit there while the first talked to itself.
That is the general shape, and it is why I am writing it down. Placement has two halves: getting a process onto a machine, and telling every process where the others are. Orchestrators do the first. Nothing does the second for you, because the addresses are application state. A system can be perfectly scheduled and still entirely single-node.
Choosing the mechanism
For a harness that holds GPUs for many hours at a stretch, reaps engine process groups with a signal to the group, and refuses to start on devices that are not clean, the options are not equivalent:
- SSH fan-out is the smallest change when the pool already tracks free node indices. It needs a hostname map and the ids that are currently discarded. It also preserves the property that the whole run lives in a terminal session on a login node, which is worth more than it sounds during bring-up.
- Slurm is the natural answer if the cluster already has it, because rank and address propagation is the part it is genuinely good at.
- Kubernetes is the heaviest and works against the grain here. Long GPU holds, process-group signals and a refusal to start on dirty devices are all things you end up fighting the pod lifecycle over.
The test that would have caught it
Not "does the config parse", and not "does the two-node path produce a plan". Both of those passed. The test is whether a second machine ever sees a process, and the cheap version of it is two assertions: that a rendered launch command contains a resolved address rather than a placeholder, and that at least one command in a multi-node trial targets a host that is not the local one.
I have started treating unexpanded placeholders as a class. Anything of the
form $NAME surviving into a rendered command line is a bug that
will not announce itself, because the receiving process usually accepts the
string and fails somewhere else much later.
A configuration value that is read everywhere and obeyed nowhere is worse than a missing one, because everything downstream of it reports success.
Sources
- srun(1): what a scheduler propagates for you, which is the half orchestrators tend not to.
- torchrun and elastic rendezvous: the standard answer to how ranks find each other, and the shape any hand-rolled version ends up reinventing.
- Indexed jobs: the closest Kubernetes gets, and still only the placement half.
- The pod that was already dead: the other failure where every layer reported healthy.