The most expensive failures I have chased are not the ones that crash. They are the ones where a process is dead in every way that matters to a user and alive in every way that matters to a scheduler. Multi-rank inference serving produces this shape reliably, and I want to write down why, because the first two times I saw it I misread the evidence completely.
Here is the sequence, stripped of anything identifying. A decode replica starts.
Five and a half minutes in, one rank hits a fabric error on the RDMA path and calls
abort. The Python process for that rank is gone. The container is not. Nine minutes
later the pod still reports 1/1 Running, is still emitting log lines,
and is still receiving requests.
What the fabric error looks like
On an EFA fabric the abort is preceded by a specific pair of lines: a receive
error on a packet that belongs to neither the send nor the receive queue, followed
by a queue-pair error written to the event queue. The provider surfaces it as
Invalid argument with an internal queue-pair error underneath, and the
runtime turns that into a fatal abort.
The useful property of this failure is that it prints. An EFA abort leaves a line in the log at the moment of death. That means its absence is evidence too, and I have used that more than once to rule it out as a cause.
Why nothing catches it
Three mechanisms that should have caught this each miss for a different reason, and it is worth separating them because the fixes are different.
- The exit code is zero. When the container eventually cycles
it reports
Completed, not a crash. Every alert keyed on non-zero exit status stays silent. This is the one that surprised me most: the pod had restarted several times and nothing exit-code-based had ever fired. - The liveness probe passes for a while. The HTTP server lives
in a surviving rank. It answers
/liveperfectly well while the tensor parallel group behind it is missing a member. The probe is testing the wrong layer: it asks whether the server is up, not whether the model is intact. - Readiness never flips. Because liveness is what eventually
times out, the pod stays
Readythroughout the window between the abort and the kill. Ready means the load balancer keeps sending it work.
I have started calling the gap between those two moments the zombie window. In the case I am describing it ran about ten minutes, and the whole restart cycle end to end was closer to sixteen. Every request routed into that window is either hanging or returning something wrong.
The part I got wrong
My first instinct on a second failing replica was that I had found the same bug twice. I had not. A prefill replica in the same deployment had also cycled, and a loose grep for the string "efa" returned hundreds of hits, which looked like strong confirmation. They were all substring matches inside an unrelated identifier that happened to contain those three letters.
The actual prefill failure had a different signature entirely: rank zero disappeared without printing anything, and the surviving ranks unwound from a connection reset on the coordination store, then took a quit signal. No fabric error, no CUDA error, no out-of-memory, no traceback. That container had been up long enough that its logs had rotated, so the window before the death was simply gone.
I could not determine why rank zero died, and I think saying so is more useful than picking the most plausible cause and presenting it as the finding. What I could establish was a negative: an EFA abort prints at the moment of death, the death was inside the retained window, and there was no such line. So it was not this bug. That is a smaller claim and it is actually supported.
What I would change
- Probe the model, not the server. A liveness endpoint that runs a one-token forward pass across the full tensor parallel group fails immediately when a rank is missing. An endpoint that returns a static two hundred tells you the web framework is fine.
- Make a dead rank kill the container. If the supervisor treats a rank abort as fatal to the whole process group, the zombie window collapses to nothing and the exit code stops lying.
- Alert on restart count, not exit code. A replica that has restarted several times with clean exits is a stronger signal than any single crash, and it is the signal that was sitting in plain view the whole time.
- Do not let logs rotate out of the diagnostic window. The prefill failure is unsolved specifically because its evidence expired. Ship logs off the node or size the buffer for the longest expected uptime, not the average.
Health checks answer the question they were written to ask. Almost nobody writes one that asks whether the model still has all its ranks.
Sources
- fi_efa(7), libfabric: the provider's own description of when it aborts communication and hands the error back to the application.
- EFA RDM protocol v4: the packet and queue-pair model the error messages refer to.