The trigger is not the cause: a GitOps outage autopsy

This is the story of a P0 that wasn't caused by the change that triggered it. It's the cleanest example I've seen of the difference between a trigger and a cause, and it produced a guard rail that should be in every GitOps pipeline.

The outage

At 19:08 UTC, a parallel WAF-authenticated public ALB was added to 45 Argo Applications. The change was additive by design: it rendered a second ALB Ingress alongside the existing public one, behind a WAF that requires a bearer token.

By 19:08, 39 public Ingresses had been deleted from the cluster. Argo's `prune: true` turned "desired state lost an object" into "delete it from production". The `inference-public` listener rules dropped from 63 to 23. ~39 endpoints returned 404 for 42 minutes.

The mechanism

Each archetype's `_alb-ingress.tpl` ended its final variable assignment with `-}}`. In Helm, `-}}` trims all following whitespace, including the newline. The `---` on the next line therefore landed at the end of the previous document's last rendered line:

                  number: 8000---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: qwen3-reranker-alb-auth

A YAML parser sees one document. Duplicate keys resolve last-wins, so `metadata.name` becomes `qwen3-reranker-alb-auth` and the public Ingress ceases to exist as an object.

Why it stayed hidden

The bug requires two or more ALB Ingresses rendered from one template file. The first document's `---` sits at the start of the output, where gluing is harmless. Before the auth feature, virtually every app rendered exactly one public ALB Ingress, so the bug had no trigger.

The auth feature added a second render to 45 apps. It did not touch the separator logic. Its only template changes were the `access_logs.s3.prefix` attribute and the `wafv2-acl-arn` annotation. Nothing in code review could have caught it, because the code was correct. The violation happened one layer down, in YAML serialisation.

The invariant

The fix was 8 files, one character each: drop the trailing `-` so the newline survives. But the real deliverable was the assertion that catches this class forever:

raw    = len(re.findall(r'(?m)^kind: ', out))
parsed = sum(1 for d in yaml.safe_load_all(out) if d)
assert raw == parsed, "glued separator?"

Render all 65 models with the exact value files the ApplicationSet injects, assert `count(^kind:)` equals the parsed document count, fail the build on mismatch. Before the fix: 45 models lost at least one Ingress. After: 0. Ingress objects in desired state went from 60 to 109.

What made it an outage

Two properties turned a template bug into a 42-minute P0:

• `prune: true` converts "desired state lost an object" into "delete it from production". Correct GitOps behaviour, but it raises the bar on render validation to match the blast radius it grants.
• No alerting on routing collapse. Rule count 63→23 and 39 Ingresses losing their load-balancer address were both trivially observable. Neither paged. Detection was a person running `curl`.

Reading the error's provenance

The detail that ruled out the backend in one step: `Server: awselb/2.0` with an empty body. That's the ALB's own default action, meaning no listener rule matched. The backend was `8/8` Running the whole time. The ALB answered; the routes were gone.

This is the discipline that makes RCAs fast: read where the error came from before you chase what it says.

The takeaway

Reverting the trigger restores service and fixes nothing. The cause was latent for weeks; the trigger was just the first change to render two documents from one template. Guard the invariant, not the incident.

Sources

• Helm template guide: whitespace control: the `-}}` trimming semantics at the heart of the bug.
• Argo CD auto-sync and prune docs: how `prune: true` + `selfHeal: true` turn render errors into deletions.
• YAML spec: document separators: why `---` glued to a line merges documents.
• AWS Load Balancer Controller docs: how IngressGroup members share group-scope config.

Back to the blog