Add incident/ADR: prod Flux webhook Receiver 500 root cause (GRO-2756)

2026-08-22 11:41:27 +00:00
parent 1f4012caeb
commit 4cc678051c
1 changed files with 70 additions and 0 deletions
@@ -0,0 +1,70 @@
+# Incident / ADR — Prod Flux Webhook Receiver returns 500 (2026-08-22)
+
+**Owner:** The Dogfather (CTO) · **Investigation:** GRO-2756 · **Fix owner:** CEO (GRO-2755) · **Parent:** GRO-2752
+
+## Summary
+
+Prod does not reconcile on merge. The Gitea webhook on `groombook/infra` (hook 69) is correctly signed, but Flux notification-controller returns **500** on every delivery. Prod still reconciles on its GitRepository interval (timer fallback), so merges land eventually but not promptly — this is the mechanism behind the prod reconcile stall in GRO-2750.
+
+## Root cause — cross-namespace resource reference
+
+The prod Receiver is healthy at rest (`Ready: True`), so the 500 is at **delivery time**:
+
+- Receiver `groombook/groombook-prod-webhook` lives in the **`groombook`** namespace (forced there by `apps/overlays/prod/kustomization.yaml` → `namespace: groombook`).
+- Its `resources` point at **`flux-system/groombook`** GitRepository + Kustomization.
+- notification-controller rejects the cross-namespace reference → 500.
+
+Evidence matches: wrong/no HMAC → 400 (signature check passes first); correct HMAC → 500 (fails later, at resource resolution).
+
+### Proof by contrast
+
+| Env | Receiver ns | resources ns | same-ns | result |
+|-----|-------------|--------------|---------|--------|
+| dev | groombook-dev | groombook-dev | yes | works |
+| uat | groombook-uat | groombook-uat | yes | works |
+| prod | groombook | flux-system | **no** | **500** |
+
+dev/uat work because their Flux objects live in the tenant namespace, co-located with the Receiver. Prod's Flux objects are cluster-bootstrapped in `flux-system`, so the reference crosses a boundary.
+
+Resources **exist** (sourceRef `flux-system/groombook`, `CLAUDE.md` reconcile commands, prod reconciles on interval) — this is not a naming/existence bug.
+
+## Why the tenant cannot self-fix
+
+- `flux-system` is 403 to the tenant/sandbox SA (all flux resources, notification-controller, pods).
+- Tenant SA `flux-system:groombook-flux` is RoleBound only into `groombook`, `groombook-uat`, `groombook-dev` — not `flux-system`.
+- The prod overlay's `namespace: groombook` transform forces any Receiver into `groombook`, so the tenant repo cannot host a flux-system Receiver.
+
+Therefore the fix is cluster-admin (CEO) scope.
+
+## Decision / recommended fix (Option 1)
+
+Co-locate the Receiver with the resources it triggers, in `flux-system` (same-namespace refs need neither `--no-cross-namespace-refs=false` nor cross-ns SA RBAC):
+
+```yaml
+apiVersion: notification.toolkit.fluxcd.io/v1
+kind: Receiver
+metadata:
+  name: groombook-prod-webhook
+  namespace: flux-system
+spec:
+  type: github
+  events: [ping, push]
+  secretRef:
+    name: webhook-token        # flux-system source secret; same HMAC value as the groombook mirror
+  resources:
+    - {kind: GitRepository, name: groombook, namespace: flux-system}
+    - {kind: Kustomization, name: groombook, namespace: flux-system}
+```
+
+Then: read new `.status.webhookPath` → update Gitea hook 69 URL (path hash changes on ns move; HMAC unchanged) → signed test delivery should return 200 → Engineer removes the dead `apps/overlays/prod/webhook-receiver.yaml` (GRO-2752 follow-up).
+
+**Fallback (not recommended):** set `--no-cross-namespace-refs=false` on notification-controller — weakens multi-tenant isolation cluster-wide and touches the cluster repo.
+
+## Principle (ADR takeaway)
+
+A Flux Receiver must be co-located with the GitRepository/Kustomization it triggers. Where an environment's Flux source is cluster-owned (`flux-system`, as prod is), its webhook Receiver must also be cluster-owned — it cannot be delivered through a tenant overlay that namespaces everything into the tenant namespace.
+
+## Follow-ups
+
+- **CEO (GRO-2755):** apply flux-system Receiver, update hook 69, verify 200; verify prod GitRepository `spec.interval` and align to `1m` (dev/uat are `1m`) — while the webhook was broken the interval was the only reconcile path.
+- **Engineer (GRO-2752):** retire the dead `groombook`-ns prod Receiver from the infra overlay after the flux-system Receiver is live.