diff --git a/Incident 2026-08-22 Prod Flux Webhook 500.-.md b/Incident 2026-08-22 Prod Flux Webhook 500.-.md new file mode 100644 index 0000000..75e8416 --- /dev/null +++ b/Incident 2026-08-22 Prod Flux Webhook 500.-.md @@ -0,0 +1,70 @@ +# Incident / ADR — Prod Flux Webhook Receiver returns 500 (2026-08-22) + +**Owner:** The Dogfather (CTO) · **Investigation:** GRO-2756 · **Fix owner:** CEO (GRO-2755) · **Parent:** GRO-2752 + +## Summary + +Prod does not reconcile on merge. The Gitea webhook on `groombook/infra` (hook 69) is correctly signed, but Flux notification-controller returns **500** on every delivery. Prod still reconciles on its GitRepository interval (timer fallback), so merges land eventually but not promptly — this is the mechanism behind the prod reconcile stall in GRO-2750. + +## Root cause — cross-namespace resource reference + +The prod Receiver is healthy at rest (`Ready: True`), so the 500 is at **delivery time**: + +- Receiver `groombook/groombook-prod-webhook` lives in the **`groombook`** namespace (forced there by `apps/overlays/prod/kustomization.yaml` → `namespace: groombook`). +- Its `resources` point at **`flux-system/groombook`** GitRepository + Kustomization. +- notification-controller rejects the cross-namespace reference → 500. + +Evidence matches: wrong/no HMAC → 400 (signature check passes first); correct HMAC → 500 (fails later, at resource resolution). + +### Proof by contrast + +| Env | Receiver ns | resources ns | same-ns | result | +|-----|-------------|--------------|---------|--------| +| dev | groombook-dev | groombook-dev | yes | works | +| uat | groombook-uat | groombook-uat | yes | works | +| prod | groombook | flux-system | **no** | **500** | + +dev/uat work because their Flux objects live in the tenant namespace, co-located with the Receiver. Prod's Flux objects are cluster-bootstrapped in `flux-system`, so the reference crosses a boundary. + +Resources **exist** (sourceRef `flux-system/groombook`, `CLAUDE.md` reconcile commands, prod reconciles on interval) — this is not a naming/existence bug. + +## Why the tenant cannot self-fix + +- `flux-system` is 403 to the tenant/sandbox SA (all flux resources, notification-controller, pods). +- Tenant SA `flux-system:groombook-flux` is RoleBound only into `groombook`, `groombook-uat`, `groombook-dev` — not `flux-system`. +- The prod overlay's `namespace: groombook` transform forces any Receiver into `groombook`, so the tenant repo cannot host a flux-system Receiver. + +Therefore the fix is cluster-admin (CEO) scope. + +## Decision / recommended fix (Option 1) + +Co-locate the Receiver with the resources it triggers, in `flux-system` (same-namespace refs need neither `--no-cross-namespace-refs=false` nor cross-ns SA RBAC): + +```yaml +apiVersion: notification.toolkit.fluxcd.io/v1 +kind: Receiver +metadata: + name: groombook-prod-webhook + namespace: flux-system +spec: + type: github + events: [ping, push] + secretRef: + name: webhook-token # flux-system source secret; same HMAC value as the groombook mirror + resources: + - {kind: GitRepository, name: groombook, namespace: flux-system} + - {kind: Kustomization, name: groombook, namespace: flux-system} +``` + +Then: read new `.status.webhookPath` → update Gitea hook 69 URL (path hash changes on ns move; HMAC unchanged) → signed test delivery should return 200 → Engineer removes the dead `apps/overlays/prod/webhook-receiver.yaml` (GRO-2752 follow-up). + +**Fallback (not recommended):** set `--no-cross-namespace-refs=false` on notification-controller — weakens multi-tenant isolation cluster-wide and touches the cluster repo. + +## Principle (ADR takeaway) + +A Flux Receiver must be co-located with the GitRepository/Kustomization it triggers. Where an environment's Flux source is cluster-owned (`flux-system`, as prod is), its webhook Receiver must also be cluster-owned — it cannot be delivered through a tenant overlay that namespaces everything into the tenant namespace. + +## Follow-ups + +- **CEO (GRO-2755):** apply flux-system Receiver, update hook 69, verify 200; verify prod GitRepository `spec.interval` and align to `1m` (dev/uat are `1m`) — while the webhook was broken the interval was the only reconcile path. +- **Engineer (GRO-2752):** retire the dead `groombook`-ns prod Receiver from the infra overlay after the flux-system Receiver is live.