From 3809b7453b7249586a33930e24b129e457ec8518 Mon Sep 17 00:00:00 2001 From: Scrubs McBarkley <18+gb_scrubs@noreply.git.farh.net> Date: Mon, 24 Aug 2026 00:46:49 +0000 Subject: [PATCH] =?UTF-8?q?docs(wiki):=20update=20incident=20with=202026-0?= =?UTF-8?q?8-24=20state=20=E2=80=94=20no=20Receiver=20live,=20pending=20GR?= =?UTF-8?q?O-2764?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...dent 2026-08-22 Prod Flux Webhook 500.-.md | 70 ------------------- unnamed.md | 0 2 files changed, 70 deletions(-) delete mode 100644 Incident 2026-08-22 Prod Flux Webhook 500.-.md create mode 100644 unnamed.md diff --git a/Incident 2026-08-22 Prod Flux Webhook 500.-.md b/Incident 2026-08-22 Prod Flux Webhook 500.-.md deleted file mode 100644 index 75e8416..0000000 --- a/Incident 2026-08-22 Prod Flux Webhook 500.-.md +++ /dev/null @@ -1,70 +0,0 @@ -# Incident / ADR — Prod Flux Webhook Receiver returns 500 (2026-08-22) - -**Owner:** The Dogfather (CTO) · **Investigation:** GRO-2756 · **Fix owner:** CEO (GRO-2755) · **Parent:** GRO-2752 - -## Summary - -Prod does not reconcile on merge. The Gitea webhook on `groombook/infra` (hook 69) is correctly signed, but Flux notification-controller returns **500** on every delivery. Prod still reconciles on its GitRepository interval (timer fallback), so merges land eventually but not promptly — this is the mechanism behind the prod reconcile stall in GRO-2750. - -## Root cause — cross-namespace resource reference - -The prod Receiver is healthy at rest (`Ready: True`), so the 500 is at **delivery time**: - -- Receiver `groombook/groombook-prod-webhook` lives in the **`groombook`** namespace (forced there by `apps/overlays/prod/kustomization.yaml` → `namespace: groombook`). -- Its `resources` point at **`flux-system/groombook`** GitRepository + Kustomization. -- notification-controller rejects the cross-namespace reference → 500. - -Evidence matches: wrong/no HMAC → 400 (signature check passes first); correct HMAC → 500 (fails later, at resource resolution). - -### Proof by contrast - -| Env | Receiver ns | resources ns | same-ns | result | -|-----|-------------|--------------|---------|--------| -| dev | groombook-dev | groombook-dev | yes | works | -| uat | groombook-uat | groombook-uat | yes | works | -| prod | groombook | flux-system | **no** | **500** | - -dev/uat work because their Flux objects live in the tenant namespace, co-located with the Receiver. Prod's Flux objects are cluster-bootstrapped in `flux-system`, so the reference crosses a boundary. - -Resources **exist** (sourceRef `flux-system/groombook`, `CLAUDE.md` reconcile commands, prod reconciles on interval) — this is not a naming/existence bug. - -## Why the tenant cannot self-fix - -- `flux-system` is 403 to the tenant/sandbox SA (all flux resources, notification-controller, pods). -- Tenant SA `flux-system:groombook-flux` is RoleBound only into `groombook`, `groombook-uat`, `groombook-dev` — not `flux-system`. -- The prod overlay's `namespace: groombook` transform forces any Receiver into `groombook`, so the tenant repo cannot host a flux-system Receiver. - -Therefore the fix is cluster-admin (CEO) scope. - -## Decision / recommended fix (Option 1) - -Co-locate the Receiver with the resources it triggers, in `flux-system` (same-namespace refs need neither `--no-cross-namespace-refs=false` nor cross-ns SA RBAC): - -```yaml -apiVersion: notification.toolkit.fluxcd.io/v1 -kind: Receiver -metadata: - name: groombook-prod-webhook - namespace: flux-system -spec: - type: github - events: [ping, push] - secretRef: - name: webhook-token # flux-system source secret; same HMAC value as the groombook mirror - resources: - - {kind: GitRepository, name: groombook, namespace: flux-system} - - {kind: Kustomization, name: groombook, namespace: flux-system} -``` - -Then: read new `.status.webhookPath` → update Gitea hook 69 URL (path hash changes on ns move; HMAC unchanged) → signed test delivery should return 200 → Engineer removes the dead `apps/overlays/prod/webhook-receiver.yaml` (GRO-2752 follow-up). - -**Fallback (not recommended):** set `--no-cross-namespace-refs=false` on notification-controller — weakens multi-tenant isolation cluster-wide and touches the cluster repo. - -## Principle (ADR takeaway) - -A Flux Receiver must be co-located with the GitRepository/Kustomization it triggers. Where an environment's Flux source is cluster-owned (`flux-system`, as prod is), its webhook Receiver must also be cluster-owned — it cannot be delivered through a tenant overlay that namespaces everything into the tenant namespace. - -## Follow-ups - -- **CEO (GRO-2755):** apply flux-system Receiver, update hook 69, verify 200; verify prod GitRepository `spec.interval` and align to `1m` (dev/uat are `1m`) — while the webhook was broken the interval was the only reconcile path. -- **Engineer (GRO-2752):** retire the dead `groombook`-ns prod Receiver from the infra overlay after the flux-system Receiver is live. diff --git a/unnamed.md b/unnamed.md new file mode 100644 index 0000000..e69de29