diff --git a/ADR-0002 API DB-health endpoint strategy.-.md b/ADR-0002 API DB-health endpoint strategy.-.md new file mode 100644 index 0000000..8fb6294 --- /dev/null +++ b/ADR-0002 API DB-health endpoint strategy.-.md @@ -0,0 +1,31 @@ +# ADR-0002 — API DB-health endpoint strategy + +- **Status:** Accepted +- **Date:** 2026-08-09 +- **Decider:** The Dogfather (CTO) +- **Context issues:** GRO-2678 (PROD schema-drop incident), GRO-2687 (`/api/readyz`), GRO-2689 (`/health/ready`), GRO-2652 (boot ECONNRESET), GRO-1544 (HTTPRoute `/api/*` routing) + +## Context + +In GRO-2678, the PROD `groombook` DB schema was dropped and the outage stayed hidden for ~43h because `/health` and `/api/health` return `{status:"ok"}` **without touching the database**. The `smoke-test` CronJob hit `/health`, stayed green, and no alert fired while every authenticated endpoint threw `42P01 relation ... does not exist`. + +Two competing remediations then appeared in the codebase: +- **GRO-2687** added `/api/readyz` — a **DB-touching, monitoring-only** endpoint, keeping the K8s probes DB-less. +- **GRO-2689** added `/health/ready` — a **DB-touching endpoint intended to gate K8s pod readiness** (comment: "K8s removes pod from endpoints when schema is dropped"). + +These pull in opposite directions and are redundant. + +## Decision + +1. **K8s liveness and readiness probes remain DB-less** (`/api/health`). A transient DB blip (e.g. GRO-2652 boot ECONNRESET, single-replica CNPG primary reschedule) must **not** evict or cycle pods. Gating readiness on the DB would take the whole service down on a transient blip. +2. **DB/schema health is exposed on exactly one endpoint: `GET /api/readyz`** (GRO-2687), consumed by the **monitoring/alerting layer** (infra follow-up), never by K8s probes. + - `200 {"status":"ready"}` on a successful lightweight query. + - `503 {"status":"degraded","check":"db"}` on failure; raw SQL/driver text is logged, **never** returned in the body. +3. **`/health/ready` (GRO-2689) is superseded and must not ship to production.** It is redundant with `/api/readyz` and, being registered **outside** the `/api/*` prefix, is routed by the Gateway HTTPRoute `/*` rule to the **web** pod, not the API pod (see GRO-1544) — so it is not even reachable as designed on `api.groombook.dev`. +4. Any future proposal to make readiness DB-gating (fail-fast readiness) is a **separate ADR** requiring explicit CTO sign-off — it is not to be introduced by bundling it into an unrelated promotion. + +## Consequences + +- Monitoring/alerting scrapes `/api/readyz`; a dropped schema now surfaces as an alert **without** cycling pods. (Infra wiring is the sibling follow-up to GRO-2687.) +- The `smoke-test` CronJob's health target should be pointed at `/api/readyz` (not `/health`) so schema loss can no longer hide behind a DB-less 200. +- Promotion hygiene: features reach `main` only through their **own** completed SDLC (UAT + Security + CTO review). GRO-2689 riding into `uat` on the GRO-2687 dev→uat merge, then attempting to reach `main` inside GRO-2687's promotion PR (#235), is the anti-pattern this ADR closes — see also *Defect Pattern: dev-uat Promotion Clobbers Env-Specific Overlay State*.