groombook/org
13
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(adr): ADR-0002 API DB-health endpoint strategy — DB-less probes + single /api/readyz (GRO-2687/2678/2689)

The Dogfather
2026-08-09 09:44:13 +00:00
parent 17d0f27b73
commit 9e3d300695
1 changed files with 31 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ADR-0002 API DB-health endpoint strategy.-.md
+31
@@ -0,0 +1,31 @@
# ADR-0002 — API DB-health endpoint strategy
- **Status:** Accepted
- **Date:** 2026-08-09
- **Decider:** The Dogfather (CTO)
- **Context issues:** GRO-2678 (PROD schema-drop incident), GRO-2687 (`/api/readyz`), GRO-2689 (`/health/ready`), GRO-2652 (boot ECONNRESET), GRO-1544 (HTTPRoute `/api/*` routing)
## Context
In GRO-2678, the PROD `groombook` DB schema was dropped and the outage stayed hidden for ~43h because `/health` and `/api/health` return `{status:"ok"}` **without touching the database**. The `smoke-test` CronJob hit `/health`, stayed green, and no alert fired while every authenticated endpoint threw `42P01 relation ... does not exist`.
Two competing remediations then appeared in the codebase:
- **GRO-2687** added `/api/readyz` — a **DB-touching, monitoring-only** endpoint, keeping the K8s probes DB-less.
- **GRO-2689** added `/health/ready` — a **DB-touching endpoint intended to gate K8s pod readiness** (comment: "K8s removes pod from endpoints when schema is dropped").
These pull in opposite directions and are redundant.
## Decision
1. **K8s liveness and readiness probes remain DB-less** (`/api/health`). A transient DB blip (e.g. GRO-2652 boot ECONNRESET, single-replica CNPG primary reschedule) must **not** evict or cycle pods. Gating readiness on the DB would take the whole service down on a transient blip.
2. **DB/schema health is exposed on exactly one endpoint: `GET /api/readyz`** (GRO-2687), consumed by the **monitoring/alerting layer** (infra follow-up), never by K8s probes.
- `200 {"status":"ready"}` on a successful lightweight query.
- `503 {"status":"degraded","check":"db"}` on failure; raw SQL/driver text is logged, **never** returned in the body.
3. **`/health/ready` (GRO-2689) is superseded and must not ship to production.** It is redundant with `/api/readyz` and, being registered **outside** the `/api/*` prefix, is routed by the Gateway HTTPRoute `/*` rule to the **web** pod, not the API pod (see GRO-1544) — so it is not even reachable as designed on `api.groombook.dev`.
4. Any future proposal to make readiness DB-gating (fail-fast readiness) is a **separate ADR** requiring explicit CTO sign-off — it is not to be introduced by bundling it into an unrelated promotion.
## Consequences
- Monitoring/alerting scrapes `/api/readyz`; a dropped schema now surfaces as an alert **without** cycling pods. (Infra wiring is the sibling follow-up to GRO-2687.)
- The `smoke-test` CronJob's health target should be pointed at `/api/readyz` (not `/health`) so schema loss can no longer hide behind a DB-less 200.
- Promotion hygiene: features reach `main` only through their **own** completed SDLC (UAT + Security + CTO review). GRO-2689 riding into `uat` on the GRO-2687 dev→uat merge, then attempting to reach `main` inside GRO-2687's promotion PR (#235), is the anti-pattern this ADR closes — see also *Defect Pattern: dev-uat Promotion Clobbers Env-Specific Overlay State*.