1

ADR-0002 API DB-health endpoint strategy

The Dogfather edited this page 2026-08-09 09:44:13 +00:00

Table of Contents

• ADR-0002 — API DB-health endpoint strategy
• Context
• Decision
• Consequences

ADR-0002 — API DB-health endpoint strategy

• Status: Accepted
• Date: 2026-08-09
• Decider: The Dogfather (CTO)
• Context issues: GRO-2678 (PROD schema-drop incident), GRO-2687 (/api/readyz), GRO-2689 (/health/ready), GRO-2652 (boot ECONNRESET), GRO-1544 (HTTPRoute /api/* routing)

Context

In GRO-2678, the PROD groombook DB schema was dropped and the outage stayed hidden for ~43h because /health and /api/health return {status:"ok"} without touching the database. The smoke-test CronJob hit /health, stayed green, and no alert fired while every authenticated endpoint threw 42P01 relation ... does not exist.

Two competing remediations then appeared in the codebase:

• GRO-2687 added /api/readyz — a DB-touching, monitoring-only endpoint, keeping the K8s probes DB-less.
• GRO-2689 added /health/ready — a DB-touching endpoint intended to gate K8s pod readiness (comment: "K8s removes pod from endpoints when schema is dropped").

These pull in opposite directions and are redundant.

Decision

1. K8s liveness and readiness probes remain DB-less (/api/health). A transient DB blip (e.g. GRO-2652 boot ECONNRESET, single-replica CNPG primary reschedule) must not evict or cycle pods. Gating readiness on the DB would take the whole service down on a transient blip.
2. DB/schema health is exposed on exactly one endpoint: GET /api/readyz (GRO-2687), consumed by the monitoring/alerting layer (infra follow-up), never by K8s probes.
   • 200 {"status":"ready"} on a successful lightweight query.
   • 503 {"status":"degraded","check":"db"} on failure; raw SQL/driver text is logged, never returned in the body.
3. /health/ready (GRO-2689) is superseded and must not ship to production. It is redundant with /api/readyz and, being registered outside the /api/* prefix, is routed by the Gateway HTTPRoute /* rule to the web pod, not the API pod (see GRO-1544) — so it is not even reachable as designed on api.groombook.dev.
4. Any future proposal to make readiness DB-gating (fail-fast readiness) is a separate ADR requiring explicit CTO sign-off — it is not to be introduced by bundling it into an unrelated promotion.

Consequences

• Monitoring/alerting scrapes /api/readyz; a dropped schema now surfaces as an alert without cycling pods. (Infra wiring is the sibling follow-up to GRO-2687.)
• The smoke-test CronJob's health target should be pointed at /api/readyz (not /health) so schema loss can no longer hide behind a DB-less 200.
• Promotion hygiene: features reach main only through their own completed SDLC (UAT + Security + CTO review). GRO-2689 riding into uat on the GRO-2687 dev→uat merge, then attempting to reach main inside GRO-2687's promotion PR (#235), is the anti-pattern this ADR closes — see also Defect Pattern: dev-uat Promotion Clobbers Env-Specific Overlay State.