feat(api): add DB-touching /health/ready readiness probe (GRO-2678)

Register GET /health/ready before the /api/* auth middleware so it is
public and reachable by K8s readinessProbe on port 3000 without auth.
On success → 200 {"status":"ready"}; on any DB/schema failure → 503
{"status":"degraded"}. Logs the pg error code; never leaks SQL in body.
A dropped schema (42P01) surfaces as non-200, closing the /health mask
that allowed the GRO-2678 incident to go undetected for ~43h.

Add health-ready.test.ts covering the 200 success path, 503 on schema
drop (42P01), and 503 on connection error; all assert no SQL leakage.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

.mcp.json

@@ -0,0 +1,11 @@
+{
+  "mcpServers": {
+    "gitea": {
+      "type": "http",
+      "url": "https://git-mcp.farh.net/mcp",
+      "headers": {
+        "Authorization": "Bearer ${GITEA_TOKEN}"
+      }
+    }
+  }
+}

AGENTS.md

@@ -1,54 +0,0 @@
-# AGENTS.md
-
-This repository (`groombook/api`) is part of the GroomBook application stack. The
-authoritative process, quality bar, and safety rules live in the shared
-[`groombook/org`](https://git.farh.net/groombook/org) skills repository. Read
-those first; this file is only a pointer.
-
-## Authoritative skills
-
-- **SDLC (branching, PRs, phases, handoffs):**
-  [`groombook/org/skills/sdlc/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/sdlc/SKILL.md)
-- **Coding standards (priority ordering, PR discipline, tests, no-hardcoded-values, CalVer):**
-  [`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md)
-- **Safety (no plaintext secrets, no direct `kubectl apply` to `groombook`, no self-merge, board approval for destructive actions):**
-  [`groombook/org/skills/safety/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/safety/SKILL.md)
-
-For human contributors and humans reviewing agent work, see
-[`CONTRIBUTING.md`](./CONTRIBUTING.md) in this repo for the phase-by-phase PR
-flow and the `uat→main` merge-gate policy summary.
-
-## Non-negotiable operational rules
-
-These mirror the org skills; they are restated here so any agent landing in
-this repo sees them without a cross-repo fetch.
-
-- **All changes go through a PR.** Never push directly to `dev`, `uat`, or `main`.
-- **Branch strategy:** `feature/<name>` → `dev` → `uat` → `main`. Engineers
-  always target `dev` first.
-- **No self-merge contract.** The engineer who opened a PR clicks merge only
-  after the named reviewer (CI / QA / UAT / Security / CTO per phase)
-  approves. Issue-thread QA / UAT / security approvals do **not** clear the
-  Gitea `required_approvals` gate on `uat→main` — only a Gitea **Approve**
-  click from a member of the `approvals_whitelist_username` does. On this
-  repo that whitelist is `["gb_flea", "gb_dogfather"]` (engineer team).
-  Board-level accounts cannot give the Approve click by policy.
-- **Always include `cc @cpfarhood`** at the bottom of every PR body for
-  board visibility (not as a reviewer).
-- **Secrets in code are forbidden.** Use Bitnami Sealed Secrets; never commit
-  plaintext. See the `safety` skill.
-- **Production (`groombook` namespace) is Flux-managed.** Never
-  `kubectl apply` directly. Infrastructure changes go through PRs in
-  `groombook/infra`.
-
-## Local development
-
-See the repo's own README, package scripts, and CI workflow. The
-authoritative pipeline (Gitea Actions, image build, deploy hooks) is the
-shared `groombook/infra` overlay; do not reimplement it here.
-
-## When uncertain
-
-If a task conflicts with the org skills, **the org skills win**. Open an
-issue in `groombook/org` to propose a change rather than encoding a local
-exception.

CONTRIBUTING.md

@@ -1,117 +0,0 @@
-# Contributing to `groombook/api`
-
-Thanks for contributing. This document is the human-facing companion to
-[`AGENTS.md`](./AGENTS.md) and the authoritative
-[`groombook/org`](https://git.farh.net/groombook/org) skills. The org skills
-govern; this file is a quick-reference for the human/agent PR flow in this
-repo.
-
-## Branch strategy
-
-Three long-lived branches; one PR per promotion step.
-
-| Branch  | Environment | Who merges | Prerequisites for merge |
-|---------|-------------|-----------|-------------------------|
-| `dev`   | Dev         | Engineer  | CI passes               |
-| `uat`   | UAT         | Engineer  | QA code review approval |
-| `main`  | Production  | Engineer  | UAT validation + CTO Gitea Approve when the `uat→main` merge-gate policy applies (see below) |
-
-Engineers always target `dev` first. Feature branches: `<agent-name>/<short-description>`.
-
-## Phase-by-phase PR flow
-
-### Phase 1 — Dev
-
-1. Branch from `dev`: `git checkout -b <name>/<short-description> origin/dev`.
-2. Write code + tests. Run unit tests, type check, and lint locally (or rely on CI).
-3. Open a PR against `dev`:
-   ```bash
-   tea pr create --base dev --title "..." --body "..."
-   ```
-   Include `cc @cpfarhood` at the bottom of the body for board visibility.
-4. CI must pass. CI green → engineer self-merges.
-5. CI builds and deploys to Dev automatically.
-
-### Phase 2 — UAT promotion
-
-1. Open a PR from `dev` to `uat`.
-2. CI must pass.
-3. **QA (Lint Roller)** reviews and approves on the Gitea PR.
-4. QA approved → engineer self-merges.
-5. CI builds and deploys to UAT automatically.
-
-### Phase 3 — UAT regression + Security review
-
-1. **UAT (Shedward Scissorhands)** runs full regression against UAT — every
-   feature, old and new, no exceptions.
-2. **Security (Barkley Trimsworth)** reviews the changes.
-3. Failures in either gate bounce back to Phase 1.
-
-### Phase 4 — Production promotion (`uat → main`)
-
-This is the gate the org PR
-[`groombook/org#13`](https://git.farh.net/groombook/org/pulls/13) defines.
-The full rule is in
-[`groombook/org/skills/sdlc/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/sdlc/SKILL.md)
-and
-[`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md);
-the summary is below.
-
-**The CTO Gitea Approve click is NOT the default gate.** Once the four
-pre-gates (QA, UAT deploy, UAT regression, security) are green, the engineer
-self-merges.
-
-**A CTO Gitea Approve click IS required** only for PRs in one of three
-categories:
-
-1. **Novel auth / session paths** — login, OIDC, OOBE, session middleware,
-   token issuance, password reset, MFA, new auth provider integrations.
-   Routine auth-gated UI (button styling, error messages, form layout) is
-   **not** in this category.
-2. **Infra / prod-affecting merges** — deploys, infra manifests, secrets,
-   GitOps overlays, CI/CD, `main` branch protection, production
-   routing/ingress, prod state mutations. All Phase 5 infra overlay PRs in
-   `groombook/infra` require CTO Gitea Approve without exception.
-3. **Risk-flagged merges** — `risk:cto-approve` label, or explicit CTO/CEO
-   sign-off request in the PR or issue thread.
-
-The engineer opens the `uat→main` PR, classifies it against the three
-categories above, and adds `cc @cpfarhood`. If the PR is in scope, the CTO
-clicks Approve; once approved (and the four pre-gates are green), the
-engineer merges.
-
-### Phase 5 — Production deployment
-
-A separate PR in `groombook/infra` bumps the overlay image tag for prod.
-Handed to QA (Lint Roller) for review, then self-merged by the engineer.
-
-## The four pre-gates (uat→main)
-
-A `uat→main` PR is mergeable when **all four** are green:
-
-1. **QA code review** — done on the dev→uat promotion PR.
-2. **UAT deploy** — the UAT image built from the uat tip is live in UAT.
-3. **UAT regression** — Shedward's full-feature UAT pass is green (no
-   pre-existing defects, no new defects).
-4. **Security review** — Barkley's security code review is green.
-
-Issue-thread QA / UAT / security approvals do **not** clear the Gitea
-`required_approvals` gate. Only a Gitea **Approve** click from a member of
-the `approvals_whitelist_username` for `main` clears it. In this repo that
-whitelist is the engineer team (`gb_flea`, `gb_dogfather`).
-
-## Style, tests, and quality bar
-
-See
-[`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md)
-for the engineering priority ordering, test requirements, no-hardcoded-values
-rules, CalVer versioning policy, and the `git.farh.net` container registry
-policy.
-
-## Safety
-
-See
-[`groombook/org/skills/safety/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/safety/SKILL.md)
-for the non-negotiable rules: no plaintext secrets, no `kubectl apply` to
-`groombook`, no self-merge, no direct `tofu` runs, board approval for
-destructive actions, escalation protocol.

UAT_PLAYBOOK.md

@@ -65,8 +65,11 @@ Expected: one row, `role = 'groomer'`. If zero rows return, the request hit the
 | # | Scenario | Steps | Expected |
 |---|----------|-------|----------|
 | TC-API-0.1 | Unauthenticated health check | GET /api/health | 200 OK, `{"status":"ok"}` |
+| TC-API-0.2 | DB-touching readiness check — healthy (GRO-2678) | GET /api/readyz | 200 OK, `{"status":"ready"}` |
+| TC-API-0.3 | DB-touching readiness check — response body safe | GET /api/readyz and inspect body | Body contains only `status` and (on error) `check` fields — no raw SQL, driver messages, or stack traces |
 
 > **Note (GRO-1544):** Health endpoint registered on `api` basePath before auth middleware at `/api/health`. The old path `/health` was incorrect (routed to web pod via HTTPRoute `/*` rule).
+> **Note (GRO-2678):** `/api/readyz` is a separate DB-touching endpoint for monitoring. It is intentionally NOT used for K8s liveness/readiness probes — those remain DB-less to avoid pod cycling on transient DB blips.
 
 ### 4.1 Authentication
 

packages/db/src/reset.ts

@@ -32,7 +32,7 @@
  */
 import postgres from "postgres";
 import { drizzle } from "drizzle-orm/postgres-js";
-import { migrate } from "drizzle-orm/postgres-js/migrator";
+import { execSync } from "node:child_process";
 import { fileURLToPath } from "node:url";
 import { dirname, resolve } from "node:path";
 import * as schema from "./schema.js";
@@ -46,7 +46,6 @@ import {
 
 const __filename = fileURLToPath(import.meta.url);
 const __dirname = dirname(__filename);
-const MIGRATIONS_FOLDER = resolve(__dirname, "../migrations");
 
 async function reset() {
   const url = process.env.DATABASE_URL;
@@ -73,7 +72,7 @@ async function reset() {
   // across processes regardless of how many connections the work uses —
   // it does NOT require the work to share the lock's session.
   //
-  // Therefore `max` must be ≥ 2: 1 reserved for the lock + ≥1 free for
+  // Therefore `max` must be >= 2: 1 reserved for the lock + >=1 free for
   // the work. `max: 1` would let `reserve()` consume the only connection
   // and every query inside the callback would block forever waiting for
   // a connection that never frees (connection-starvation deadlock). We
@@ -122,7 +121,15 @@ async function reset() {
       console.log("✓ All tables and enums dropped\n");
 
       console.log("Running migrations...");
-      await migrate(db, { migrationsFolder: MIGRATIONS_FOLDER });
+      // GRO-2672: drizzle-orm's migrate() has a high-water-mark bug that skips
+      // migrations with stale `when` timestamps (0001, 0003, 0010, 0011). Use
+      // drizzle-kit instead -- it applies migrations by hash, matching the K8s
+      // migrate Job behaviour exactly.
+      execSync("pnpm exec drizzle-kit migrate", {
+        stdio: "inherit",
+        env: { ...process.env },
+        cwd: resolve(__dirname, ".."),
+      });
       console.log("✓ Migrations applied\n");
 
       console.log("Seeding database...");

src/__tests__/health-ready.test.ts

@@ -0,0 +1,102 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import { Hono } from "hono";
+
+// ─── Mock db module ───────────────────────────────────────────────────────────
+
+let selectImpl: () => Promise<unknown>;
+
+vi.mock("@groombook/db", () => {
+  const staff = new Proxy(
+    { _name: "staff" },
+    {
+      get(_target, prop) {
+        if (prop === "_name") return "staff";
+        return { table: "staff", column: prop };
+      },
+    }
+  );
+
+  return {
+    getDb: () => ({
+      select: (_fields: unknown) => ({
+        from: (_table: unknown) => ({
+          limit: (_n: number) => selectImpl(),
+        }),
+      }),
+    }),
+    staff,
+  };
+});
+
+// ─── Build test app ───────────────────────────────────────────────────────────
+
+async function makeApp() {
+  const { getDb, staff } = await import("@groombook/db");
+
+  const app = new Hono();
+  app.get("/health/ready", async (c) => {
+    try {
+      await getDb().select({ id: staff.id }).from(staff).limit(1);
+      return c.json({ status: "ready" }, 200);
+    } catch (err) {
+      const pgCode = (err as Record<string, unknown>).code ?? "unknown";
+      console.error("[health/ready] DB check failed:", pgCode);
+      return c.json({ status: "degraded" }, 503);
+    }
+  });
+  return app;
+}
+
+// ─── Tests ────────────────────────────────────────────────────────────────────
+
+describe("GET /health/ready", () => {
+  beforeEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  it("returns 200 {status:'ready'} when DB query succeeds", async () => {
+    selectImpl = () => Promise.resolve([{ id: "staff-1" }]);
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(200);
+    expect(body.status).toBe("ready");
+  });
+
+  it("returns 503 {status:'degraded'} when DB query throws (schema dropped)", async () => {
+    const schemaErr = Object.assign(new Error("relation \"staff\" does not exist"), { code: "42P01" });
+    selectImpl = () => Promise.reject(schemaErr);
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+
+    // Must not leak SQL error details in the response body
+    expect(JSON.stringify(body)).not.toContain("42P01");
+    expect(JSON.stringify(body)).not.toContain("relation");
+
+    consoleSpy.mockRestore();
+  });
+
+  it("returns 503 {status:'degraded'} on any DB connection error", async () => {
+    selectImpl = () => Promise.reject(new Error("ECONNREFUSED"));
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+
+    consoleSpy.mockRestore();
+  });
+});

src/__tests__/readyz.test.ts

@@ -0,0 +1,87 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import { Hono } from "hono";
+
+// ─── Mock db module ───────────────────────────────────────────────────────────
+
+let selectImpl: () => Promise<unknown>;
+
+vi.mock("@groombook/db", () => {
+  const staff = new Proxy(
+    { _name: "staff" },
+    {
+      get(_target, prop) {
+        if (prop === "_name") return "staff";
+        return { table: "staff", column: prop };
+      },
+    }
+  );
+
+  return {
+    getDb: () => ({
+      select: (_fields: unknown) => ({
+        from: (_table: unknown) => ({
+          limit: (_n: number) => selectImpl(),
+        }),
+      }),
+    }),
+    staff,
+  };
+});
+
+// ─── Build test app ───────────────────────────────────────────────────────────
+
+async function makeApp() {
+  // Import after mocks are in place
+  const { getDb, staff } = await import("@groombook/db");
+
+  const app = new Hono();
+  app.get("/api/readyz", async (c) => {
+    try {
+      await getDb().select({ id: staff.id }).from(staff).limit(1);
+      return c.json({ status: "ready" }, 200);
+    } catch (err) {
+      console.error("[readyz] DB check failed:", err);
+      return c.json({ status: "degraded", check: "db" }, 503);
+    }
+  });
+  return app;
+}
+
+// ─── Tests ────────────────────────────────────────────────────────────────────
+
+describe("GET /api/readyz", () => {
+  beforeEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  it("returns 200 {status:'ready'} when DB query succeeds", async () => {
+    selectImpl = () => Promise.resolve([{ id: "staff-1" }]);
+
+    const app = await makeApp();
+    const res = await app.request("/api/readyz", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(200);
+    expect(body.status).toBe("ready");
+  });
+
+  it("returns 503 {status:'degraded',check:'db'} when DB query throws", async () => {
+    selectImpl = () => Promise.reject(new Error("42P01: relation staff does not exist"));
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/api/readyz", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+    expect(body.check).toBe("db");
+
+    // Raw SQL / driver error must NOT appear in the response body
+    expect(JSON.stringify(body)).not.toContain("42P01");
+    expect(JSON.stringify(body)).not.toContain("relation");
+
+    consoleSpy.mockRestore();
+  });
+});

src/index.ts

@@ -65,6 +65,28 @@ app.use(
 app.get("/health", (c) => c.json({ status: "ok" }));
 // /api/health: used by Gateway HTTPRoute (/api/* → API pod)
 app.get("/api/health", (c) => c.json({ status: "ok" }));
+// /health/ready: DB-touching readiness probe — K8s removes pod from endpoints when schema is dropped (GRO-2689)
+app.get("/health/ready", async (c) => {
+  try {
+    await getDb().select({ id: staff.id }).from(staff).limit(1);
+    return c.json({ status: "ready" }, 200);
+  } catch (err) {
+    const pgCode = (err as Record<string, unknown>).code ?? "unknown";
+    console.error("[health/ready] DB check failed:", pgCode);
+    return c.json({ status: "degraded" }, 503);
+  }
+});
+// /api/readyz: DB-touching deep health check consumed by monitoring (not K8s probes)
+// Distinct from /health so a dropped schema triggers an alert without cycling pods (GRO-2678)
+app.get("/api/readyz", async (c) => {
+  try {
+    await getDb().select({ id: staff.id }).from(staff).limit(1);
+    return c.json({ status: "ready" }, 200);
+  } catch (err) {
+    console.error("[readyz] DB check failed:", err);
+    return c.json({ status: "degraded", check: "db" }, 503);
+  }
+});
 
 // Public booking routes — no auth required, must be registered before auth middleware
 app.route("/api/book", bookRouter);