promote: dev → uat (GRO-2687 /api/readyz DB health check)

Merging dev→uat after QA approval (Lint Roller). All 6 CI checks green.

.mcp.json

@@ -1,11 +0,0 @@
-{
-  "mcpServers": {
-    "gitea": {
-      "type": "http",
-      "url": "https://git-mcp.farh.net/mcp",
-      "headers": {
-        "Authorization": "Bearer ${GITEA_TOKEN}"
-      }
-    }
-  }
-}

AGENTS.md

@@ -0,0 +1,54 @@
+# AGENTS.md
+
+This repository (`groombook/api`) is part of the GroomBook application stack. The
+authoritative process, quality bar, and safety rules live in the shared
+[`groombook/org`](https://git.farh.net/groombook/org) skills repository. Read
+those first; this file is only a pointer.
+
+## Authoritative skills
+
+- **SDLC (branching, PRs, phases, handoffs):**
+  [`groombook/org/skills/sdlc/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/sdlc/SKILL.md)
+- **Coding standards (priority ordering, PR discipline, tests, no-hardcoded-values, CalVer):**
+  [`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md)
+- **Safety (no plaintext secrets, no direct `kubectl apply` to `groombook`, no self-merge, board approval for destructive actions):**
+  [`groombook/org/skills/safety/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/safety/SKILL.md)
+
+For human contributors and humans reviewing agent work, see
+[`CONTRIBUTING.md`](./CONTRIBUTING.md) in this repo for the phase-by-phase PR
+flow and the `uat→main` merge-gate policy summary.
+
+## Non-negotiable operational rules
+
+These mirror the org skills; they are restated here so any agent landing in
+this repo sees them without a cross-repo fetch.
+
+- **All changes go through a PR.** Never push directly to `dev`, `uat`, or `main`.
+- **Branch strategy:** `feature/<name>` → `dev` → `uat` → `main`. Engineers
+  always target `dev` first.
+- **No self-merge contract.** The engineer who opened a PR clicks merge only
+  after the named reviewer (CI / QA / UAT / Security / CTO per phase)
+  approves. Issue-thread QA / UAT / security approvals do **not** clear the
+  Gitea `required_approvals` gate on `uat→main` — only a Gitea **Approve**
+  click from a member of the `approvals_whitelist_username` does. On this
+  repo that whitelist is `["gb_flea", "gb_dogfather"]` (engineer team).
+  Board-level accounts cannot give the Approve click by policy.
+- **Always include `cc @cpfarhood`** at the bottom of every PR body for
+  board visibility (not as a reviewer).
+- **Secrets in code are forbidden.** Use Bitnami Sealed Secrets; never commit
+  plaintext. See the `safety` skill.
+- **Production (`groombook` namespace) is Flux-managed.** Never
+  `kubectl apply` directly. Infrastructure changes go through PRs in
+  `groombook/infra`.
+
+## Local development
+
+See the repo's own README, package scripts, and CI workflow. The
+authoritative pipeline (Gitea Actions, image build, deploy hooks) is the
+shared `groombook/infra` overlay; do not reimplement it here.
+
+## When uncertain
+
+If a task conflicts with the org skills, **the org skills win**. Open an
+issue in `groombook/org` to propose a change rather than encoding a local
+exception.

CONTRIBUTING.md

@@ -0,0 +1,117 @@
+# Contributing to `groombook/api`
+
+Thanks for contributing. This document is the human-facing companion to
+[`AGENTS.md`](./AGENTS.md) and the authoritative
+[`groombook/org`](https://git.farh.net/groombook/org) skills. The org skills
+govern; this file is a quick-reference for the human/agent PR flow in this
+repo.
+
+## Branch strategy
+
+Three long-lived branches; one PR per promotion step.
+
+| Branch  | Environment | Who merges | Prerequisites for merge |
+|---------|-------------|-----------|-------------------------|
+| `dev`   | Dev         | Engineer  | CI passes               |
+| `uat`   | UAT         | Engineer  | QA code review approval |
+| `main`  | Production  | Engineer  | UAT validation + CTO Gitea Approve when the `uat→main` merge-gate policy applies (see below) |
+
+Engineers always target `dev` first. Feature branches: `<agent-name>/<short-description>`.
+
+## Phase-by-phase PR flow
+
+### Phase 1 — Dev
+
+1. Branch from `dev`: `git checkout -b <name>/<short-description> origin/dev`.
+2. Write code + tests. Run unit tests, type check, and lint locally (or rely on CI).
+3. Open a PR against `dev`:
+   ```bash
+   tea pr create --base dev --title "..." --body "..."
+   ```
+   Include `cc @cpfarhood` at the bottom of the body for board visibility.
+4. CI must pass. CI green → engineer self-merges.
+5. CI builds and deploys to Dev automatically.
+
+### Phase 2 — UAT promotion
+
+1. Open a PR from `dev` to `uat`.
+2. CI must pass.
+3. **QA (Lint Roller)** reviews and approves on the Gitea PR.
+4. QA approved → engineer self-merges.
+5. CI builds and deploys to UAT automatically.
+
+### Phase 3 — UAT regression + Security review
+
+1. **UAT (Shedward Scissorhands)** runs full regression against UAT — every
+   feature, old and new, no exceptions.
+2. **Security (Barkley Trimsworth)** reviews the changes.
+3. Failures in either gate bounce back to Phase 1.
+
+### Phase 4 — Production promotion (`uat → main`)
+
+This is the gate the org PR
+[`groombook/org#13`](https://git.farh.net/groombook/org/pulls/13) defines.
+The full rule is in
+[`groombook/org/skills/sdlc/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/sdlc/SKILL.md)
+and
+[`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md);
+the summary is below.
+
+**The CTO Gitea Approve click is NOT the default gate.** Once the four
+pre-gates (QA, UAT deploy, UAT regression, security) are green, the engineer
+self-merges.
+
+**A CTO Gitea Approve click IS required** only for PRs in one of three
+categories:
+
+1. **Novel auth / session paths** — login, OIDC, OOBE, session middleware,
+   token issuance, password reset, MFA, new auth provider integrations.
+   Routine auth-gated UI (button styling, error messages, form layout) is
+   **not** in this category.
+2. **Infra / prod-affecting merges** — deploys, infra manifests, secrets,
+   GitOps overlays, CI/CD, `main` branch protection, production
+   routing/ingress, prod state mutations. All Phase 5 infra overlay PRs in
+   `groombook/infra` require CTO Gitea Approve without exception.
+3. **Risk-flagged merges** — `risk:cto-approve` label, or explicit CTO/CEO
+   sign-off request in the PR or issue thread.
+
+The engineer opens the `uat→main` PR, classifies it against the three
+categories above, and adds `cc @cpfarhood`. If the PR is in scope, the CTO
+clicks Approve; once approved (and the four pre-gates are green), the
+engineer merges.
+
+### Phase 5 — Production deployment
+
+A separate PR in `groombook/infra` bumps the overlay image tag for prod.
+Handed to QA (Lint Roller) for review, then self-merged by the engineer.
+
+## The four pre-gates (uat→main)
+
+A `uat→main` PR is mergeable when **all four** are green:
+
+1. **QA code review** — done on the dev→uat promotion PR.
+2. **UAT deploy** — the UAT image built from the uat tip is live in UAT.
+3. **UAT regression** — Shedward's full-feature UAT pass is green (no
+   pre-existing defects, no new defects).
+4. **Security review** — Barkley's security code review is green.
+
+Issue-thread QA / UAT / security approvals do **not** clear the Gitea
+`required_approvals` gate. Only a Gitea **Approve** click from a member of
+the `approvals_whitelist_username` for `main` clears it. In this repo that
+whitelist is the engineer team (`gb_flea`, `gb_dogfather`).
+
+## Style, tests, and quality bar
+
+See
+[`groombook/org/skills/coding-standards/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/coding-standards/SKILL.md)
+for the engineering priority ordering, test requirements, no-hardcoded-values
+rules, CalVer versioning policy, and the `git.farh.net` container registry
+policy.
+
+## Safety
+
+See
+[`groombook/org/skills/safety/SKILL.md`](https://git.farh.net/groombook/org/src/branch/main/skills/safety/SKILL.md)
+for the non-negotiable rules: no plaintext secrets, no `kubectl apply` to
+`groombook`, no self-merge, no direct `tofu` runs, board approval for
+destructive actions, escalation protocol.

UAT_PLAYBOOK.md

@@ -65,8 +65,11 @@ Expected: one row, `role = 'groomer'`. If zero rows return, the request hit the
 | # | Scenario | Steps | Expected |
 |---|----------|-------|----------|
 | TC-API-0.1 | Unauthenticated health check | GET /api/health | 200 OK, `{"status":"ok"}` |
+| TC-API-0.2 | DB-touching readiness check — healthy (GRO-2678) | GET /api/readyz | 200 OK, `{"status":"ready"}` |
+| TC-API-0.3 | DB-touching readiness check — response body safe | GET /api/readyz and inspect body | Body contains only `status` and (on error) `check` fields — no raw SQL, driver messages, or stack traces |
 
 > **Note (GRO-1544):** Health endpoint registered on `api` basePath before auth middleware at `/api/health`. The old path `/health` was incorrect (routed to web pod via HTTPRoute `/*` rule).
+> **Note (GRO-2678):** `/api/readyz` is a separate DB-touching endpoint for monitoring. It is intentionally NOT used for K8s liveness/readiness probes — those remain DB-less to avoid pod cycling on transient DB blips.
 
 ### 4.1 Authentication
 

apps/api/src/__tests__/auth.test.ts

@@ -69,9 +69,14 @@ describe("auth init", () => {
   beforeEach(() => {
     dbSelectResult = [];
     vi.clearAllMocks();
+    // Stub fetch so OIDC discovery requests resolve instantly during tests.
+    // Without this, AbortSignal.timeout(5000) in auth.ts races with vitest's
+    // 5000ms default test timeout and causes flaky failures.
+    vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: false, status: 503 }));
   });
 
   afterEach(() => {
+    vi.unstubAllGlobals();
     process.env = { ...originalEnv };
   });
 

packages/db/src/reset.ts

@@ -32,7 +32,7 @@
  */
 import postgres from "postgres";
 import { drizzle } from "drizzle-orm/postgres-js";
-import { migrate } from "drizzle-orm/postgres-js/migrator";
+import { execSync } from "node:child_process";
 import { fileURLToPath } from "node:url";
 import { dirname, resolve } from "node:path";
 import * as schema from "./schema.js";
@@ -46,7 +46,6 @@ import {
 
 const __filename = fileURLToPath(import.meta.url);
 const __dirname = dirname(__filename);
-const MIGRATIONS_FOLDER = resolve(__dirname, "../migrations");
 
 async function reset() {
   const url = process.env.DATABASE_URL;
@@ -73,7 +72,7 @@ async function reset() {
   // across processes regardless of how many connections the work uses —
   // it does NOT require the work to share the lock's session.
   //
-  // Therefore `max` must be ≥ 2: 1 reserved for the lock + ≥1 free for
+  // Therefore `max` must be >= 2: 1 reserved for the lock + >=1 free for
   // the work. `max: 1` would let `reserve()` consume the only connection
   // and every query inside the callback would block forever waiting for
   // a connection that never frees (connection-starvation deadlock). We
@@ -122,7 +121,15 @@ async function reset() {
       console.log("✓ All tables and enums dropped\n");
 
       console.log("Running migrations...");
-      await migrate(db, { migrationsFolder: MIGRATIONS_FOLDER });
+      // GRO-2672: drizzle-orm's migrate() has a high-water-mark bug that skips
+      // migrations with stale `when` timestamps (0001, 0003, 0010, 0011). Use
+      // drizzle-kit instead -- it applies migrations by hash, matching the K8s
+      // migrate Job behaviour exactly.
+      execSync("pnpm exec drizzle-kit migrate", {
+        stdio: "inherit",
+        env: { ...process.env },
+        cwd: resolve(__dirname, ".."),
+      });
       console.log("✓ Migrations applied\n");
 
       console.log("Seeding database...");

src/__tests__/auth.test.ts

@@ -69,9 +69,14 @@ describe("auth init", () => {
   beforeEach(() => {
     dbSelectResult = [];
     vi.clearAllMocks();
+    // Stub fetch so OIDC discovery requests resolve instantly during tests.
+    // Without this, AbortSignal.timeout(5000) in auth.ts races with vitest's
+    // 5000ms default test timeout and causes flaky failures.
+    vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: false, status: 503 }));
   });
 
   afterEach(() => {
+    vi.unstubAllGlobals();
     process.env = { ...originalEnv };
   });
 

src/__tests__/health-ready.test.ts

@@ -0,0 +1,102 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import { Hono } from "hono";
+
+// ─── Mock db module ───────────────────────────────────────────────────────────
+
+let selectImpl: () => Promise<unknown>;
+
+vi.mock("@groombook/db", () => {
+  const staff = new Proxy(
+    { _name: "staff" },
+    {
+      get(_target, prop) {
+        if (prop === "_name") return "staff";
+        return { table: "staff", column: prop };
+      },
+    }
+  );
+
+  return {
+    getDb: () => ({
+      select: (_fields: unknown) => ({
+        from: (_table: unknown) => ({
+          limit: (_n: number) => selectImpl(),
+        }),
+      }),
+    }),
+    staff,
+  };
+});
+
+// ─── Build test app ───────────────────────────────────────────────────────────
+
+async function makeApp() {
+  const { getDb, staff } = await import("@groombook/db");
+
+  const app = new Hono();
+  app.get("/health/ready", async (c) => {
+    try {
+      await getDb().select({ id: staff.id }).from(staff).limit(1);
+      return c.json({ status: "ready" }, 200);
+    } catch (err) {
+      const pgCode = (err as Record<string, unknown>).code ?? "unknown";
+      console.error("[health/ready] DB check failed:", pgCode);
+      return c.json({ status: "degraded" }, 503);
+    }
+  });
+  return app;
+}
+
+// ─── Tests ────────────────────────────────────────────────────────────────────
+
+describe("GET /health/ready", () => {
+  beforeEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  it("returns 200 {status:'ready'} when DB query succeeds", async () => {
+    selectImpl = () => Promise.resolve([{ id: "staff-1" }]);
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(200);
+    expect(body.status).toBe("ready");
+  });
+
+  it("returns 503 {status:'degraded'} when DB query throws (schema dropped)", async () => {
+    const schemaErr = Object.assign(new Error("relation \"staff\" does not exist"), { code: "42P01" });
+    selectImpl = () => Promise.reject(schemaErr);
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+
+    // Must not leak SQL error details in the response body
+    expect(JSON.stringify(body)).not.toContain("42P01");
+    expect(JSON.stringify(body)).not.toContain("relation");
+
+    consoleSpy.mockRestore();
+  });
+
+  it("returns 503 {status:'degraded'} on any DB connection error", async () => {
+    selectImpl = () => Promise.reject(new Error("ECONNREFUSED"));
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/health/ready", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+
+    consoleSpy.mockRestore();
+  });
+});

src/__tests__/readyz.test.ts

@@ -0,0 +1,87 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import { Hono } from "hono";
+
+// ─── Mock db module ───────────────────────────────────────────────────────────
+
+let selectImpl: () => Promise<unknown>;
+
+vi.mock("@groombook/db", () => {
+  const staff = new Proxy(
+    { _name: "staff" },
+    {
+      get(_target, prop) {
+        if (prop === "_name") return "staff";
+        return { table: "staff", column: prop };
+      },
+    }
+  );
+
+  return {
+    getDb: () => ({
+      select: (_fields: unknown) => ({
+        from: (_table: unknown) => ({
+          limit: (_n: number) => selectImpl(),
+        }),
+      }),
+    }),
+    staff,
+  };
+});
+
+// ─── Build test app ───────────────────────────────────────────────────────────
+
+async function makeApp() {
+  // Import after mocks are in place
+  const { getDb, staff } = await import("@groombook/db");
+
+  const app = new Hono();
+  app.get("/api/readyz", async (c) => {
+    try {
+      await getDb().select({ id: staff.id }).from(staff).limit(1);
+      return c.json({ status: "ready" }, 200);
+    } catch (err) {
+      console.error("[readyz] DB check failed:", err);
+      return c.json({ status: "degraded", check: "db" }, 503);
+    }
+  });
+  return app;
+}
+
+// ─── Tests ────────────────────────────────────────────────────────────────────
+
+describe("GET /api/readyz", () => {
+  beforeEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  it("returns 200 {status:'ready'} when DB query succeeds", async () => {
+    selectImpl = () => Promise.resolve([{ id: "staff-1" }]);
+
+    const app = await makeApp();
+    const res = await app.request("/api/readyz", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(200);
+    expect(body.status).toBe("ready");
+  });
+
+  it("returns 503 {status:'degraded',check:'db'} when DB query throws", async () => {
+    selectImpl = () => Promise.reject(new Error("42P01: relation staff does not exist"));
+
+    const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {});
+
+    const app = await makeApp();
+    const res = await app.request("/api/readyz", { method: "GET" });
+    const body = (await res.json()) as Record<string, unknown>;
+
+    expect(res.status).toBe(503);
+    expect(body.status).toBe("degraded");
+    expect(body.check).toBe("db");
+
+    // Raw SQL / driver error must NOT appear in the response body
+    expect(JSON.stringify(body)).not.toContain("42P01");
+    expect(JSON.stringify(body)).not.toContain("relation");
+
+    consoleSpy.mockRestore();
+  });
+});

src/index.ts

@@ -65,6 +65,28 @@ app.use(
 app.get("/health", (c) => c.json({ status: "ok" }));
 // /api/health: used by Gateway HTTPRoute (/api/* → API pod)
 app.get("/api/health", (c) => c.json({ status: "ok" }));
+// /health/ready: DB-touching readiness probe — K8s removes pod from endpoints when schema is dropped (GRO-2689)
+app.get("/health/ready", async (c) => {
+  try {
+    await getDb().select({ id: staff.id }).from(staff).limit(1);
+    return c.json({ status: "ready" }, 200);
+  } catch (err) {
+    const pgCode = (err as Record<string, unknown>).code ?? "unknown";
+    console.error("[health/ready] DB check failed:", pgCode);
+    return c.json({ status: "degraded" }, 503);
+  }
+});
+// /api/readyz: DB-touching deep health check consumed by monitoring (not K8s probes)
+// Distinct from /health so a dropped schema triggers an alert without cycling pods (GRO-2678)
+app.get("/api/readyz", async (c) => {
+  try {
+    await getDb().select({ id: staff.id }).from(staff).limit(1);
+    return c.json({ status: "ready" }, 200);
+  } catch (err) {
+    console.error("[readyz] DB check failed:", err);
+    return c.json({ status: "degraded", check: "db" }, 503);
+  }
+});
 
 // Public booking routes — no auth required, must be registered before auth middleware
 app.route("/api/book", bookRouter);

src/lib/auth.ts

@@ -329,5 +329,10 @@ export async function initAuth(): Promise<void> {
     });
   })();
 
-  await authInitPromise;
+  try {
+    await authInitPromise;
+  } catch (err) {
+    authInitPromise = null; // allow retry on next call
+    throw err;
+  }
 }