Mock global.fetch in the test-connection test so it no longer hits the
real network. The route has a 10s fetch timeout, but the test used the
default 5s vitest timeout, causing intermittent failures when DNS/network
was slow. Add afterEach(vi.restoreAllMocks) to clean up between tests,
and reconcile the misleading "returns 400" test title to match the 200
assertion the route actually produces.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
UAT_PLAYBOOK.md §3 — new test case TC-API-3.30:
trigger reset-demo-data CronJob manually, confirm job succeeds,
public tables/enums and drizzle schema survive, demo data reseeded,
/api/readyz stays 200.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
GRO-2722: packages/db/src/reset.ts and apps/api/src/db/reset.ts no longer
emit any DROP TABLE / DROP TYPE / DROP SCHEMA / DROP DATABASE DDL. The four
destructive DO-blocks are replaced with a single:
TRUNCATE <all public tables> RESTART IDENTITY CASCADE
Table names are enumerated dynamically via pg_tables WHERE schemaname='public'
so new tables are picked up automatically. The drizzle schema and
__drizzle_migrations table are untouched (different schema), keeping
drizzle-kit migrate a no-op on an already-migrated DB.
Preserves: production guard (NODE_ENV=production && ALLOW_RESET!='true'),
advisory lock, migrations (drizzle-kit migrate), and seed (runSeedBody).
Fixes the GRO-2678 prod outage root cause.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
/api/readyz is a public monitoring endpoint like /api/health.
app.basePath("/api") + api.use("*", authMiddleware) applies to all
/api/* paths regardless of route registration order (GRO-2692 UAT
failure: Hono basePath middleware bypass).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
/api/readyz was returning 401 Unauthorized in UAT (GRO-2692 Shedward
regression). The authMiddleware for /api/* did not whitelist /api/readyz,
causing every unauthenticated probe request to be rejected.
Add /api/readyz to the bypass condition alongside /api/auth/* and
/api/health. Add readyz-auth-bypass.test.ts confirming the route passes
through the middleware without auth and that non-whitelisted paths still
block (503 when auth not configured).
Closes GRO-2692 regression (UAT fail).
CTO architectural ruling (PR #235 review): K8s readiness/liveness probes
must remain DB-less to prevent transient DB blips from cycling pods. The
/api/readyz endpoint (GRO-2687) is the canonical DB-health signal for the
monitoring layer and satisfies the GRO-2678 detection-gap requirement.
Removes:
- GET /health/ready route from src/index.ts
- src/__tests__/health-ready.test.ts
Refs GRO-2689, GRO-2687, GRO-2678
Register GET /health/ready before the /api/* auth middleware so it is
public and reachable by K8s readinessProbe on port 3000 without auth.
On success → 200 {"status":"ready"}; on any DB/schema failure → 503
{"status":"degraded"}. Logs the pg error code; never leaks SQL in body.
A dropped schema (42P01) surfaces as non-200, closing the /health mask
that allowed the GRO-2678 incident to go undetected for ~43h.
Add health-ready.test.ts covering the 200 success path, 503 on schema
drop (42P01), and 503 on connection error; all assert no SQL leakage.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Register GET /api/readyz after /api/health in src/index.ts (before authMiddleware)
- Runs `getDb().select({id: staff.id}).from(staff).limit(1)` inside try/catch
- Success → 200 {"status":"ready"}; failure → 503 {"status":"degraded","check":"db"}
- Raw driver error is console.error'd but never leaked to the response body
- /health and /api/health are unchanged (K8s probes remain DB-less per CTO decision)
- New unit tests: mock getDb to resolve → assert 200/ready; throw → assert 503/degraded
- Updated UAT_PLAYBOOK.md §4.0 with TC-API-0.2 and TC-API-0.3
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Root cause: drizzle-orm's HWM-based migrate() marked migrations as applied in the ledger but rolled back schema changes when it detected a dirty state. Replacing with execSync("pnpm exec drizzle-kit migrate") matches the K8s migrate-schema Job, ensuring reset leaves a fully-migrated schema before seeding. Fixes reset-demo-data CronJob failures and resolves seed-test-data relation-does-not-exist reconcile loop.
QA approved: gb_lint (PR #229, head f7f90a7)
drizzle-orm's migrate() has a high-water-mark (HWM) bug: on a fresh DB,
migration 0000 sets the watermark to 2026-03-17. Migrations 0001, 0003,
0010, 0011 have stale 2025-era `when` timestamps and are silently
skipped. Migration 0003 (recurring_series) is the blocker — its skip
leaves `recurring_series`, `appointments.series_id`, and
`appointments.series_index` missing. A downstream migration inside
migrate()'s single Postgres transaction then fails, rolling back
everything including 0000's `staff` and `services` tables.
Replace the drizzle-orm migrate() call with `pnpm exec drizzle-kit
migrate` (hash-based). drizzle-kit applies every unhashed migration
regardless of `when` ordering, matching the K8s migrate Job exactly.
Merge origin/main into uat to resolve divergence (main was 27 commits
ahead). Only real conflict: UAT_PLAYBOOK.md — kept uat's §4.19 Boot
Resilience and §4.20 OOBE clients-from-auth test cases; excluded
.mcp.json and trigger-uat-1779751324.txt from merge result (CTO review
5042 — these must not land on main).
Co-Authored-By: Paperclip <noreply@paperclip.ing>
AbortSignal.timeout(5000) in initAuth's discovery fetch races with
vitest's 5000ms default test timeout, causing intermittent failures on CI push
runs. Stub fetch to return ok:false instantly so tests complete in <1s.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Previously the initAuth() function set authInitPromise to the async IIFE's
Promise but never cleared it on rejection. The index.ts retry loop called
initAuth() up to 10 times, but on attempt 2+ the check
`if (authInitPromise) { await authInitPromise; return; }` would immediately
re-throw the original rejection without performing a real retry.
Fix: wrap the final `await authInitPromise` in try/catch and reset
authInitPromise = null on error. This allows the index.ts retry loop to
create a fresh attempt on each call after a failure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Previously the initAuth() function set authInitPromise to the async IIFE's
Promise but never cleared it on rejection. The index.ts retry loop called
initAuth() up to 10 times, but on attempt 2+ the check
`if (authInitPromise) { await authInitPromise; return; }` would immediately
re-throw the original rejection without performing a real retry.
Fix: wrap the final `await authInitPromise` in try/catch and reset
authInitPromise = null on error. This allows the index.ts retry loop to
create a fresh attempt on each call after a failure.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Merges feature/GRO-2652-boot-econnreset-resilience into dev. CI passed (run #440). Fixes PROD CrashLoopBackOff caused by bare top-level await initAuth() + unhandled ECONNRESET in auth_provider_config DB query.
Gitea OCI registry sporadically rejects cache blob writes with
"error writing layer blob: unknown" — this fails the build step
even though the image itself was pushed successfully. Adding
ignore-error=true makes cache write failures non-fatal so the
build proceeds regardless of registry-side cache issues.
Fixes recurring CI failure on Build and push Seed image step.
Previously: await initAuth() was a top-level ESM await. Any boot-time
ECONNRESET from Postgres propagated as an uncaught module-evaluation
error and killed the process before the server even started.
Now:
- serve() starts immediately so /health and public routes are available
- initAuth() runs in a retry loop (up to 10 attempts, exponential
500 ms → 30 s); a permanent failure degrades to 503 on auth routes
(existing catch-to-503 in authRouter) rather than crashing the pod
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The auth_provider_config DB query at boot has no error handling;
a transient ECONNRESET causes authInitPromise to reject, propagating
to the top-level await initAuth() and crashing the process (exit 1).
Add up to 5 retry attempts with exponential backoff (1 s, 2 s, 4 s, 8 s)
so a single connection reset does not abort initialization.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Flea Flicker
Flea Flicker
Scrubs McBarkley