fix(GRO-2123): serialize seed.ts with Postgres advisory lock

The reset-demo-data CronJob in groombook-uat intermittently failed with
FK 23503 on invoice_tip_splits because two pods could run the seed
concurrently: the new pod's TRUNCATE deleted rows the old pod was still
inserting.

Acquire a session-level advisory lock for the full duration of the seed.
CRITICAL: with postgres-js connection pooling, a pg_advisory_lock
acquired on one pooled connection and released on a different one is a
no-op (the lock is bound to the pg-backend that took it). We therefore
reserve a dedicated connection for the lock, take pg_advisory_lock(KEY)
on it, run the seed on the pooled connections, and release the lock +
reserved connection in a try/finally so a thrown seed error cannot leak
the lock or the connection.

Defence-in-depth with the infra PR that switches
concurrencyPolicy: Replace → Forbid on the reset-demo-data CronJob.

- Adds withSeedAdvisoryLock helper and runSeedBody extracted function
- Wraps seed() body in the helper; client.end() runs after the lock
  releases so a reserved connection is not returned to a closed pool
- SEED_ADVISORY_LOCK_KEY = 0x47524f4f ("GROO" in ASCII) — arbitrary
  stable 32-bit key, referenced in runbooks
- UAT_PLAYBOOK.md §3.29 documents the regression check

cc @cpfarhood

Co-Authored-By: Paperclip <noreply@paperclip.ing>

UAT_PLAYBOOK.md

@@ -166,6 +166,7 @@ Expected: one row, `role = 'groomer'`. If zero rows return, the request hit the
 | TC-API-3.26 | Verify 25-35% medicalAlerts distribution | GET /api/pets (first 30 pets), count how many have non-empty medicalAlerts | Ratio is 25-35% (seed uses rand() < 0.3 for ~30% distribution) |
 | TC-API-3.27 | Verify coat_type enum has all seed values | After UAT seed completes, inspect the coat_type enum on the UAT DB — it must contain: short, medium, long, double, wire, silky, curly, hairless | UAT seed jobs (`reset-demo-data`, `seed-test-data`) complete 1/1 with no `enum_in` error; coat_type includes all 8 values used by seed.ts `coatTypePool` |
 | TC-API-3.28 | Verify pet_size_category enum has all seed values | After UAT seed completes, inspect the pet_size_category enum on the UAT DB — it must contain: small, medium, large, extra_large | UAT seed jobs (`reset-demo-data`, `seed-test-data`) complete 1/1 with no `enum_in` error; pet_size_category includes all 4 values used by seed.ts `petSizeCategoryPool` (regression for GRO-1999, mirrors TC-API-3.27) |
+| TC-API-3.29 | Verify `reset-demo-data` CronJob does not fail with FK 23503 on `invoice_tip_splits` (GRO-2123) | Trigger the CronJob manually: `kubectl create job --from=cronjob/reset-demo-data verify-gro2123 -n groombook-uat`. Wait for pod to terminate. Inspect logs: `kubectl logs -n groombook-uat -l job-name=verify-gro2123` | Pod reaches `Completed` state; logs show `✓ Acquired seed advisory lock` and `✓ Released seed advisory lock` from `seed.ts`; no `PostgresError: … violates foreign key constraint "invoice_tip_splits_invoice_id_invoices_id_fk"` (code 23503); final counts unchanged (500 clients, ~4000 invoices) |
 
 ### 4.4 Appointment Scheduling
 

packages/db/src/seed.ts

@@ -976,6 +976,63 @@ async function seedKnownUsers() {
 
 // ── Main seed ────────────────────────────────────────────────────────────────
 
+// ── GRO-2123: serialize reset+seed with a Postgres advisory lock ────────
+// The reset-demo-data CronJob runs on an hourly schedule. With
+// concurrencyPolicy=Replace, a new pod can start while the previous one
+// is still mid-seed; the new pod's TRUNCATE then deletes rows the old pod
+// is still inserting, producing FK 23503 errors non-deterministically
+// (see GRO-2123: invoice_tip_splits → invoices).
+//
+// We hold a session-level advisory lock for the full duration of the
+// seed so that overlapping invocations block then proceed in order —
+// not skip. The key is a stable 32-bit constant so it can be referenced
+// from runbooks without ambiguity and binds to the single-argument
+// `pg_advisory_lock(int)` form, which postgres-js serializes as a plain
+// number (no bigint type plumbing required).
+const SEED_ADVISORY_LOCK_KEY = 0x47524f4f; // "GROO" in ASCII — arbitrary, stable
+
+/**
+ * Reserve a dedicated connection from `pool`, take the seed advisory lock
+ * on it, run `fn`, and release the lock + connection in a try/finally.
+ *
+ * CRITICAL: with postgres-js connection pooling, a session-level
+ * `pg_advisory_lock(KEY)` acquired on one pooled connection and released
+ * on a *different* one is a no-op (the lock is bound to the session /
+ * pg-backend that took it). We therefore reserve a dedicated connection
+ * for the lock and release it from the same reserved connection. The
+ * seed work itself still runs on the pooled connections.
+ */
+async function withSeedAdvisoryLock<T>(
+  pool: ReturnType<typeof postgres>,
+  fn: () => Promise<T>,
+): Promise<T> {
+  const lockConnection = await pool.reserve();
+  let lockHeld = false;
+  try {
+    await lockConnection`SELECT pg_advisory_lock(${SEED_ADVISORY_LOCK_KEY})`;
+    lockHeld = true;
+    console.log(`✓ Acquired seed advisory lock (key=${SEED_ADVISORY_LOCK_KEY})`);
+    const result = await fn();
+    await lockConnection`SELECT pg_advisory_unlock(${SEED_ADVISORY_LOCK_KEY})`;
+    lockHeld = false;
+    console.log(`✓ Released seed advisory lock`);
+    return result;
+  } finally {
+    if (lockHeld) {
+      try {
+        await lockConnection`SELECT pg_advisory_unlock(${SEED_ADVISORY_LOCK_KEY})`;
+      } catch (err) {
+        console.error("Failed to release seed advisory lock during cleanup:", err);
+      }
+    }
+    try {
+      lockConnection.release();
+    } catch (err) {
+      console.error("Failed to release reserved lock connection:", err);
+    }
+  }
+}
+
 async function seed() {
   const url = process.env.DATABASE_URL;
   if (!url) {
@@ -993,6 +1050,22 @@ async function seed() {
   const client = postgres(url, { max: 5 });
   const db = drizzle(client, { schema });
 
+  // GRO-2123: hold the seed advisory lock for the full body of runSeedBody.
+  // See the withSeedAdvisoryLock comment for why a reserved connection is
+  // required (postgres-js pooling would silently drop the lock otherwise).
+  await withSeedAdvisoryLock(client, async () => {
+    return await runSeedBody(client, db, profile, cfg);
+  });
+
+  await client.end();
+}
+
+async function runSeedBody(
+  client: ReturnType<typeof postgres>,
+  db: ReturnType<typeof drizzle>,
+  profile: SeedProfile,
+  cfg: ProfileConfig,
+): Promise<void> {
   console.log(`Seeding Groom Book database (profile: ${profile})...\n`);
 
   // ── Staff ──
@@ -1614,8 +1687,6 @@ async function seed() {
   }
   console.log(`✓ Created ${visitLogCount} grooming visit logs`);
   console.log("\nSeed complete!");
-
-  await client.end();
 }
 
 seed().catch((err) => {