fix(GRO-2652): boot ECONNRESET resilience — retry-with-backoff in initAuth, server-first startup #222

2026-08-05T08:22:26Z

Flea Flicker commented

2026-08-05 08:22:26 +00:00

Problem

PROD groombook/api pod (rev-50, 2026.06.26-98b1171) entered CrashLoopBackOff after a transient Postgres ECONNRESET at boot. The process exited immediately with no HTTP traffic served.

Root cause (two compounding issues):

src/index.ts: await initAuth() was a bare top-level ESM await. Any rejection propagated as an uncaught module-evaluation error → process.exit(1) before the server even bound to port 3000. /health was never reachable during the crash window, so K8s liveness/readiness probes failed immediately.
src/lib/auth.ts: The DB query inside initAuth() had no error handling. A single ECONNRESET from the auth_provider_config query caused authInitPromise to reject, propagating out to the top-level await above.

The src/lib/auth.ts file is byte-for-byte identical between rev-49 and rev-50 — the timing correlation with rev-50's deploy is incidental. Rollback to rev-49 would not fix the root bug; forward fix is mandatory (AC#2).

Changes

`src/lib/auth.ts`

Wraps the auth_provider_config DB query in retry-with-backoff: up to 5 attempts, exponential 1 s / 2 s / 4 s / 8 s delay
A single transient ECONNRESET no longer rejects authInitPromise

`src/index.ts`

serve() + console.log moved before initAuth() — HTTP server binds immediately so /health and all public routes are available from pod start
initAuth() now runs in a retry loop: up to 10 attempts, exponential 500 ms → 30 s
A permanent failure (all 10 attempts exhausted) logs a fatal message and breaks the loop — the process stays up and auth routes return 503 via the existing catch-to-503 in authRouter
startReminderScheduler() also moved before the auth retry loop so background jobs are not blocked

`UAT_PLAYBOOK.md`

New section 4.19 Boot Resilience — ECONNRESET Recovery (TC-API-19.1 through TC-API-19.8)
Covers: health before init, 503 graceful degradation during retry, no-crash-on-reset, log line verification, auth recovery after transient hiccup, public-route continuity

DB Migration Compatibility

GRO-2586 (rev-50) made zero schema changes. Rollback to rev-49 is schema-safe. These changes also make no schema changes — no migrations.

Test Coverage

TC-API-19.1–19.8 added to UAT_PLAYBOOK.md
Existing auth tests (TC-WEB-SSO-*) cover the golden path regression check (TC-API-19.7)

SDLC

Phase 1 of 4 — feature → dev. CI must pass before merge.

cc @cpfarhood

## Problem PROD `groombook/api` pod (rev-50, `2026.06.26-98b1171`) entered CrashLoopBackOff after a transient Postgres `ECONNRESET` at boot. The process exited immediately with no HTTP traffic served. **Root cause** (two compounding issues): 1. **`src/index.ts`**: `await initAuth()` was a bare top-level ESM `await`. Any rejection propagated as an uncaught module-evaluation error → `process.exit(1)` before the server even bound to port 3000. `/health` was never reachable during the crash window, so K8s liveness/readiness probes failed immediately. 2. **`src/lib/auth.ts`**: The DB query inside `initAuth()` had no error handling. A single `ECONNRESET` from the `auth_provider_config` query caused `authInitPromise` to reject, propagating out to the top-level await above. The `src/lib/auth.ts` file is byte-for-byte identical between rev-49 and rev-50 — the timing correlation with rev-50's deploy is incidental. Rollback to rev-49 would not fix the root bug; forward fix is mandatory (AC#2). ## Changes ### `src/lib/auth.ts` - Wraps the `auth_provider_config` DB query in retry-with-backoff: up to 5 attempts, exponential 1 s / 2 s / 4 s / 8 s delay - A single transient `ECONNRESET` no longer rejects `authInitPromise` ### `src/index.ts` - `serve()` + `console.log` moved **before** `initAuth()` — HTTP server binds immediately so `/health` and all public routes are available from pod start - `initAuth()` now runs in a retry loop: up to 10 attempts, exponential 500 ms → 30 s - A permanent failure (all 10 attempts exhausted) logs a fatal message and breaks the loop — the process stays up and auth routes return `503` via the existing `catch-to-503` in `authRouter` - `startReminderScheduler()` also moved before the auth retry loop so background jobs are not blocked ### `UAT_PLAYBOOK.md` - New section **4.19 Boot Resilience — ECONNRESET Recovery** (TC-API-19.1 through TC-API-19.8) - Covers: health before init, 503 graceful degradation during retry, no-crash-on-reset, log line verification, auth recovery after transient hiccup, public-route continuity ## DB Migration Compatibility GRO-2586 (rev-50) made zero schema changes. Rollback to rev-49 is schema-safe. These changes also make no schema changes — no migrations. ## Test Coverage - TC-API-19.1–19.8 added to `UAT_PLAYBOOK.md` - Existing auth tests (TC-WEB-SSO-*) cover the golden path regression check (TC-API-19.7) ## SDLC Phase 1 of 4 — feature → dev. CI must pass before merge. cc @cpfarhood

Flea Flicker added 3 commits 2026-08-05 08:22:27 +00:00

fix(GRO-2652): retry DB query in initAuth on transient ECONNRESET 632dadd064

The auth_provider_config DB query at boot has no error handling;
a transient ECONNRESET causes authInitPromise to reject, propagating
to the top-level await initAuth() and crashing the process (exit 1).

Add up to 5 retry attempts with exponential backoff (1 s, 2 s, 4 s, 8 s)
so a single connection reset does not abort initialization.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

fix(GRO-2652): start server before initAuth; retry auth init on ECONNRESET 095efb1cca

Previously: await initAuth() was a top-level ESM await. Any boot-time
ECONNRESET from Postgres propagated as an uncaught module-evaluation
error and killed the process before the server even started.

Now:
- serve() starts immediately so /health and public routes are available
- initAuth() runs in a retry loop (up to 10 attempts, exponential
  500 ms → 30 s); a permanent failure degrades to 503 on auth routes
  (existing catch-to-503 in authRouter) rather than crashing the pod

Co-Authored-By: Paperclip <noreply@paperclip.ing>

test(GRO-2652): add boot resilience UAT test cases (TC-API-19.x)

CI / Test (pull_request) Successful in 25s

Details

CI / Lint & Typecheck (pull_request) Successful in 28s

Details

CI / Build & Push Docker Images (pull_request) Successful in 56s

Details

8d42bfffc9

New section 4.19 verifies:
- /health available before initAuth completes
- auth routes return 503 (not crash) during init retry window
- pod restart count stays stable after deploy
- retry log lines emitted correctly
- auth recovers after transient DB hiccup

Co-Authored-By: Paperclip <noreply@paperclip.ing>

Flea Flicker added 1 commit 2026-08-05 08:54:03 +00:00

ci: add ignore-error=true to all cache-to targets 713e8bf415

Gitea OCI registry sporadically rejects cache blob writes with
"error writing layer blob: unknown" — this fails the build step
even though the image itself was pushed successfully. Adding
ignore-error=true makes cache write failures non-fatal so the
build proceeds regardless of registry-side cache issues.

Fixes recurring CI failure on Build and push Seed image step.

Flea Flicker added 1 commit 2026-08-05 08:56:32 +00:00

ci: trigger CI run for ignore-error=true cache-to fix af1af5c211

Flea Flicker closed this pull request

2026-08-05 08:58:07 +00:00

Flea Flicker reopened this pull request

2026-08-05 08:58:11 +00:00

Flea Flicker merged commit 299a157561 into dev

2026-08-05 09:01:13 +00:00

Flea Flicker referenced this pull request

2026-08-05 09:46:39 +00:00

fix(GRO-2652): reset authInitPromise on failure so retry loop actually retries #225

Sign in to join this conversation.