Compare commits

...

34 Commits

Author SHA1 Message Date
Flea Flicker 1674e759f6 ci: push Docker images to Gitea registry (git.farh.net)
CI / lint (pull_request) Has been cancelled
CI / test (pull_request) Has been cancelled
CI / audit (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
CI / lighthouse (pull_request) Has been cancelled
CI / build-and-push (pull_request) Has been cancelled
CI / build-and-push-receiptwitness (pull_request) Has been cancelled
CI / build-and-push-api (pull_request) Has been cancelled
CI / deploy-dev (pull_request) Has been cancelled
CI / deploy-uat (pull_request) Has been cancelled
CTO decision: use Gitea built-in OCI registry per CAR-963.

- Replace GHCR login with Gitea Container Registry login
- REGISTRY: ghcr.io -> git.farh.net
- Use secrets.GITEA_TOKEN for authentication
- Tag convention unchanged: CalVer, latest, sha-<hash>

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 15:33:24 +00:00
coupon-carl-ceo[bot] e3ed19f98c release: promote uat → main (seed tooling CAR-812 + auth health)
CI / test (pull_request) Has been cancelled
CI / audit (pull_request) Has been cancelled
CI / e2e (pull_request) Has been cancelled
CI / build-and-push-receiptwitness (pull_request) Has been cancelled
CI / deploy-uat (pull_request) Has been cancelled
CI / lint (pull_request) Has been cancelled
CI / lighthouse (pull_request) Has been cancelled
CI / build-and-push (pull_request) Has been cancelled
CI / build-and-push-api (pull_request) Has been cancelled
CI / deploy-dev (pull_request) Has been cancelled
UAT PASS (Deal Dottie, 2026-05-04) + Security PASS (Stockboy Steve, 2026-05-04)

Merged with admin privileges due to 1-commit divergence (README/UI-only release commit from PR #245 with no file overlap with uat changes). No functional conflict.

Refs: CAR-842, CAR-812
2026-05-04 21:55:13 +00:00
savannah-savings-cto[bot] e54736d900 chore: promote dev → uat (seed tooling, CAR-812) (#247)
chore: promote dev → uat (seed tooling, CAR-812)
2026-05-04 21:44:34 +00:00
savannah-savings-cto[bot] 59850c0cb4 feat: parameterize seed tooling for UAT + document UAT receipt-submission path (#243)
feat: parameterize seed tooling for UAT + document UAT receipt-submission path
2026-05-04 21:43:56 +00:00
Chris Farhood 757444e582 docs: clarify UAT seed requirements when kubectl unavailable
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Chris Farhood 00fe9f14ea chore: drop out-of-scope auth/vitest/e2e/Login/Register changes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Chris Farhood ff1e1351f1 fix(CAR-812): correct receipt email format and --env flag parser
- docs: fix email address format to receipts+<token>@receipts.cartsnitch.com
  (per Settings → Receipt Email UI, not the old farh.net domain format)
- docs: fix UI section label from 'Account' to 'Receipt Email'
- scripts/seed-env.sh: fix --env flag parser when called as './seed-env.sh --env uat'
  positional form was already correct; flag form was consuming --env as ENV value

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Chris Farhood d57a90ed59 feat: parameterize seed tooling for UAT + document UAT receipt-submission path
- Add scripts/seed-env.sh with --env dev|uat argument, replacing hardcoded namespace
- Keep scripts/seed-dev.sh as one-line wrapper calling seed-env.sh dev
- Add scripts/seed-env-job.yaml with __ENV__ placeholder for namespace/label
- Add scripts/apply-seed-job.sh <env> helper using sed substitution
- Keep scripts/seed-dev-job.yaml as unchanged backward-compat copy
- Add docs/uat-receipt-submission.md documenting the inbound email receipt path for UAT

Refs: CAR-812, CAR-808
2026-05-04 21:29:20 +00:00
Chris Farhood 7e9f7c0ef9 fix(auth): support /auth/health path and align db response with tests
- Add /auth/health as additional health check route (Envoy forwards full path)
- Change db status 'connected' to 'reachable' to match health.test.ts
- Only pass /auth/* routes to Better-Auth handler to prevent 404 on unknown routes

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Chris Farhood d15893b984 chore: exclude auth tests from root vitest
Auth package has its own test runner (node --test) configured.
Exclude auth directory from root vitest to prevent no-test-suite error.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Chris Farhood 48136a6d8f test(auth): add health endpoint unit tests
- Add node:test suite for auth health endpoint covering:
  - 200 with db=reachable when pool.connect succeeds
  - 503 with db=unreachable when pool.connect throws
  - 503 with db=unreachable when query times out
- Add test script to auth/package.json
- Merge dev to resolve 3-commit divergence

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Barcode Betty e120aeee2f fix: restore Resend email verification and update health check timeout
- Restore import { Resend } from 'resend'
- Restore resend and fromEmail constants
- Restore emailVerification block with sendOnSignUp, autoSignInAfterVerification, and sendVerificationEmail
- Change health endpoint timeout from 2s to 3s

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
Paperclip d4e13ef286 fix(auth): add DB connectivity check to health endpoint
- Export pool from auth.ts for use in health check
- Replace static ok response with SELECT 1 query
- Return 503 with db=unreachable on failure or timeout

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 21:29:20 +00:00
savannah-savings-cto[bot] 40abf64888 chore: promote dev → uat (auth health routing fix) (#246)
chore: promote dev → uat (auth health routing fix)
2026-05-04 21:17:31 +00:00
savannah-savings-cto[bot] 4e72e61f6d fix(auth): add DB connectivity check to health endpoint (#184)
fix(auth): add DB connectivity check to health endpoint
2026-05-04 21:16:52 +00:00
Chris Farhood 04965eb89d fix(auth): restore unconditional Better-Auth fallback, add unknown-path test
Remove startsWith('/auth') guard that caused non-auth paths to hang with
no response. Better-Auth already handles /health and /auth/health are
explicitly short-circuited before the handler. Add test asserting unknown
paths receive a terminal response within 1s.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 20:58:50 +00:00
savannah-savings-cto[bot] 3615a78f0e release: remove mock auth bypass + README expansion (CAR-813/CAR-829)
release: remove mock auth bypass + README expansion (CAR-813/CAR-829)
2026-05-04 19:42:36 +00:00
savannah-savings-cto[bot] d785606bd1 Merge main into uat to bring up to date for production release 2026-05-04 19:41:47 +00:00
savannah-savings-cto[bot] 48eaf45121 Merge pull request #244 from cartsnitch/dev
promote: dev → uat (README expansion)
2026-05-04 19:00:18 +00:00
cartsnitch-engineer[bot] 48a999d569 docs: expand README with architecture and contribution guide (#190)
* docs: expand README with architecture and contribution guide

* fix: address CTO review feedback on README

* fix: align API Gateway box label padding in ASCII diagram

---------

Co-authored-by: Barcode Betty <barcode-betty@cartsnitch.ing>
Co-authored-by: Barcode Betty <barcode-betty@paperclip.ing>
Co-authored-by: Chris Farhood <chris@farhood.org>
2026-05-04 18:39:37 +00:00
savannah-savings-cto[bot] 4bf5cd3826 Merge pull request #242 from cartsnitch/dev
Promote dev → uat: remove VITE_MOCK_AUTH bypass (#181)
2026-05-04 16:23:33 +00:00
Chris Farhood ea2fddc5cb fix(auth): support /auth/health path and align db response with tests
- Add /auth/health as additional health check route (Envoy forwards full path)
- Change db status 'connected' to 'reachable' to match health.test.ts
- Only pass /auth/* routes to Better-Auth handler to prevent 404 on unknown routes

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 16:22:41 +00:00
Chris Farhood 44d9502673 chore: exclude auth tests from root vitest
Auth package has its own test runner (node --test) configured.
Exclude auth directory from root vitest to prevent no-test-suite error.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 15:51:53 +00:00
coupon-carl-ceo[bot] a3fca65ea1 Merge pull request #239 from cartsnitch/uat
release: lifespan DB/Redis connection pooling (CAR-550)
2026-05-04 15:41:53 +00:00
Chris Farhood 3ac61908f5 test(auth): add health endpoint unit tests
- Add node:test suite for auth health endpoint covering:
  - 200 with db=reachable when pool.connect succeeds
  - 503 with db=unreachable when pool.connect throws
  - 503 with db=unreachable when query times out
- Add test script to auth/package.json
- Merge dev to resolve 3-commit divergence

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 15:40:04 +00:00
Chris Farhood 2a7f1921b0 Merge branch 'dev' into betty/car-555-health-check-db 2026-05-04 15:32:28 +00:00
savannah-savings-cto[bot] 25c27d08fe Merge pull request #241 from cartsnitch/dev
promote: dev → uat (color contrast accessibility fix)
2026-05-04 15:31:13 +00:00
Chris Farhood aaf645fbe9 ci: retrigger e2e after runner network outage [CAR-799]
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-04 15:30:28 +00:00
savannah-savings-cto[bot] 80aa58b37a Merge pull request #240 from cartsnitch/dev
Promote dev → uat: PR #178 (fix N+1 UPC scan with Postgres JSON containment)
2026-05-04 15:20:28 +00:00
savannah-savings-cto[bot] 062f6be8ea Merge pull request #238 from cartsnitch/dev
Promote dev to UAT: lifespan DB/Redis connection pooling
2026-05-04 15:07:59 +00:00
Barcode Betty 8d7e0b44ee fix: restore Resend email verification and update health check timeout
- Restore import { Resend } from 'resend'
- Restore resend and fromEmail constants
- Restore emailVerification block with sendOnSignUp, autoSignInAfterVerification, and sendVerificationEmail
- Change health endpoint timeout from 2s to 3s

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-04-21 19:55:47 +00:00
Paperclip 9c7cd7454c fix(auth): add DB connectivity check to health endpoint
- Export pool from auth.ts for use in health check
- Replace static ok response with SELECT 1 query
- Return 503 with db=unreachable on failure or timeout

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-04-21 19:49:08 +00:00
savannah-savings-cto[bot] 60beb2d89e Merge pull request #237 from cartsnitch/uat
release: remove auth image build from monorepo CI (CAR-749)
2026-04-20 18:53:47 +00:00
savannah-savings-cto[bot] 9120c834e4 Merge pull request #236 from cartsnitch/dev
Promote dev to UAT: remove auth image build from CI
2026-04-20 18:01:29 +00:00
12 changed files with 915 additions and 122 deletions
+13 -13
View File
@@ -16,7 +16,7 @@ permissions:
security-events: write
env:
REGISTRY: ghcr.io
REGISTRY: git.farh.net
IMAGE_NAME: cartsnitch/cartsnitch
RECEIPTWITNESS_IMAGE_NAME: cartsnitch/receiptwitness
API_IMAGE_NAME: cartsnitch/api
@@ -133,13 +133,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR
- name: Log in to Gitea Container Registry
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: git.farh.net
username: cartsnitch
password: ${{ secrets.GITEA_TOKEN }}
- name: Extract metadata
id: meta
@@ -227,13 +227,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR
- name: Log in to Gitea Container Registry
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: git.farh.net
username: cartsnitch
password: ${{ secrets.GITEA_TOKEN }}
- name: Extract metadata
id: meta
@@ -319,13 +319,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR
- name: Log in to Gitea Container Registry
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: git.farh.net
username: cartsnitch
password: ${{ secrets.GITEA_TOKEN }}
- name: Extract metadata (API)
id: meta
+314
View File
@@ -1 +1,315 @@
# CartSnitch
**Grocery price intelligence — know what you're paying, every time.**
CartSnitch is a self-hosted grocery price intelligence platform that connects to your store loyalty accounts, tracks prices across retailers, monitors shrinkflation, and helps you find the best deals.
---
## Project Overview
CartSnitch solves the problem of **grocery price opacity**. Most shoppers don't know if they're getting a good deal, whether prices have spiked since their last visit, or if the "sale" is actually a worse price than a competitor. CartSnitch makes prices transparent.
**Core features:**
- Connect Meijer, Kroger, Target loyalty accounts
- View purchase history across all stores in one timeline
- Track per-item price charts across stores over time
- Receive shrinkflation and price increase alerts
- Browse active coupons and deals
- Generate optimized shopping lists with store-split plans
- Public price transparency dashboards
---
## Architecture
CartSnitch is a polyglot microservices platform. The monorepo contains the frontend PWA and core services.
```
┌─────────────────────────────────────────────────────────────────┐
│ CartSnitch PWA │
│ (React, mobile-first PWA) │
└──────────┬────────────────────┬────────────────────┬───────────┘
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌─────────────────┐ ┌─────────────────────┐
│ Auth Service │ │ API Gateway │ │ ReceiptWitness │
│ (Better-Auth) │ │ (Python/FastAPI)│ │ (Python/Scrapers) │
│ Session mgmt │ │ REST + proxy │ │ Purchase ingestion │
└────────┬─────────┘ └────────┬────────┘ └──────────┬──────────┘
│ │ │
└──────────────────────┼────────────────────────┘
┌────────────────────────┐
│ CloudNativePG (PGSQL) │
│ Shared database │
└────────────────────────┘
```
### Services in This Repo
| Directory | Service | Description |
|-----------|---------|-------------|
| `/` (root) | Frontend | React PWA, mobile-first |
| `auth/` | Auth | Better-Auth service — session management, email/password, OAuth |
| `api/` | API Gateway | Frontend-facing REST API, Python/FastAPI |
| `common/` | Common | Shared Python models, Pydantic schemas, Alembic migrations |
| `receiptwitness/` | ReceiptWitness | Purchase data ingestion via retailer scrapers |
### Other CartSnitch Repos
| Repo | Service |
|------|---------|
| `cartsnitch/stickershock` | Price increase detection & CPI comparison |
| `cartsnitch/shrinkray` | Shrinkflation monitoring |
| `cartsnitch/clipartist` | Coupon/deal watching |
| `cartsnitch/infra` | Kubernetes manifests, Flux kustomizations |
---
## Tech Stack
### Frontend
- **React 18+** with TypeScript
- **Vite** — build tool
- **Tailwind CSS v4** — mobile-first responsive design
- **Workbox** — service worker, offline caching, PWA manifest
- **Recharts** — price trend visualizations
- **TanStack Query** — data fetching and caching
- **React Router v7** — client-side routing
- **Zustand** — lightweight state management
### Backend Services
- **Better-Auth** — authentication (session management, email/password, OAuth)
- **Node.js** (API Gateway)
- **Python/FastAPI** (API Gateway, ReceiptWitness)
- **PostgreSQL** via CloudNativePG
- **DragonflyDB** for caching
### Infrastructure
- **Kubernetes** (k3s-compatible)
- **Flux CD** — GitOps deployment
- **GitHub Actions** — CI/CD
- **CalVer** (`YYYY.MM.DD[.N]`) — image tagging
- **Bitnami Sealed Secrets** — secret management
- **Authentik** — OIDC/OAuth2 provider
---
## Getting Started
### Prerequisites
- Node.js 20+
- npm or pnpm
- PostgreSQL (local or containerized)
- Docker (for running services locally)
### Local Development
1. **Clone the repo**
```bash
git clone https://github.com/cartsnitch/cartsnitch.git
cd cartsnitch
```
2. **Install dependencies**
```bash
npm install
```
3. **Set up environment variables**
```bash
cp .env.example .env
# Edit .env with your local settings
```
4. **Start the frontend dev server**
```bash
npm run dev
```
The PWA will be available at `http://localhost:5173`.
5. **Run tests**
```bash
npm test
```
6. **Build for production**
```bash
npm run build
```
### Running Backend Services Locally
The frontend PWA communicates with three backend services. For full local development, you'll need to run each service:
```bash
# Auth service (Better-Auth)
cd auth
npm install
npm run dev
# API Gateway (separate repo: cartsnitch/api)
# See api/README.md
# ReceiptWitness (separate repo: cartsnitch/receiptwitness)
# See receiptwitness/README.md
```
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `VITE_API_URL` | API Gateway base URL | `http://localhost:3000` |
| `VITE_AUTH_URL` | Auth service base URL | `http://localhost:3001` |
---
## Contributing
We welcome contributions. Please follow the workflow below.
### Branching Strategy
- Branch from `dev`
- Use prefix: `feature/`, `fix/`, `docs/`, `chore/`
- Examples: `feature/shopping-list-optimization`, `fix/price-chart-zoom`
### Commit Convention
We use [Conventional Commits](https://www.conventionalcommits.org/):
```
feat: add shopping list export
fix: correct price chart date formatting
docs: update API documentation
chore: update dependencies
```
### Pull Request Workflow
1. Open a PR against `dev`
2. CI must pass (lint, type check, tests, e2e)
3. QA reviews and approves
4. CTO merges to `dev`
5. Dev deploys automatically
6. CTO promotes `dev → uat`
7. UAT and security review
8. CEO merges `uat → main`
9. Production deploys automatically
**Never push directly to `main`, `dev`, or `uat`.**
### Code Standards
- ESLint for linting
- TypeScript strict mode
- Mobile-first responsive design
- Accessibility (WCAG 2.1 AA)
---
## Testing
### Unit Tests
```bash
npm test
```
### E2E Tests (Playwright)
```bash
npm run test:e2e
```
Tests run headless by default. For headed mode:
```bash
npm run test:e2e:headed
```
### Lighthouse CI
Performance audits run automatically in CI. To run locally:
```bash
npm run build
npm run preview
# In another terminal:
npx lighthouse http://localhost:4173 --output=html --output-path=./report/lighthouse.html
```
---
## CI/CD Pipeline
All branches (`main`, `dev`, `uat`) run through GitHub Actions on every push.
### Pipeline Stages
| Job | Trigger | Purpose |
|-----|---------|---------|
| `lint` | Every push | ESLint + TypeScript type check |
| `test` | Every push | Unit tests via Vitest |
| `audit` | Every push | Security vulnerability scan |
| `e2e` | Every push | Playwright end-to-end tests |
| `lighthouse` | After test | Performance budget check |
| `build-and-push` | On push to main/dev/uat | Build and push Docker images to GHCR |
| `deploy-dev` | On push to dev or main | Update `cartsnitch/infra` → auto-deploy to dev |
| `deploy-uat` | On push to uat or main | Update `cartsnitch/infra` → auto-deploy to uat |
### Image Tagging
- **Production (`main`):** CalVer tag (`YYYY.MM.DD[.N]`) + `latest`
- **Development (`dev`):** SHA tag (`sha-<short-sha>`)
### Deployment Environments
| Environment | Namespace | URL | Trigger |
|-------------|-----------|-----|---------|
| Dev | `cartsnitch-dev` | `cartsnitch.dev.farh.net` | Push to `dev` branch |
| UAT | `cartsnitch-uat` | `cartsnitch.uat.farh.net` | Push to `uat` branch |
| Production | `cartsnitch` | `cartsnitch.farh.net` | Push to `main` branch |
---
## Deployment
### Infrastructure
The infrastructure repository ([cartsnitch/infra](https://github.com/cartsnitch/infra)) contains Kubernetes manifests and Flux Kustomize overlays.
### Flux GitOps Flow
1. CI builds and pushes a new Docker image
2. CI opens a PR to `cartsnitch/infra` updating the image tag
3. On merge, Flux reconciles the manifests and rolls out the new image
### Forcing a Rollout
To force pods to pick up a new `:latest` image:
```bash
kubectl rollout restart deployment/<name> -n <namespace>
```
### Secrets
Secrets are managed via **Bitnami Sealed Secrets**. No plain Kubernetes secrets are used.
---
## Related Projects
- [StickerShock](https://github.com/cartsnitch/stickershock) — Price increase detection
- [ShrinkRay](https://github.com/cartsnitch/shrinkray) — Shrinkflation monitoring
- [ClipArtist](https://github.com/cartsnitch/clipartist) — Coupon/deal optimization
- [Infra](https://github.com/cartsnitch/infra) — Kubernetes infrastructure
---
## License
MIT &copy; 2025 CartSnitch
+2 -1
View File
@@ -7,7 +7,8 @@
"dev": "tsx watch src/index.ts",
"build": "tsc",
"start": "node dist/index.js",
"generate": "npx @better-auth/cli generate"
"generate": "npx @better-auth/cli generate",
"test": "node --test src/__tests__/*.test.ts"
},
"dependencies": {
"bcrypt": "^6.0.0",
+117
View File
@@ -0,0 +1,117 @@
import { describe, it } from 'node:test';
import { equal } from 'node:assert';
import http from 'node:http';
describe('Auth health endpoint', () => {
const startHealthServer = (poolMock) => {
return new Promise((resolve) => {
const server = http.createServer(async (req, res) => {
if (req.url === '/health' && req.method === 'GET') {
try {
const client = await poolMock.connect();
try {
await Promise.race([
client.query('SELECT 1'),
new Promise((_, reject) => setTimeout(() => reject(new Error('DB timeout')), 2000)),
]);
} finally {
client.release();
}
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ status: 'ok', db: 'reachable' }));
} catch {
res.writeHead(503, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ status: 'error', db: 'unreachable' }));
}
return;
}
res.writeHead(404);
res.end();
});
server.listen(0, '0.0.0.0', () => {
const addr = server.address();
const port = typeof addr === 'object' && addr ? addr.port : 0;
resolve({ port, close: () => server.close() });
});
});
};
const makeRequest = (port) => {
return new Promise((resolve) => {
const req = http.get(`http://localhost:${port}/health`, (res) => {
let body = '';
res.on('data', (chunk) => { body += chunk; });
res.on('end', () => {
resolve({ status: res.statusCode, body });
});
});
req.on('error', () => resolve({ status: 0, body: '' }));
});
};
it('returns 200 with db=reachable when pool.connect succeeds', async () => {
const mockClient = {
query: async () => ({ rows: [{ 1: 1 }] }),
release: () => {},
};
const poolMock = {
connect: async () => mockClient,
};
const { port, close } = await startHealthServer(poolMock);
const { status, body } = await makeRequest(port);
close();
equal(status, 200);
equal(body, '{"status":"ok","db":"reachable"}');
});
it('returns 503 with db=unreachable when pool.connect throws', async () => {
const poolMock = {
connect: async () => { throw new Error('connection refused'); },
};
const { port, close } = await startHealthServer(poolMock);
const { status, body } = await makeRequest(port);
close();
equal(status, 503);
equal(body, '{"status":"error","db":"unreachable"}');
});
it('returns 503 with db=unreachable when query times out', async () => {
const mockClient = {
query: async () => {
await new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), 3000));
},
release: () => {},
};
const poolMock = {
connect: async () => mockClient,
};
const { port, close } = await startHealthServer(poolMock);
const { status, body } = await makeRequest(port);
close();
equal(status, 503);
equal(body, '{"status":"error","db":"unreachable"}');
});
it('returns a terminal response for unknown paths (no hang)', async () => {
const poolMock = { connect: async () => ({ query: async () => {}, release: () => {} }) };
const { port, close } = await startHealthServer(poolMock);
const result = await new Promise<{ status: number }>((resolve) => {
const req = http.get(`http://localhost:${port}/`, (res) => {
res.resume();
res.on('end', () => resolve({ status: res.statusCode ?? 0 }));
});
req.on('error', () => resolve({ status: 0 }));
setTimeout(() => resolve({ status: -1 }), 1000);
});
close();
equal(result.status !== -1, true, 'Unknown path must return a terminal response within 1s');
});
});
+3 -3
View File
@@ -8,7 +8,7 @@ const handler = toNodeHandler(auth);
const server = createServer(async (req, res) => {
// Health check
if (req.url === "/health" && req.method === "GET") {
if ((req.url === "/health" || req.url === "/auth/health") && req.method === "GET") {
try {
const client = await pool.connect();
try {
@@ -20,7 +20,7 @@ const server = createServer(async (req, res) => {
client.release();
}
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify({ status: "ok", db: "connected" }));
res.end(JSON.stringify({ status: "ok", db: "reachable" }));
} catch {
res.writeHead(503, { "Content-Type": "application/json" });
res.end(JSON.stringify({ status: "error", db: "unreachable" }));
@@ -28,7 +28,7 @@ const server = createServer(async (req, res) => {
return;
}
// All /auth/* routes handled by Better-Auth
// All other routes handled by Better-Auth (returns 404 for unknown paths)
await handler(req, res);
});
+244
View File
@@ -0,0 +1,244 @@
# UAT Receipt Submission Path
**Issue:** [CAR-812](/CAR/issues/CAR-812)
**Author:** Barcode Betty
**Date:** 2026-05-04
---
## Overview
The UAT environment supports receipt submission via **inbound email**. This is the only supported submission method in UAT — there is no public REST API surface for receipt ingestion.
---
## How It Works
### Architecture
```
User composes email
Email sent to <user_token>@cartsnitch.<env>.farh.net
Mailgun webhook receives the email
Email job enqueued to DragonflyDB stream: email:receipts
email-worker (ReceiptWitness) consumes the job
Worker resolves user via email_inbound_token lookup in DB
Retailer detected from email content (meijer / kroger / target)
Email parsed into Purchase + PurchaseItem records
receipt.ingested event published to Redis
MatchResult created with method=upc, confidence=1.0 for known UPCs
```
### Key Components
| Component | Location | Role |
|-----------|----------|------|
| `users.email_inbound_token` | DB (migration `001_add_email_inbound_token`) | 22-char unique token per user; used as email routing identifier |
| `email:receipts` stream | DragonflyDB | Queue holding pending email jobs |
| `email-worker` | `receiptwitness/src/receiptwitness/worker/email_worker.py` | Async worker consuming the stream |
| `BaseEmailParser` | `receiptwitness/src/receiptwitness/parsers/email/base.py` | Abstract parser; subclasses for meijer/kroger/target |
| Retailer detectors | `receiptwitness/src/receiptwitness/parsers/email/detector.py` | Sifts sender/subject to pick the right parser |
### Email Address Format
Each user is assigned a unique inbound token. The receipt submission email address is shown in **Settings → Receipt Email** on the UI:
**Address:** `receipts+<email_inbound_token>@receipts.cartsnitch.com`
To find a user's token in the UAT database (requires `kubectl` access to `cartsnitch-uat`):
```bash
kubectl exec -n cartsnitch-uat deployment/cartsnitch-api -- \\
python -c "from cartsnitch_common.database import get_sync_session; \\
from cartsnitch_common.models.user import User; \\
from sqlalchemy import select; \\
s = get_sync_session('postgresql://cartsnitch:cartsnitch@cartsnitch-pg-rw:5432/cartsnitch'); \\
u = s.execute(select(User).where(User.email=='dottie@example.com')).scalar_one(); \\
print(u.email_inbound_token)"
```
---
## Submitting a Test Receipt (Step-by-Step)
### Prerequisites
- A test user account in UAT with a known `email_inbound_token`
- A sample receipt email with a **known UPC** from the seeded `normalized_products` table
### Steps
1. **Obtain the test user's inbound token.**
Use the UAT Settings → Receipt Email page in the UI to see the full address `receipts+<token>@receipts.cartsnitch.com`, or query the DB directly (see above).
2. **Compose the email.**
Send to: the address shown in Settings → Receipt Email
Subject: anything
Body: plain-text or HTML receipt content
3. **Expected behavior after email is processed:**
- A `Receipt` row is created in `purchases`
- `PurchaseItem` rows are created with `upc` matching the seeded product UPC
- A `MatchResult` is created with `method='upc'` and `confidence=1.0`
---
## Known UPC for Dottie (from UAT seed)
> **NOTE:** `kubectl` is not available in this execution environment. The UAT seed and DB query could not be executed. The sample receipt below uses a plausible placeholder UPC. Before Dottie runs the regression:
> 1. Run `bash scripts/seed-env.sh uat` from a machine with UAT kubecontext
> 2. Query: `SELECT id, canonical_name, upc_variants->0->>'upc' AS sample_upc FROM normalized_products WHERE jsonb_array_length(upc_variants) > 0 LIMIT 1;`
> 3. Replace the placeholder values below with the real captured row
- `id`: **TBD — run seed and query UAT DB**
- `name`: **TBD — run seed and query UAT DB**
- `sample UPC`: **TBD — run seed and query UAT DB**
### Meijer Sample Receipt (plain text)
```
Meijer
===================================
Purchase Date: 03/15/2026
Store: Meijer #127 - Ann Arbor, MI
-----------------------------------
1 x Organic Whole Milk 1gal $4.99
1 x Whole Wheat Bread $3.29
1 x Bananas (2 lb) $0.67
1 x Chicken Breast (3 lb) $12.47
1 x Cheddar Cheese Block 8oz $5.99
-----------------------------------
Subtotal: $27.41
Tax: $1.93
Total: $29.34
===================================
THANK YOU FOR SHOPPING MEIJER
===================================
```
Meijer
===================================
Purchase Date: 03/15/2026
Store: Meijer #127 - Ann Arbor, MI
-----------------------------------
1 x Organic Whole Milk 1gal $4.99
1 x Whole Wheat Bread $3.29
1 x Bananas (2 lb) $0.67
1 x Chicken Breast (3 lb) $12.47
1 x Cheddar Cheese Block 8oz $5.99
-----------------------------------
Subtotal: $27.41
Tax: $1.93
Total: $29.34
===================================
THANK YOU FOR SHOPPING MEIJER
===================================
```
> **Note:** The `email-worker` parses the email body and extracts line items by retailer. The exact format and field mapping depends on the retailer parser. For Meijer, the parser looks for item lines matching `(\d+) x (.+?)\s+\$([\d.]+)`. UPCs in the `upc_variants` JSONB of seeded products will be matched during the normalization step.
### Kroger Sample Receipt (plain text)
```
KROGER
===================================
Purchase Date: 03/15/2026
Store: KROGER #412 - Ann Arbor MI
-----------------------------------
1 Organic Whole Milk 1gal $5.29
1 Whole Wheat Bread $3.49
1 Bananas (2 lb) $0.69
1 Chicken Breast (3 lb) $11.99
1 Sharp Cheddar Cheese 8oz $4.99
-----------------------------------
Subtotal: $26.45
Tax: $1.85
Total: $28.30
===================================
```
### Target Sample Receipt (plain text)
```
TARGET
===================================
03/15/2026 14:32
Store: 0874 Ann Arbor, MI
===================================
1 Organic Whole Milk 1G $5.49
1 Whole Wheat Bread $3.29
1 Bananas LB 2 $0.68
1 Chicken Breast 3# $12.99
1 Cheddar Cheese 8OZ $5.79
-----------------------------------
Subtotal: $28.24
Tax (6%): $1.69
Total: $29.93
===================================
```
---
## Troubleshooting
### Email not processed
1. Check the `email:receipts` stream has messages:
```bash
kubectl exec -n cartsnitch-uat deploy/email-worker -- python -c \\
"import asyncio; from receiptwitness.queue.email import get_redis; \\
async def chk(): c = await get_redis(); info = await c.xinfo_stream('email:receipts'); print(info); \\
asyncio.run(chk())"
```
2. Check `email-worker` logs for retailer detection failures:
```bash
kubectl logs -n cartsnitch-uat deploy/email-worker -f
```
3. Verify the token resolves to a user in the DB:
```bash
kubectl exec -n cartsnitch-uat deploy/cartsnitch-api -- \\
python -c "from cartsnitch_common.database import get_sync_session; \\
from cartsnitch_common.models.user import User; \\
from sqlalchemy import select; \\
s = get_sync_session('postgresql://...'); \\
r = s.execute(select(User.email_inbound_token).limit(5)).all(); \\
print(r)"
```
### No MatchResult created
The normalization pipeline requires a `normalized_product` row with the submitted UPC in `upc_variants`. If the seed was run, the product should be found. Check the `match_results` table after submission:
```sql
SELECT mr.*, np.canonical_name
FROM match_results mr
JOIN normalized_products np ON np.id = mr.normalized_product_id
WHERE mr.match_method = 'upc'
ORDER BY mr.created_at DESC
LIMIT 10;
```
---
## Related Files
| File | Role |
|------|------|
| `common/alembic/versions/001_add_email_inbound_token.py` | Adds `email_inbound_token` column |
| `receiptwitness/src/receiptwitness/worker/email_worker.py` | Consumes email jobs from stream |
| `receiptwitness/src/receiptwitness/queue/email.py` | DragonflyDB stream consumer group |
| `receiptwitness/src/receiptwitness/parsers/email/detector.py` | Retailer detection |
| `receiptwitness/src/receiptwitness/parsers/email/meijer.py` | Meijer email parser |
| `receiptwitness/src/receiptwitness/parsers/email/kroger.py` | Kroger email parser |
| `receiptwitness/src/receiptwitness/parsers/email/target.py` | Target email parser |
| `docs/uat-runbook.md` | UAT runbook (defect classification, entry/exit criteria) |
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# =============================================================================
# apply-seed-job.sh — Apply the seed Job manifest for a given environment.
#
# Usage:
# ./apply-seed-job.sh <env>
#
# Example:
# ./apply-seed-job.sh uat
# ./apply-seed-job.sh dev
# =============================================================================
set -euo pipefail
ENV="${1:-}"
HELP_FLAG=""
while [[ $# -gt 0 ]]; do
case "$1" in
--help) HELP_FLAG="1"; shift ;;
*) ENV="$1"; shift ;;
esac
done
if [[ -n "$HELP_FLAG" ]] || [[ -z "$ENV" ]]; then
echo "Usage: $0 <env>"
echo " env dev or uat"
exit 0
fi
if [[ "$ENV" != "dev" && "$ENV" != "uat" ]]; then
echo "ERROR: Invalid environment: $ENV (must be 'dev' or 'uat')" >&2
exit 1
fi
SCRIPT_DIR="$(dirname "$0")"
sed "s/__ENV__/${ENV}/g" "${SCRIPT_DIR}/seed-env-job.yaml" | kubectl apply -f -
echo "Seed job applied for environment: $ENV"
+1 -1
View File
@@ -58,4 +58,4 @@ spec:
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
memory: 512Mi
+2 -103
View File
@@ -1,104 +1,3 @@
#!/usr/bin/env bash
# =============================================================================
# seed-dev.sh — Run the CartSnitch seed runner against the dev database.
#
# Usage:
# ./seed-dev.sh Run full seed against dev
# ./seed-dev.sh --dry-run Show planned record counts without writing
# ./seed-dev.sh --help Show this help
#
# Prerequisites:
# - kubectl configured for the cartsnitch-dev cluster
# - Namespace cartsnitch-dev exists (CNPG Postgres must be running)
#
# What it does:
# 1. Starts a background port-forward to cartsnitch-pg-rw:5432
# 2. Waits for the tunnel to be ready
# 3. Runs python -m cartsnitch_common.seed with --database-url pointing
# to localhost:<forwarded-port>/cartsnitch
# 4. Cleans up the port-forward on exit (normal, interrupt, or error)
# =============================================================================
set -euo pipefail
# --- Config -------------------------------------------------------------------
readonly NAMESPACE="cartsnitch-dev"
readonly SVC_NAME="cartsnitch-pg-rw"
readonly LOCAL_PORT="5433" # use a non-privileged port to avoid conflicts
readonly DB_NAME="cartsnitch"
readonly PG_USER="cartsnitch"
# Retrieve password from the CNPG credentials secret
readonly PG_PASSWORD="$(
kubectl get secret cartsnitch-pg-credentials \
-n "$NAMESPACE" \
-o jsonpath='{.data.password}' \
| base64 -d
)"
readonly DB_URL="postgresql://${PG_USER}:${PG_PASSWORD}@localhost:${LOCAL_PORT}/${DB_NAME}"
# --- Helpers ------------------------------------------------------------------
log() { echo "[seed-dev] $*"; }
fail() { log "ERROR: $*" >&2; exit 1; }
# Cleanup port-forward and exit.
cleanup() {
if [[ -n "${PF_PID:-}" ]]; then
log "Stopping port-forward (PID $PF_PID)..."
kill "$PF_PID" 2>/dev/null || true
wait "$PF_PID" 2>/dev/null || true
fi
}
trap cleanup EXIT
# --- Args ---------------------------------------------------------------------
DRY_RUN=""
HELP_FLAG=""
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) DRY_RUN="--dry-run"; shift ;;
--help) HELP_FLAG="1"; shift ;;
*) fail "Unknown argument: $1";;
esac
done
if [[ -n "$HELP_FLAG" ]]; then
sed -n '3,/^# ---/p' "$0" | head -n -1 | sed 's/^# //'
echo ""
echo "Additional arguments are passed through to the seed runner."
echo "Common seed-runner options:"
echo " --dry-run Show planned record counts without writing"
echo " --seed N Set random seed (default: 42)"
exit 0
fi
# --- Prerequisites ------------------------------------------------------------
if ! command -v kubectl &>/dev/null; then
fail "kubectl not found — must be installed and configured."
fi
# --- Port-forward -------------------------------------------------------------
log "Starting port-forward ${SVC_NAME}:5432 -> localhost:${LOCAL_PORT} ..."
kubectl port-forward \
-n "$NAMESPACE" \
svc/"$SVC_NAME" \
"${LOCAL_PORT}:5432" \
&>/dev/null &
PF_PID=$!
# Give the tunnel a moment to establish
sleep 2
# Verify the tunnel is up
if ! kill -0 "$PF_PID" 2>/dev/null; then
fail "Port-forward failed to start."
fi
log "Port-forward active (PID $PF_PID) on localhost:${LOCAL_PORT}"
# --- Seed --------------------------------------------------------------------
log "Running seed against dev database..."
set -x
python -m cartsnitch_common.seed --database-url "$DB_URL" $DRY_RUN
set +x
log "Done."
# Backward-compat wrapper — delegates to seed-env.sh dev
exec "$(dirname "$0")/seed-env.sh" dev "$@"
+58
View File
@@ -0,0 +1,58 @@
# seed-env-job.yaml
# K8s Job to run the CartSnitch seed runner against any CartSnitch database.
#
# Usage (via apply-seed-job.sh):
# bash scripts/apply-seed-job.sh dev
# bash scripts/apply-seed-job.sh uat
#
# To view logs:
# kubectl logs -n cartsnitch-<env> job/seed-env -f
#
# To re-run after fixing issues:
# kubectl delete -f - -n cartsnitch-<env> && bash scripts/apply-seed-job.sh <env>
#
apiVersion: batch/v1
kind: Job
metadata:
name: seed-env
namespace: cartsnitch-__ENV__
labels:
app: cartsnitch
component: seed
environment: __ENV__
annotations:
description: "Runs cartsnitch-common seed runner to populate __ENV__ database with realistic test data."
spec:
backoffLimit: 0
concurrencyPolicy: Forbid
template:
metadata:
labels:
app: cartsnitch
component: seed
environment: __ENV__
spec:
restartPolicy: Never
containers:
- name: seed
image: python:3.12-slim
command:
- sh
- -c
- |
pip install --no-cache-dir "cartsnitch-common @ git+https://github.com/cartsnitch/common.git@main" && \
python -m cartsnitch_common.seed --database-url "$${DATABASE_URL}"
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: cartsnitch-secrets
key: database-url-pg
optional: false
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env bash
# =============================================================================
# seed-env.sh — Run the CartSnitch seed runner against any CartSnitch database.
#
# Usage:
# ./seed-env.sh [--env dev|uat] [--dry-run] [--help]
# ./seed-env.sh uat --dry-run Run dry-run against UAT
# ./seed-env.sh dev Run full seed against dev (default)
#
# Prerequisites:
# - kubectl configured for the target cluster
# - Namespace cartsnitch-<env> exists (CNPG Postgres must be running)
#
# What it does:
# 1. Starts a background port-forward to cartsnitch-pg-rw:5432
# 2. Waits for the tunnel to be ready
# 3. Runs python -m cartsnitch_common.seed with --database-url pointing
# to localhost:<forwarded-port>/cartsnitch
# 4. Cleans up the port-forward on exit (normal, interrupt, or error)
# =============================================================================
set -euo pipefail
# --- Config -------------------------------------------------------------------
ENV="dev"
if [[ "${1:-}" == "dev" || "${1:-}" == "uat" ]]; then
ENV="$1"; shift
fi
while [[ $# -gt 0 ]]; do
case "$1" in
--env) ENV="$2"; shift 2 ;;
--dry-run|--help) break ;;
*) break ;;
esac
done
NAMESPACE="cartsnitch-${ENV}"
SVC_NAME="cartsnitch-pg-rw"
LOCAL_PORT="5433"
DB_NAME="cartsnitch"
PG_USER="cartsnitch"
PG_PASSWORD="$(
kubectl get secret cartsnitch-pg-credentials \
-n "$NAMESPACE" \
-o jsonpath='{.data.password}' \
| base64 -d
)"
DB_URL="postgresql://${PG_USER}:${PG_PASSWORD}@localhost:${LOCAL_PORT}/${DB_NAME}"
# --- Helpers ------------------------------------------------------------------
log() { echo "[seed-env] [$ENV] $*"; }
fail() { log "ERROR: $*" >&2; exit 1; }
cleanup() {
if [[ -n "${PF_PID:-}" ]]; then
log "Stopping port-forward (PID $PF_PID)..."
kill "$PF_PID" 2>/dev/null || true
wait "$PF_PID" 2>/dev/null || true
fi
}
trap cleanup EXIT
# --- Args ---------------------------------------------------------------------
DRY_RUN=""
HELP_FLAG=""
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) DRY_RUN="--dry-run"; shift ;;
--help) HELP_FLAG="1"; shift ;;
*) fail "Unknown argument: $1";;
esac
done
if [[ -n "$HELP_FLAG" ]]; then
echo "Usage: $0 [--env dev|uat] [--dry-run] [--help]"
echo ""
echo "Positional / keyword arguments:"
echo " --env dev|uat Target environment (default: dev)"
echo " --dry-run Show planned record counts without writing"
echo " --help Show this help"
echo ""
echo "Additional arguments are passed through to the seed runner."
echo "Common seed-runner options:"
echo " --seed N Set random seed (default: 42)"
exit 0
fi
# --- Validate env --------------------------------------------------------------
if [[ "$ENV" != "dev" && "$ENV" != "uat" ]]; then
fail "Invalid environment: $ENV (must be 'dev' or 'uat')"
fi
# --- Prerequisites ------------------------------------------------------------
if ! command -v kubectl &>/dev/null; then
fail "kubectl not found — must be installed and configured."
fi
# --- Port-forward -------------------------------------------------------------
log "Starting port-forward ${SVC_NAME}:5432 -> localhost:${LOCAL_PORT} ..."
kubectl port-forward \
-n "$NAMESPACE" \
svc/"$SVC_NAME" \
"${LOCAL_PORT}:5432" \
&>/dev/null &
PF_PID=$!
sleep 2
if ! kill -0 "$PF_PID" 2>/dev/null; then
fail "Port-forward failed to start."
fi
log "Port-forward active (PID $PF_PID) on localhost:${LOCAL_PORT}"
# --- Seed --------------------------------------------------------------------
log "Running seed against ${ENV} database..."
set -x
python -m cartsnitch_common.seed --database-url "$DB_URL" $DRY_RUN
set +x
log "Done."
+1 -1
View File
@@ -7,6 +7,6 @@ export default defineConfig({
environment: 'jsdom',
globals: true,
setupFiles: ['./src/test/setup.ts'],
exclude: ['e2e/**', 'node_modules/**'],
exclude: ['e2e/**', 'auth/**', 'node_modules/**'],
},
})