forked from cartsnitch/cartsnitch
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Chris Farhood
|
752d7ed3d0 |
@@ -1,315 +1 @@
|
||||
# CartSnitch
|
||||
|
||||
**Grocery price intelligence — know what you're paying, every time.**
|
||||
|
||||
CartSnitch is a self-hosted grocery price intelligence platform that connects to your store loyalty accounts, tracks prices across retailers, monitors shrinkflation, and helps you find the best deals.
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
CartSnitch solves the problem of **grocery price opacity**. Most shoppers don't know if they're getting a good deal, whether prices have spiked since their last visit, or if the "sale" is actually a worse price than a competitor. CartSnitch makes prices transparent.
|
||||
|
||||
**Core features:**
|
||||
- Connect Meijer, Kroger, Target loyalty accounts
|
||||
- View purchase history across all stores in one timeline
|
||||
- Track per-item price charts across stores over time
|
||||
- Receive shrinkflation and price increase alerts
|
||||
- Browse active coupons and deals
|
||||
- Generate optimized shopping lists with store-split plans
|
||||
- Public price transparency dashboards
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
CartSnitch is a polyglot microservices platform. The monorepo contains the frontend PWA and core services.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ CartSnitch PWA │
|
||||
│ (React, mobile-first PWA) │
|
||||
└──────────┬────────────────────┬────────────────────┬───────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────┐ ┌─────────────────┐ ┌─────────────────────┐
|
||||
│ Auth Service │ │ API Gateway │ │ ReceiptWitness │
|
||||
│ (Better-Auth) │ │ (Python/FastAPI)│ │ (Python/Scrapers) │
|
||||
│ Session mgmt │ │ REST + proxy │ │ Purchase ingestion │
|
||||
└────────┬─────────┘ └────────┬────────┘ └──────────┬──────────┘
|
||||
│ │ │
|
||||
└──────────────────────┼────────────────────────┘
|
||||
▼
|
||||
┌────────────────────────┐
|
||||
│ CloudNativePG (PGSQL) │
|
||||
│ Shared database │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
### Services in This Repo
|
||||
|
||||
| Directory | Service | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `/` (root) | Frontend | React PWA, mobile-first |
|
||||
| `auth/` | Auth | Better-Auth service — session management, email/password, OAuth |
|
||||
| `api/` | API Gateway | Frontend-facing REST API, Python/FastAPI |
|
||||
| `common/` | Common | Shared Python models, Pydantic schemas, Alembic migrations |
|
||||
| `receiptwitness/` | ReceiptWitness | Purchase data ingestion via retailer scrapers |
|
||||
|
||||
### Other CartSnitch Repos
|
||||
|
||||
| Repo | Service |
|
||||
|------|---------|
|
||||
| `cartsnitch/stickershock` | Price increase detection & CPI comparison |
|
||||
| `cartsnitch/shrinkray` | Shrinkflation monitoring |
|
||||
| `cartsnitch/clipartist` | Coupon/deal watching |
|
||||
| `cartsnitch/infra` | Kubernetes manifests, Flux kustomizations |
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
### Frontend
|
||||
- **React 18+** with TypeScript
|
||||
- **Vite** — build tool
|
||||
- **Tailwind CSS v4** — mobile-first responsive design
|
||||
- **Workbox** — service worker, offline caching, PWA manifest
|
||||
- **Recharts** — price trend visualizations
|
||||
- **TanStack Query** — data fetching and caching
|
||||
- **React Router v7** — client-side routing
|
||||
- **Zustand** — lightweight state management
|
||||
|
||||
### Backend Services
|
||||
- **Better-Auth** — authentication (session management, email/password, OAuth)
|
||||
- **Node.js** (API Gateway)
|
||||
- **Python/FastAPI** (API Gateway, ReceiptWitness)
|
||||
- **PostgreSQL** via CloudNativePG
|
||||
- **DragonflyDB** for caching
|
||||
|
||||
### Infrastructure
|
||||
- **Kubernetes** (k3s-compatible)
|
||||
- **Flux CD** — GitOps deployment
|
||||
- **GitHub Actions** — CI/CD
|
||||
- **CalVer** (`YYYY.MM.DD[.N]`) — image tagging
|
||||
- **Bitnami Sealed Secrets** — secret management
|
||||
- **Authentik** — OIDC/OAuth2 provider
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js 20+
|
||||
- npm or pnpm
|
||||
- PostgreSQL (local or containerized)
|
||||
- Docker (for running services locally)
|
||||
|
||||
### Local Development
|
||||
|
||||
1. **Clone the repo**
|
||||
```bash
|
||||
git clone https://github.com/cartsnitch/cartsnitch.git
|
||||
cd cartsnitch
|
||||
```
|
||||
|
||||
2. **Install dependencies**
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
3. **Set up environment variables**
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your local settings
|
||||
```
|
||||
|
||||
4. **Start the frontend dev server**
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
The PWA will be available at `http://localhost:5173`.
|
||||
|
||||
5. **Run tests**
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
6. **Build for production**
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Running Backend Services Locally
|
||||
|
||||
The frontend PWA communicates with three backend services. For full local development, you'll need to run each service:
|
||||
|
||||
```bash
|
||||
# Auth service (Better-Auth)
|
||||
cd auth
|
||||
npm install
|
||||
npm run dev
|
||||
|
||||
# API Gateway (separate repo: cartsnitch/api)
|
||||
# See api/README.md
|
||||
|
||||
# ReceiptWitness (separate repo: cartsnitch/receiptwitness)
|
||||
# See receiptwitness/README.md
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `VITE_API_URL` | API Gateway base URL | `http://localhost:3000` |
|
||||
| `VITE_AUTH_URL` | Auth service base URL | `http://localhost:3001` |
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions. Please follow the workflow below.
|
||||
|
||||
### Branching Strategy
|
||||
|
||||
- Branch from `dev`
|
||||
- Use prefix: `feature/`, `fix/`, `docs/`, `chore/`
|
||||
- Examples: `feature/shopping-list-optimization`, `fix/price-chart-zoom`
|
||||
|
||||
### Commit Convention
|
||||
|
||||
We use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat: add shopping list export
|
||||
fix: correct price chart date formatting
|
||||
docs: update API documentation
|
||||
chore: update dependencies
|
||||
```
|
||||
|
||||
### Pull Request Workflow
|
||||
|
||||
1. Open a PR against `dev`
|
||||
2. CI must pass (lint, type check, tests, e2e)
|
||||
3. QA reviews and approves
|
||||
4. CTO merges to `dev`
|
||||
5. Dev deploys automatically
|
||||
6. CTO promotes `dev → uat`
|
||||
7. UAT and security review
|
||||
8. CEO merges `uat → main`
|
||||
9. Production deploys automatically
|
||||
|
||||
**Never push directly to `main`, `dev`, or `uat`.**
|
||||
|
||||
### Code Standards
|
||||
|
||||
- ESLint for linting
|
||||
- TypeScript strict mode
|
||||
- Mobile-first responsive design
|
||||
- Accessibility (WCAG 2.1 AA)
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
### E2E Tests (Playwright)
|
||||
|
||||
```bash
|
||||
npm run test:e2e
|
||||
```
|
||||
|
||||
Tests run headless by default. For headed mode:
|
||||
|
||||
```bash
|
||||
npm run test:e2e:headed
|
||||
```
|
||||
|
||||
### Lighthouse CI
|
||||
|
||||
Performance audits run automatically in CI. To run locally:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
npm run preview
|
||||
# In another terminal:
|
||||
npx lighthouse http://localhost:4173 --output=html --output-path=./report/lighthouse.html
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
All branches (`main`, `dev`, `uat`) run through GitHub Actions on every push.
|
||||
|
||||
### Pipeline Stages
|
||||
|
||||
| Job | Trigger | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `lint` | Every push | ESLint + TypeScript type check |
|
||||
| `test` | Every push | Unit tests via Vitest |
|
||||
| `audit` | Every push | Security vulnerability scan |
|
||||
| `e2e` | Every push | Playwright end-to-end tests |
|
||||
| `lighthouse` | After test | Performance budget check |
|
||||
| `build-and-push` | On push to main/dev/uat | Build and push Docker images to GHCR |
|
||||
| `deploy-dev` | On push to dev or main | Update `cartsnitch/infra` → auto-deploy to dev |
|
||||
| `deploy-uat` | On push to uat or main | Update `cartsnitch/infra` → auto-deploy to uat |
|
||||
|
||||
### Image Tagging
|
||||
|
||||
- **Production (`main`):** CalVer tag (`YYYY.MM.DD[.N]`) + `latest`
|
||||
- **Development (`dev`):** SHA tag (`sha-<short-sha>`)
|
||||
|
||||
### Deployment Environments
|
||||
|
||||
| Environment | Namespace | URL | Trigger |
|
||||
|-------------|-----------|-----|---------|
|
||||
| Dev | `cartsnitch-dev` | `cartsnitch.dev.farh.net` | Push to `dev` branch |
|
||||
| UAT | `cartsnitch-uat` | `cartsnitch.uat.farh.net` | Push to `uat` branch |
|
||||
| Production | `cartsnitch` | `cartsnitch.farh.net` | Push to `main` branch |
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Infrastructure
|
||||
|
||||
The infrastructure repository ([cartsnitch/infra](https://github.com/cartsnitch/infra)) contains Kubernetes manifests and Flux Kustomize overlays.
|
||||
|
||||
### Flux GitOps Flow
|
||||
|
||||
1. CI builds and pushes a new Docker image
|
||||
2. CI opens a PR to `cartsnitch/infra` updating the image tag
|
||||
3. On merge, Flux reconciles the manifests and rolls out the new image
|
||||
|
||||
### Forcing a Rollout
|
||||
|
||||
To force pods to pick up a new `:latest` image:
|
||||
|
||||
```bash
|
||||
kubectl rollout restart deployment/<name> -n <namespace>
|
||||
```
|
||||
|
||||
### Secrets
|
||||
|
||||
Secrets are managed via **Bitnami Sealed Secrets**. No plain Kubernetes secrets are used.
|
||||
|
||||
---
|
||||
|
||||
## Related Projects
|
||||
|
||||
- [StickerShock](https://github.com/cartsnitch/stickershock) — Price increase detection
|
||||
- [ShrinkRay](https://github.com/cartsnitch/shrinkray) — Shrinkflation monitoring
|
||||
- [ClipArtist](https://github.com/cartsnitch/clipartist) — Coupon/deal optimization
|
||||
- [Infra](https://github.com/cartsnitch/infra) — Kubernetes infrastructure
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT © 2025 CartSnitch
|
||||
|
||||
+1
-2
@@ -7,8 +7,7 @@
|
||||
"dev": "tsx watch src/index.ts",
|
||||
"build": "tsc",
|
||||
"start": "node dist/index.js",
|
||||
"generate": "npx @better-auth/cli generate",
|
||||
"test": "node --test src/__tests__/*.test.ts"
|
||||
"generate": "npx @better-auth/cli generate"
|
||||
},
|
||||
"dependencies": {
|
||||
"bcrypt": "^6.0.0",
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
import { describe, it } from 'node:test';
|
||||
import { equal } from 'node:assert';
|
||||
import http from 'node:http';
|
||||
|
||||
describe('Auth health endpoint', () => {
|
||||
const startHealthServer = (poolMock) => {
|
||||
return new Promise((resolve) => {
|
||||
const server = http.createServer(async (req, res) => {
|
||||
if (req.url === '/health' && req.method === 'GET') {
|
||||
try {
|
||||
const client = await poolMock.connect();
|
||||
try {
|
||||
await Promise.race([
|
||||
client.query('SELECT 1'),
|
||||
new Promise((_, reject) => setTimeout(() => reject(new Error('DB timeout')), 2000)),
|
||||
]);
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ status: 'ok', db: 'reachable' }));
|
||||
} catch {
|
||||
res.writeHead(503, { 'Content-Type': 'application/json' });
|
||||
res.end(JSON.stringify({ status: 'error', db: 'unreachable' }));
|
||||
}
|
||||
return;
|
||||
}
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
});
|
||||
server.listen(0, '0.0.0.0', () => {
|
||||
const addr = server.address();
|
||||
const port = typeof addr === 'object' && addr ? addr.port : 0;
|
||||
resolve({ port, close: () => server.close() });
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
const makeRequest = (port) => {
|
||||
return new Promise((resolve) => {
|
||||
const req = http.get(`http://localhost:${port}/health`, (res) => {
|
||||
let body = '';
|
||||
res.on('data', (chunk) => { body += chunk; });
|
||||
res.on('end', () => {
|
||||
resolve({ status: res.statusCode, body });
|
||||
});
|
||||
});
|
||||
req.on('error', () => resolve({ status: 0, body: '' }));
|
||||
});
|
||||
};
|
||||
|
||||
it('returns 200 with db=reachable when pool.connect succeeds', async () => {
|
||||
const mockClient = {
|
||||
query: async () => ({ rows: [{ 1: 1 }] }),
|
||||
release: () => {},
|
||||
};
|
||||
const poolMock = {
|
||||
connect: async () => mockClient,
|
||||
};
|
||||
|
||||
const { port, close } = await startHealthServer(poolMock);
|
||||
const { status, body } = await makeRequest(port);
|
||||
close();
|
||||
|
||||
equal(status, 200);
|
||||
equal(body, '{"status":"ok","db":"reachable"}');
|
||||
});
|
||||
|
||||
it('returns 503 with db=unreachable when pool.connect throws', async () => {
|
||||
const poolMock = {
|
||||
connect: async () => { throw new Error('connection refused'); },
|
||||
};
|
||||
|
||||
const { port, close } = await startHealthServer(poolMock);
|
||||
const { status, body } = await makeRequest(port);
|
||||
close();
|
||||
|
||||
equal(status, 503);
|
||||
equal(body, '{"status":"error","db":"unreachable"}');
|
||||
});
|
||||
|
||||
it('returns 503 with db=unreachable when query times out', async () => {
|
||||
const mockClient = {
|
||||
query: async () => {
|
||||
await new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), 3000));
|
||||
},
|
||||
release: () => {},
|
||||
};
|
||||
const poolMock = {
|
||||
connect: async () => mockClient,
|
||||
};
|
||||
|
||||
const { port, close } = await startHealthServer(poolMock);
|
||||
const { status, body } = await makeRequest(port);
|
||||
close();
|
||||
|
||||
equal(status, 503);
|
||||
equal(body, '{"status":"error","db":"unreachable"}');
|
||||
});
|
||||
|
||||
it('returns a terminal response for unknown paths (no hang)', async () => {
|
||||
const poolMock = { connect: async () => ({ query: async () => {}, release: () => {} }) };
|
||||
const { port, close } = await startHealthServer(poolMock);
|
||||
|
||||
const result = await new Promise<{ status: number }>((resolve) => {
|
||||
const req = http.get(`http://localhost:${port}/`, (res) => {
|
||||
res.resume();
|
||||
res.on('end', () => resolve({ status: res.statusCode ?? 0 }));
|
||||
});
|
||||
req.on('error', () => resolve({ status: 0 }));
|
||||
setTimeout(() => resolve({ status: -1 }), 1000);
|
||||
});
|
||||
close();
|
||||
|
||||
equal(result.status !== -1, true, 'Unknown path must return a terminal response within 1s');
|
||||
});
|
||||
});
|
||||
+3
-3
@@ -8,7 +8,7 @@ const handler = toNodeHandler(auth);
|
||||
|
||||
const server = createServer(async (req, res) => {
|
||||
// Health check
|
||||
if ((req.url === "/health" || req.url === "/auth/health") && req.method === "GET") {
|
||||
if (req.url === "/health" && req.method === "GET") {
|
||||
try {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
@@ -20,7 +20,7 @@ const server = createServer(async (req, res) => {
|
||||
client.release();
|
||||
}
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "ok", db: "reachable" }));
|
||||
res.end(JSON.stringify({ status: "ok", db: "connected" }));
|
||||
} catch {
|
||||
res.writeHead(503, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "error", db: "unreachable" }));
|
||||
@@ -28,7 +28,7 @@ const server = createServer(async (req, res) => {
|
||||
return;
|
||||
}
|
||||
|
||||
// All other routes handled by Better-Auth (returns 404 for unknown paths)
|
||||
// All /auth/* routes handled by Better-Auth
|
||||
await handler(req, res);
|
||||
});
|
||||
|
||||
|
||||
+1
-1
@@ -12,5 +12,5 @@
|
||||
"resolveJsonModule": true
|
||||
},
|
||||
"include": ["src"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
"exclude": ["node_modules", "dist", "src/__tests__"]
|
||||
}
|
||||
|
||||
@@ -1,244 +0,0 @@
|
||||
# UAT Receipt Submission Path
|
||||
|
||||
**Issue:** [CAR-812](/CAR/issues/CAR-812)
|
||||
**Author:** Barcode Betty
|
||||
**Date:** 2026-05-04
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The UAT environment supports receipt submission via **inbound email**. This is the only supported submission method in UAT — there is no public REST API surface for receipt ingestion.
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User composes email
|
||||
↓
|
||||
Email sent to <user_token>@cartsnitch.<env>.farh.net
|
||||
↓
|
||||
Mailgun webhook receives the email
|
||||
↓
|
||||
Email job enqueued to DragonflyDB stream: email:receipts
|
||||
↓
|
||||
email-worker (ReceiptWitness) consumes the job
|
||||
↓
|
||||
Worker resolves user via email_inbound_token lookup in DB
|
||||
↓
|
||||
Retailer detected from email content (meijer / kroger / target)
|
||||
↓
|
||||
Email parsed into Purchase + PurchaseItem records
|
||||
↓
|
||||
receipt.ingested event published to Redis
|
||||
↓
|
||||
MatchResult created with method=upc, confidence=1.0 for known UPCs
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
| Component | Location | Role |
|
||||
|-----------|----------|------|
|
||||
| `users.email_inbound_token` | DB (migration `001_add_email_inbound_token`) | 22-char unique token per user; used as email routing identifier |
|
||||
| `email:receipts` stream | DragonflyDB | Queue holding pending email jobs |
|
||||
| `email-worker` | `receiptwitness/src/receiptwitness/worker/email_worker.py` | Async worker consuming the stream |
|
||||
| `BaseEmailParser` | `receiptwitness/src/receiptwitness/parsers/email/base.py` | Abstract parser; subclasses for meijer/kroger/target |
|
||||
| Retailer detectors | `receiptwitness/src/receiptwitness/parsers/email/detector.py` | Sifts sender/subject to pick the right parser |
|
||||
|
||||
### Email Address Format
|
||||
|
||||
Each user is assigned a unique inbound token. The receipt submission email address is shown in **Settings → Receipt Email** on the UI:
|
||||
|
||||
**Address:** `receipts+<email_inbound_token>@receipts.cartsnitch.com`
|
||||
|
||||
To find a user's token in the UAT database (requires `kubectl` access to `cartsnitch-uat`):
|
||||
|
||||
```bash
|
||||
kubectl exec -n cartsnitch-uat deployment/cartsnitch-api -- \\
|
||||
python -c "from cartsnitch_common.database import get_sync_session; \\
|
||||
from cartsnitch_common.models.user import User; \\
|
||||
from sqlalchemy import select; \\
|
||||
s = get_sync_session('postgresql://cartsnitch:cartsnitch@cartsnitch-pg-rw:5432/cartsnitch'); \\
|
||||
u = s.execute(select(User).where(User.email=='dottie@example.com')).scalar_one(); \\
|
||||
print(u.email_inbound_token)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Submitting a Test Receipt (Step-by-Step)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A test user account in UAT with a known `email_inbound_token`
|
||||
- A sample receipt email with a **known UPC** from the seeded `normalized_products` table
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Obtain the test user's inbound token.**
|
||||
Use the UAT Settings → Receipt Email page in the UI to see the full address `receipts+<token>@receipts.cartsnitch.com`, or query the DB directly (see above).
|
||||
|
||||
2. **Compose the email.**
|
||||
Send to: the address shown in Settings → Receipt Email
|
||||
Subject: anything
|
||||
Body: plain-text or HTML receipt content
|
||||
|
||||
3. **Expected behavior after email is processed:**
|
||||
- A `Receipt` row is created in `purchases`
|
||||
- `PurchaseItem` rows are created with `upc` matching the seeded product UPC
|
||||
- A `MatchResult` is created with `method='upc'` and `confidence=1.0`
|
||||
|
||||
---
|
||||
|
||||
## Known UPC for Dottie (from UAT seed)
|
||||
|
||||
> **NOTE:** `kubectl` is not available in this execution environment. The UAT seed and DB query could not be executed. The sample receipt below uses a plausible placeholder UPC. Before Dottie runs the regression:
|
||||
> 1. Run `bash scripts/seed-env.sh uat` from a machine with UAT kubecontext
|
||||
> 2. Query: `SELECT id, canonical_name, upc_variants->0->>'upc' AS sample_upc FROM normalized_products WHERE jsonb_array_length(upc_variants) > 0 LIMIT 1;`
|
||||
> 3. Replace the placeholder values below with the real captured row
|
||||
|
||||
- `id`: **TBD — run seed and query UAT DB**
|
||||
- `name`: **TBD — run seed and query UAT DB**
|
||||
- `sample UPC`: **TBD — run seed and query UAT DB**
|
||||
|
||||
### Meijer Sample Receipt (plain text)
|
||||
|
||||
```
|
||||
Meijer
|
||||
===================================
|
||||
Purchase Date: 03/15/2026
|
||||
Store: Meijer #127 - Ann Arbor, MI
|
||||
-----------------------------------
|
||||
1 x Organic Whole Milk 1gal $4.99
|
||||
1 x Whole Wheat Bread $3.29
|
||||
1 x Bananas (2 lb) $0.67
|
||||
1 x Chicken Breast (3 lb) $12.47
|
||||
1 x Cheddar Cheese Block 8oz $5.99
|
||||
-----------------------------------
|
||||
Subtotal: $27.41
|
||||
Tax: $1.93
|
||||
Total: $29.34
|
||||
===================================
|
||||
THANK YOU FOR SHOPPING MEIJER
|
||||
===================================
|
||||
```
|
||||
Meijer
|
||||
===================================
|
||||
Purchase Date: 03/15/2026
|
||||
Store: Meijer #127 - Ann Arbor, MI
|
||||
-----------------------------------
|
||||
1 x Organic Whole Milk 1gal $4.99
|
||||
1 x Whole Wheat Bread $3.29
|
||||
1 x Bananas (2 lb) $0.67
|
||||
1 x Chicken Breast (3 lb) $12.47
|
||||
1 x Cheddar Cheese Block 8oz $5.99
|
||||
-----------------------------------
|
||||
Subtotal: $27.41
|
||||
Tax: $1.93
|
||||
Total: $29.34
|
||||
===================================
|
||||
THANK YOU FOR SHOPPING MEIJER
|
||||
===================================
|
||||
```
|
||||
|
||||
> **Note:** The `email-worker` parses the email body and extracts line items by retailer. The exact format and field mapping depends on the retailer parser. For Meijer, the parser looks for item lines matching `(\d+) x (.+?)\s+\$([\d.]+)`. UPCs in the `upc_variants` JSONB of seeded products will be matched during the normalization step.
|
||||
|
||||
### Kroger Sample Receipt (plain text)
|
||||
|
||||
```
|
||||
KROGER
|
||||
===================================
|
||||
Purchase Date: 03/15/2026
|
||||
Store: KROGER #412 - Ann Arbor MI
|
||||
-----------------------------------
|
||||
1 Organic Whole Milk 1gal $5.29
|
||||
1 Whole Wheat Bread $3.49
|
||||
1 Bananas (2 lb) $0.69
|
||||
1 Chicken Breast (3 lb) $11.99
|
||||
1 Sharp Cheddar Cheese 8oz $4.99
|
||||
-----------------------------------
|
||||
Subtotal: $26.45
|
||||
Tax: $1.85
|
||||
Total: $28.30
|
||||
===================================
|
||||
```
|
||||
|
||||
### Target Sample Receipt (plain text)
|
||||
|
||||
```
|
||||
TARGET
|
||||
===================================
|
||||
03/15/2026 14:32
|
||||
Store: 0874 Ann Arbor, MI
|
||||
===================================
|
||||
1 Organic Whole Milk 1G $5.49
|
||||
1 Whole Wheat Bread $3.29
|
||||
1 Bananas LB 2 $0.68
|
||||
1 Chicken Breast 3# $12.99
|
||||
1 Cheddar Cheese 8OZ $5.79
|
||||
-----------------------------------
|
||||
Subtotal: $28.24
|
||||
Tax (6%): $1.69
|
||||
Total: $29.93
|
||||
===================================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Email not processed
|
||||
|
||||
1. Check the `email:receipts` stream has messages:
|
||||
```bash
|
||||
kubectl exec -n cartsnitch-uat deploy/email-worker -- python -c \\
|
||||
"import asyncio; from receiptwitness.queue.email import get_redis; \\
|
||||
async def chk(): c = await get_redis(); info = await c.xinfo_stream('email:receipts'); print(info); \\
|
||||
asyncio.run(chk())"
|
||||
```
|
||||
|
||||
2. Check `email-worker` logs for retailer detection failures:
|
||||
```bash
|
||||
kubectl logs -n cartsnitch-uat deploy/email-worker -f
|
||||
```
|
||||
|
||||
3. Verify the token resolves to a user in the DB:
|
||||
```bash
|
||||
kubectl exec -n cartsnitch-uat deploy/cartsnitch-api -- \\
|
||||
python -c "from cartsnitch_common.database import get_sync_session; \\
|
||||
from cartsnitch_common.models.user import User; \\
|
||||
from sqlalchemy import select; \\
|
||||
s = get_sync_session('postgresql://...'); \\
|
||||
r = s.execute(select(User.email_inbound_token).limit(5)).all(); \\
|
||||
print(r)"
|
||||
```
|
||||
|
||||
### No MatchResult created
|
||||
|
||||
The normalization pipeline requires a `normalized_product` row with the submitted UPC in `upc_variants`. If the seed was run, the product should be found. Check the `match_results` table after submission:
|
||||
|
||||
```sql
|
||||
SELECT mr.*, np.canonical_name
|
||||
FROM match_results mr
|
||||
JOIN normalized_products np ON np.id = mr.normalized_product_id
|
||||
WHERE mr.match_method = 'upc'
|
||||
ORDER BY mr.created_at DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `common/alembic/versions/001_add_email_inbound_token.py` | Adds `email_inbound_token` column |
|
||||
| `receiptwitness/src/receiptwitness/worker/email_worker.py` | Consumes email jobs from stream |
|
||||
| `receiptwitness/src/receiptwitness/queue/email.py` | DragonflyDB stream consumer group |
|
||||
| `receiptwitness/src/receiptwitness/parsers/email/detector.py` | Retailer detection |
|
||||
| `receiptwitness/src/receiptwitness/parsers/email/meijer.py` | Meijer email parser |
|
||||
| `receiptwitness/src/receiptwitness/parsers/email/kroger.py` | Kroger email parser |
|
||||
| `receiptwitness/src/receiptwitness/parsers/email/target.py` | Target email parser |
|
||||
| `docs/uat-runbook.md` | UAT runbook (defect classification, entry/exit criteria) |
|
||||
@@ -1,38 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# apply-seed-job.sh — Apply the seed Job manifest for a given environment.
|
||||
#
|
||||
# Usage:
|
||||
# ./apply-seed-job.sh <env>
|
||||
#
|
||||
# Example:
|
||||
# ./apply-seed-job.sh uat
|
||||
# ./apply-seed-job.sh dev
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ENV="${1:-}"
|
||||
HELP_FLAG=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--help) HELP_FLAG="1"; shift ;;
|
||||
*) ENV="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -n "$HELP_FLAG" ]] || [[ -z "$ENV" ]]; then
|
||||
echo "Usage: $0 <env>"
|
||||
echo " env dev or uat"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$ENV" != "dev" && "$ENV" != "uat" ]]; then
|
||||
echo "ERROR: Invalid environment: $ENV (must be 'dev' or 'uat')" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(dirname "$0")"
|
||||
sed "s/__ENV__/${ENV}/g" "${SCRIPT_DIR}/seed-env-job.yaml" | kubectl apply -f -
|
||||
echo "Seed job applied for environment: $ENV"
|
||||
@@ -58,4 +58,4 @@ spec:
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
memory: 512Mi
|
||||
|
||||
+103
-2
@@ -1,3 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# Backward-compat wrapper — delegates to seed-env.sh dev
|
||||
exec "$(dirname "$0")/seed-env.sh" dev "$@"
|
||||
# =============================================================================
|
||||
# seed-dev.sh — Run the CartSnitch seed runner against the dev database.
|
||||
#
|
||||
# Usage:
|
||||
# ./seed-dev.sh Run full seed against dev
|
||||
# ./seed-dev.sh --dry-run Show planned record counts without writing
|
||||
# ./seed-dev.sh --help Show this help
|
||||
#
|
||||
# Prerequisites:
|
||||
# - kubectl configured for the cartsnitch-dev cluster
|
||||
# - Namespace cartsnitch-dev exists (CNPG Postgres must be running)
|
||||
#
|
||||
# What it does:
|
||||
# 1. Starts a background port-forward to cartsnitch-pg-rw:5432
|
||||
# 2. Waits for the tunnel to be ready
|
||||
# 3. Runs python -m cartsnitch_common.seed with --database-url pointing
|
||||
# to localhost:<forwarded-port>/cartsnitch
|
||||
# 4. Cleans up the port-forward on exit (normal, interrupt, or error)
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# --- Config -------------------------------------------------------------------
|
||||
readonly NAMESPACE="cartsnitch-dev"
|
||||
readonly SVC_NAME="cartsnitch-pg-rw"
|
||||
readonly LOCAL_PORT="5433" # use a non-privileged port to avoid conflicts
|
||||
readonly DB_NAME="cartsnitch"
|
||||
readonly PG_USER="cartsnitch"
|
||||
# Retrieve password from the CNPG credentials secret
|
||||
readonly PG_PASSWORD="$(
|
||||
kubectl get secret cartsnitch-pg-credentials \
|
||||
-n "$NAMESPACE" \
|
||||
-o jsonpath='{.data.password}' \
|
||||
| base64 -d
|
||||
)"
|
||||
readonly DB_URL="postgresql://${PG_USER}:${PG_PASSWORD}@localhost:${LOCAL_PORT}/${DB_NAME}"
|
||||
|
||||
# --- Helpers ------------------------------------------------------------------
|
||||
log() { echo "[seed-dev] $*"; }
|
||||
fail() { log "ERROR: $*" >&2; exit 1; }
|
||||
|
||||
# Cleanup port-forward and exit.
|
||||
cleanup() {
|
||||
if [[ -n "${PF_PID:-}" ]]; then
|
||||
log "Stopping port-forward (PID $PF_PID)..."
|
||||
kill "$PF_PID" 2>/dev/null || true
|
||||
wait "$PF_PID" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# --- Args ---------------------------------------------------------------------
|
||||
DRY_RUN=""
|
||||
HELP_FLAG=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN="--dry-run"; shift ;;
|
||||
--help) HELP_FLAG="1"; shift ;;
|
||||
*) fail "Unknown argument: $1";;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -n "$HELP_FLAG" ]]; then
|
||||
sed -n '3,/^# ---/p' "$0" | head -n -1 | sed 's/^# //'
|
||||
echo ""
|
||||
echo "Additional arguments are passed through to the seed runner."
|
||||
echo "Common seed-runner options:"
|
||||
echo " --dry-run Show planned record counts without writing"
|
||||
echo " --seed N Set random seed (default: 42)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- Prerequisites ------------------------------------------------------------
|
||||
if ! command -v kubectl &>/dev/null; then
|
||||
fail "kubectl not found — must be installed and configured."
|
||||
fi
|
||||
|
||||
# --- Port-forward -------------------------------------------------------------
|
||||
log "Starting port-forward ${SVC_NAME}:5432 -> localhost:${LOCAL_PORT} ..."
|
||||
kubectl port-forward \
|
||||
-n "$NAMESPACE" \
|
||||
svc/"$SVC_NAME" \
|
||||
"${LOCAL_PORT}:5432" \
|
||||
&>/dev/null &
|
||||
PF_PID=$!
|
||||
|
||||
# Give the tunnel a moment to establish
|
||||
sleep 2
|
||||
|
||||
# Verify the tunnel is up
|
||||
if ! kill -0 "$PF_PID" 2>/dev/null; then
|
||||
fail "Port-forward failed to start."
|
||||
fi
|
||||
log "Port-forward active (PID $PF_PID) on localhost:${LOCAL_PORT}"
|
||||
|
||||
# --- Seed --------------------------------------------------------------------
|
||||
log "Running seed against dev database..."
|
||||
set -x
|
||||
python -m cartsnitch_common.seed --database-url "$DB_URL" $DRY_RUN
|
||||
set +x
|
||||
|
||||
log "Done."
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
# seed-env-job.yaml
|
||||
# K8s Job to run the CartSnitch seed runner against any CartSnitch database.
|
||||
#
|
||||
# Usage (via apply-seed-job.sh):
|
||||
# bash scripts/apply-seed-job.sh dev
|
||||
# bash scripts/apply-seed-job.sh uat
|
||||
#
|
||||
# To view logs:
|
||||
# kubectl logs -n cartsnitch-<env> job/seed-env -f
|
||||
#
|
||||
# To re-run after fixing issues:
|
||||
# kubectl delete -f - -n cartsnitch-<env> && bash scripts/apply-seed-job.sh <env>
|
||||
#
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: seed-env
|
||||
namespace: cartsnitch-__ENV__
|
||||
labels:
|
||||
app: cartsnitch
|
||||
component: seed
|
||||
environment: __ENV__
|
||||
annotations:
|
||||
description: "Runs cartsnitch-common seed runner to populate __ENV__ database with realistic test data."
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
concurrencyPolicy: Forbid
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cartsnitch
|
||||
component: seed
|
||||
environment: __ENV__
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: seed
|
||||
image: python:3.12-slim
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
pip install --no-cache-dir "cartsnitch-common @ git+https://github.com/cartsnitch/common.git@main" && \
|
||||
python -m cartsnitch_common.seed --database-url "$${DATABASE_URL}"
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cartsnitch-secrets
|
||||
key: database-url-pg
|
||||
optional: false
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
@@ -1,122 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# seed-env.sh — Run the CartSnitch seed runner against any CartSnitch database.
|
||||
#
|
||||
# Usage:
|
||||
# ./seed-env.sh [--env dev|uat] [--dry-run] [--help]
|
||||
# ./seed-env.sh uat --dry-run Run dry-run against UAT
|
||||
# ./seed-env.sh dev Run full seed against dev (default)
|
||||
#
|
||||
# Prerequisites:
|
||||
# - kubectl configured for the target cluster
|
||||
# - Namespace cartsnitch-<env> exists (CNPG Postgres must be running)
|
||||
#
|
||||
# What it does:
|
||||
# 1. Starts a background port-forward to cartsnitch-pg-rw:5432
|
||||
# 2. Waits for the tunnel to be ready
|
||||
# 3. Runs python -m cartsnitch_common.seed with --database-url pointing
|
||||
# to localhost:<forwarded-port>/cartsnitch
|
||||
# 4. Cleans up the port-forward on exit (normal, interrupt, or error)
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# --- Config -------------------------------------------------------------------
|
||||
ENV="dev"
|
||||
if [[ "${1:-}" == "dev" || "${1:-}" == "uat" ]]; then
|
||||
ENV="$1"; shift
|
||||
fi
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--env) ENV="$2"; shift 2 ;;
|
||||
--dry-run|--help) break ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
|
||||
NAMESPACE="cartsnitch-${ENV}"
|
||||
SVC_NAME="cartsnitch-pg-rw"
|
||||
LOCAL_PORT="5433"
|
||||
DB_NAME="cartsnitch"
|
||||
PG_USER="cartsnitch"
|
||||
PG_PASSWORD="$(
|
||||
kubectl get secret cartsnitch-pg-credentials \
|
||||
-n "$NAMESPACE" \
|
||||
-o jsonpath='{.data.password}' \
|
||||
| base64 -d
|
||||
)"
|
||||
DB_URL="postgresql://${PG_USER}:${PG_PASSWORD}@localhost:${LOCAL_PORT}/${DB_NAME}"
|
||||
|
||||
# --- Helpers ------------------------------------------------------------------
|
||||
log() { echo "[seed-env] [$ENV] $*"; }
|
||||
fail() { log "ERROR: $*" >&2; exit 1; }
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "${PF_PID:-}" ]]; then
|
||||
log "Stopping port-forward (PID $PF_PID)..."
|
||||
kill "$PF_PID" 2>/dev/null || true
|
||||
wait "$PF_PID" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# --- Args ---------------------------------------------------------------------
|
||||
DRY_RUN=""
|
||||
HELP_FLAG=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN="--dry-run"; shift ;;
|
||||
--help) HELP_FLAG="1"; shift ;;
|
||||
*) fail "Unknown argument: $1";;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -n "$HELP_FLAG" ]]; then
|
||||
echo "Usage: $0 [--env dev|uat] [--dry-run] [--help]"
|
||||
echo ""
|
||||
echo "Positional / keyword arguments:"
|
||||
echo " --env dev|uat Target environment (default: dev)"
|
||||
echo " --dry-run Show planned record counts without writing"
|
||||
echo " --help Show this help"
|
||||
echo ""
|
||||
echo "Additional arguments are passed through to the seed runner."
|
||||
echo "Common seed-runner options:"
|
||||
echo " --seed N Set random seed (default: 42)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --- Validate env --------------------------------------------------------------
|
||||
if [[ "$ENV" != "dev" && "$ENV" != "uat" ]]; then
|
||||
fail "Invalid environment: $ENV (must be 'dev' or 'uat')"
|
||||
fi
|
||||
|
||||
# --- Prerequisites ------------------------------------------------------------
|
||||
if ! command -v kubectl &>/dev/null; then
|
||||
fail "kubectl not found — must be installed and configured."
|
||||
fi
|
||||
|
||||
# --- Port-forward -------------------------------------------------------------
|
||||
log "Starting port-forward ${SVC_NAME}:5432 -> localhost:${LOCAL_PORT} ..."
|
||||
kubectl port-forward \
|
||||
-n "$NAMESPACE" \
|
||||
svc/"$SVC_NAME" \
|
||||
"${LOCAL_PORT}:5432" \
|
||||
&>/dev/null &
|
||||
PF_PID=$!
|
||||
|
||||
sleep 2
|
||||
|
||||
if ! kill -0 "$PF_PID" 2>/dev/null; then
|
||||
fail "Port-forward failed to start."
|
||||
fi
|
||||
log "Port-forward active (PID $PF_PID) on localhost:${LOCAL_PORT}"
|
||||
|
||||
# --- Seed --------------------------------------------------------------------
|
||||
log "Running seed against ${ENV} database..."
|
||||
set -x
|
||||
python -m cartsnitch_common.seed --database-url "$DB_URL" $DRY_RUN
|
||||
set +x
|
||||
|
||||
log "Done."
|
||||
+1
-1
@@ -7,6 +7,6 @@ export default defineConfig({
|
||||
environment: 'jsdom',
|
||||
globals: true,
|
||||
setupFiles: ['./src/test/setup.ts'],
|
||||
exclude: ['e2e/**', 'auth/**', 'node_modules/**'],
|
||||
exclude: ['e2e/**', 'node_modules/**'],
|
||||
},
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user