Compare commits

...

27 Commits

Author SHA1 Message Date
Barcode Betty 01ed6dac00 fix(deps): pin safe versions of audit-flagged transitive deps (CAR-1162 audit)
The CI's npm audit (10.8.2) flagged three transitive vulnerabilities
that local newer-npm runs (11.x) miss due to advisory-DB divergence:

- @babel/plugin-transform-modules-systemjs: 7.29.0 -> ^7.29.4
  (CVE-2026-44728: arbitrary code generation, fixed in 7.29.4)
- fast-uri: 3.1.0 -> ^3.1.2
  (path traversal / host confusion via percent-encoded segments)
- brace-expansion: 5.0.5 -> >=5.0.6
  (DoS via large numeric range defeating max protection)

These are non-breaking transitive updates within the same major
version. The previous override for brace-expansion (>=1.1.13) was
too loose to exclude 5.0.2-5.0.5; tightening it to >=5.0.6.

Ref CAR-1162, CAR-1122, CAR-1078

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-03 15:53:46 +00:00
Barcode Betty a7a55bbf79 fix(ci): unblock dev PR #271 CI
- Remove .mcp.json (scope creep, unrelated to CAR-1078)
- Bump vitest to ^4.1.8 (fixes GHSA-5xrq-8626-4rwp critical)
- Run npm audit fix for non-breaking vulns
- Pin actions/checkout and actions/setup-node to commit SHAs
  in .gitea/workflows/ci.yml to force a clean cache fetch on
  the act runner (workaround for corrupted /root/.cache/act cache)

Refs CAR-1162, CAR-1122, CAR-1078
2026-06-03 11:41:19 +00:00
Flea Flicker fb0bb0102c fix(receiptwitness): pool DB engine and Redis client to prevent connection exhaustion
email_worker calls get_async_session_factory() inside every resolve_user()
call, which creates a brand-new async engine (and thus a brand-new
connection pool) on every message.  In a tight consumer loop processing
5 messages per batch, this rapidly exhausts DragonflyDB/Postgres
connection limits and manifests as ConnectionResetError.

Fix: cache the async engine in a module-level dict keyed by URL in
cartsnitch_common.database:get_async_engine(), matching the pattern
already used in receiptwitness:events.py for the Redis connection pool.
Also add pool_size=10, max_overflow=20, pool_pre_ping=True for
健壮连接管理.

Similarly, receiptwitness/queue/email.py:get_redis() was creating a new
Redis connection on every call with no pooling.  Share a
ConnectionPool (max_connections=30) across all get_redis() callers.

Fixes CAR-1078
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-28 18:53:05 +00:00
Chris Farhood d90b00d7ac Add .mcp.json 2026-05-25 21:47:10 +00:00
Savannah Savings 8983fe5d8f Merge pull request 'Promote to Production: CAR-894 Gitea workflows migration' (#270) from uat into main 2026-05-24 18:51:41 +00:00
Savannah Savings a26082d099 Merge pull request 'Promote dev → uat: Fix API crash (dispose_engine import)' (#268) from dev into uat
Merge PR #268: Promote dev → uat — Fix API crash (dispose_engine import)

Promotes fix for ImportError/CrashLoopBackOff to UAT environment.

Approved-by: Savannah Savings (CTO)
2026-05-23 15:52:56 +00:00
Savannah Savings f8b8f4feef Merge pull request 'Fix API crash: remove dead dispose_engine import' (#266) from fix/dispose-engine-import into dev
Merge PR #266: Fix API crash — remove dead dispose_engine import

Removes non-existent dispose_engine import from main.py that caused ImportError and CrashLoopBackOff on API pods.

Reviewed-by: Checkout Charlie (QA PASS)
Approved-by: Savannah Savings (CTO)
2026-05-23 15:52:33 +00:00
Flea Flicker 5464e1a671 Fix API crash: remove dead dispose_engine import
ImportError: cannot import name 'dispose_engine' from 'cartsnitch_api.database'
The function does not exist and is unused - lifespan already calls close_db() directly.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 15:11:00 +00:00
Savannah Savings c39b26050b Merge pull request 'Promote dev → uat: CI registry migration [CAR-933]' (#265) from dev into uat
Promote dev → uat: CI registry migration [CAR-933] (#265)
2026-05-23 14:39:41 +00:00
Savannah Savings 9cc1e49d86 Merge pull request 'ci(CAR-933): migrate image registry from ghcr.io to Gitea' (#264) from carl/car-933-gitea-registry into dev
Merge PR #264: Migrate CI from ghcr.io to Gitea package registry (CAR-955)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 14:35:15 +00:00
Coupon Carl 2c4e9985b1 ci: rename GITEA_TOKEN -> REGISTRY_TOKEN to match configured secret name
cpfarhood confirmed the Gitea registry token is configured as REGISTRY_TOKEN
(not GITEA_TOKEN). This applies to both the registry docker login steps
and the infra repo checkout steps in deploy-dev/deploy-uat.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 14:29:45 +00:00
cartsnitch-ci[bot] 821f1d20b3 fix(ci): replace hardcoded cs_carl username and fix kustomize image rename syntax
- Replace hardcoded 'cs_carl' Gitea registry username with '${{ github.actor }}' in all 5 login steps
- Use kustomize 'OLD=NEW:TAG' rename syntax so existing ghcr.io image entries are updated instead of duplicated

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 14:27:24 +00:00
Coupon Carl 555ced4fdc ci: migrate image registry from ghcr.io to git.farh.net
Replace GitHub Container Registry with Gitea's built-in container registry.
- REGISTRY env var: ghcr.io -> git.farh.net
- All 4 build-and-push jobs: replace GHCR login with Gitea registry login
  using cs_carl + GITEA_TOKEN (token already required for infra checkout)
- deploy-dev/deploy-uat: update kustomize image refs to git.farh.net/*
- Also fix legacy api/.gitea/workflows/ci.yml (non-executing nested file)

Required secrets drop from 5 to 3: DOCKERHUB_USERNAME, DOCKERHUB_TOKEN,
GITEA_TOKEN. GHCR_USERNAME and GHCR_TOKEN no longer needed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-23 14:20:38 +00:00
Savannah Savings 6b6a50b9ec Merge pull request 'Promote dev → uat: .gitea/workflows migration [CAR-934]' (#261) from dev into uat
Promote dev → uat: .gitea/workflows migration [CAR-934]

cc @cpfarhood
2026-05-21 19:19:40 +00:00
Savannah Savings 4797f07af9 Merge pull request 'ci: move .github/workflows to .gitea/workflows [CAR-900]' (#259) from barcode-betty/move-workflows-to-gitea into dev
Merge PR #259: ci: move .github/workflows to .gitea/workflows [CAR-897]

QA-approved. CTO merge to dev.

cc @cpfarhood
2026-05-21 19:19:20 +00:00
Flea Flicker 96331c9fa7 Move .github/workflows to .gitea/workflows
- Relocate all CI workflows from .github/workflows/ to .gitea/workflows/
- Root: .github/workflows/ci.yml -> .gitea/workflows/ci.yml
- api/: api/.github/workflows/ci.yml -> api/.gitea/workflows/ci.yml
- common/: common/.github/workflows/ci.yml -> common/.gitea/workflows/ci.yml
- Gitea uses .gitea/workflows/ for CI configuration

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-21 11:59:35 +00:00
Savannah Savings a4e0b664e1 Merge pull request 'fix: use GHCR_USERNAME and GHCR_TOKEN for GHCR login in Gitea Actions' (#258) from betty/fix-gitea-ci-secrets into dev 2026-05-21 06:21:55 +00:00
Flea Flicker f4bbddd0dd fix: use GHCR_USERNAME and GHCR_TOKEN for GHCR login in Gitea Actions
Gitea's GITHUB_TOKEN authenticates against git.farh.net, not ghcr.io.
Use explicit GHCR_USERNAME and GHCR_TOKEN secrets instead.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-21 05:25:36 +00:00
Savannah Savings 7c021c4eb5 Merge pull request 'chore: promote dev to uat - Gitea Actions workflow conversion' (#254) from dev into uat 2026-05-21 04:23:11 +00:00
Savannah Savings 5a97290356 Merge pull request 'ci: convert GitHub Actions to Gitea Actions (ubuntu-latest)' (#253) from betty/car-869-gitea-actions-cartsnitch into dev 2026-05-21 04:22:43 +00:00
Flea Flicker 32495b150b ci: convert GitHub Actions to Gitea Actions (ubuntu-latest)
- Replace all runs-on: runners-cartsnitch with ubuntu-latest
- Remove SARIF upload steps (no Gitea Security tab)
- Replace GitHub App token with secrets.GITEA_TOKEN in deploy-dev and deploy-uat

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-21 03:54:18 +00:00
savannah-savings-cto[bot] a5404dc824 promote: dev → uat (fix auth tsc build) (#252)
promote: dev → uat (fix auth tsc build)
2026-05-05 11:19:44 +00:00
savannah-savings-cto[bot] b39280ee2a fix(auth): exclude test files from tsc build (#251)
fix(auth): exclude test files from tsc build
2026-05-05 11:19:19 +00:00
Chris Farhood 752d7ed3d0 fix(auth): exclude test files from tsc compilation
Exclude src/__tests__ from tsconfig to prevent test files from being
compiled during Docker build. Fixes build-and-push-auth CI failure.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-05-05 11:11:53 +00:00
savannah-savings-cto[bot] 618da593a6 Merge pull request #250 from cartsnitch/dev
ci: promote dev → uat (auth CI pipeline)
2026-05-05 10:56:35 +00:00
savannah-savings-cto[bot] 1f317a0616 Merge pull request #249 from cartsnitch/betty/car-843-auth-ci-pipeline
ci: add auth service build/deploy to CI pipeline
2026-05-05 10:56:19 +00:00
Chris Farhood 912239a97b ci: add auth service build/deploy to CI pipeline
Add build-and-push-auth job and update deploy-dev/uat to include auth image.

- Add AUTH_IMAGE_NAME env var
- Add build-and-push-auth job (modeled on build-and-push-api)
- Add build-and-push-auth to deploy-dev and deploy-uat needs
- Add auth image tag determination and update steps in both deploy jobs
- Update commit messages to include auth
2026-05-05 06:44:15 +00:00
9 changed files with 513 additions and 480 deletions
@@ -16,17 +16,18 @@ permissions:
security-events: write security-events: write
env: env:
REGISTRY: ghcr.io REGISTRY: git.farh.net
IMAGE_NAME: cartsnitch/cartsnitch IMAGE_NAME: cartsnitch/cartsnitch
RECEIPTWITNESS_IMAGE_NAME: cartsnitch/receiptwitness RECEIPTWITNESS_IMAGE_NAME: cartsnitch/receiptwitness
API_IMAGE_NAME: cartsnitch/api API_IMAGE_NAME: cartsnitch/api
AUTH_IMAGE_NAME: cartsnitch/auth
jobs: jobs:
lint: lint:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: actions/setup-node@v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with: with:
node-version: "20" node-version: "20"
cache: npm cache: npm
@@ -37,10 +38,10 @@ jobs:
run: npx tsc --noEmit run: npx tsc --noEmit
test: test:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: actions/setup-node@v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with: with:
node-version: "20" node-version: "20"
cache: npm cache: npm
@@ -49,10 +50,10 @@ jobs:
run: npx vitest run run: npx vitest run
audit: audit:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: actions/setup-node@v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with: with:
node-version: "20" node-version: "20"
cache: npm cache: npm
@@ -61,10 +62,10 @@ jobs:
run: npm audit --audit-level=high run: npm audit --audit-level=high
e2e: e2e:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: actions/setup-node@v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with: with:
node-version: "20" node-version: "20"
cache: npm cache: npm
@@ -73,11 +74,11 @@ jobs:
- run: npx playwright test - run: npx playwright test
lighthouse: lighthouse:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
needs: [test] needs: [test]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: actions/setup-node@v4 - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with: with:
node-version: "20" node-version: "20"
cache: npm cache: npm
@@ -98,14 +99,14 @@ jobs:
CHROME_PATH="$CHROME_PATH" lhci autorun --chrome-flags="--headless=new --no-sandbox --disable-gpu --disable-dev-shm-usage" CHROME_PATH="$CHROME_PATH" lhci autorun --chrome-flags="--headless=new --no-sandbox --disable-gpu --disable-dev-shm-usage"
build-and-push: build-and-push:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
if: github.event_name == 'push' if: github.event_name == 'push'
needs: [lint, test, e2e] needs: [lint, test, e2e]
outputs: outputs:
calver_tag: ${{ steps.calver.outputs.version }} calver_tag: ${{ steps.calver.outputs.version }}
sha_tag: sha-${{ github.sha }} sha_tag: sha-${{ github.sha }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with: with:
fetch-depth: 0 fetch-depth: 0
@@ -133,13 +134,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR - name: Log in to Gitea registry
if: github.event_name == 'push' if: github.event_name == 'push'
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.REGISTRY_TOKEN }}
- name: Extract metadata - name: Extract metadata
id: meta id: meta
@@ -174,11 +175,7 @@ jobs:
only-fixed: "true" only-fixed: "true"
output-format: sarif output-format: sarif
- name: Upload frontend scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: ${{ steps.scan.outputs.sarif }}
- name: Push Docker image - name: Push Docker image
if: github.event_name == 'push' if: github.event_name == 'push'
@@ -198,14 +195,14 @@ jobs:
git push origin "v${{ steps.calver.outputs.version }}" git push origin "v${{ steps.calver.outputs.version }}"
build-and-push-receiptwitness: build-and-push-receiptwitness:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
if: github.event_name == 'push' if: github.event_name == 'push'
needs: [lint, test] needs: [lint, test]
outputs: outputs:
calver_tag: ${{ steps.calver.outputs.version }} calver_tag: ${{ steps.calver.outputs.version }}
sha_tag: sha-${{ github.sha }} sha_tag: sha-${{ github.sha }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with: with:
fetch-depth: 0 fetch-depth: 0
@@ -227,13 +224,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR - name: Log in to Gitea registry
if: github.event_name == 'push' if: github.event_name == 'push'
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.REGISTRY_TOKEN }}
- name: Extract metadata - name: Extract metadata
id: meta id: meta
@@ -270,11 +267,7 @@ jobs:
only-fixed: "true" only-fixed: "true"
output-format: sarif output-format: sarif
- name: Upload receiptwitness scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: ${{ steps.scan.outputs.sarif }}
- name: Push Docker image - name: Push Docker image
if: github.event_name == 'push' if: github.event_name == 'push'
@@ -290,14 +283,14 @@ jobs:
cache-from: type=gha cache-from: type=gha
build-and-push-api: build-and-push-api:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
if: github.event_name == 'push' if: github.event_name == 'push'
needs: [lint, test] needs: [lint, test]
outputs: outputs:
calver_tag: ${{ steps.calver.outputs.version }} calver_tag: ${{ steps.calver.outputs.version }}
sha_tag: sha-${{ github.sha }} sha_tag: sha-${{ github.sha }}
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with: with:
fetch-depth: 0 fetch-depth: 0
@@ -319,13 +312,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR - name: Log in to Gitea registry
if: github.event_name == 'push' if: github.event_name == 'push'
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.REGISTRY_TOKEN }}
- name: Extract metadata (API) - name: Extract metadata (API)
id: meta id: meta
@@ -362,11 +355,7 @@ jobs:
only-fixed: "true" only-fixed: "true"
output-format: sarif output-format: sarif
- name: Upload api scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: ${{ steps.scan.outputs.sarif }}
- name: Push Docker image - name: Push Docker image
if: github.event_name == 'push' if: github.event_name == 'push'
@@ -381,25 +370,104 @@ jobs:
APT_CACHE_BUST=${{ github.run_id }} APT_CACHE_BUST=${{ github.run_id }}
cache-from: type=gha cache-from: type=gha
build-and-push-auth:
runs-on: ubuntu-latest
if: github.event_name == 'push'
needs: [lint, test]
outputs:
calver_tag: ${{ steps.calver.outputs.version }}
sha_tag: sha-${{ github.sha }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
- name: Generate CalVer tag
id: calver
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
run: |
DATE_TAG=$(date -u +%Y.%m.%d)
EXISTING=$(git tag -l "v${DATE_TAG}*" | sort -V | tail -1)
if [ -z "$EXISTING" ]; then VERSION="$DATE_TAG"
elif [ "$EXISTING" = "v${DATE_TAG}" ]; then VERSION="${DATE_TAG}.2"
else BUILD_NUM=$(echo "$EXISTING" | sed "s/v${DATE_TAG}\.//"); VERSION="${DATE_TAG}.$((BUILD_NUM + 1))"; fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
- name: Log in to Docker Hub
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to Gitea registry
if: github.event_name == 'push'
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Extract metadata (auth)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.AUTH_IMAGE_NAME }}
tags: |
type=sha,prefix=sha-,format=long
type=raw,value=${{ steps.calver.outputs.version }},enable=${{ github.ref == 'refs/heads/main' }}
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
- name: Build Docker image
uses: docker/build-push-action@v6
with:
context: ./auth
file: ./auth/Dockerfile
load: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
APT_CACHE_BUST=${{ github.run_id }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Scan auth image for vulnerabilities
uses: anchore/scan-action@v5
id: scan
env:
GRYPE_CONFIG: .grype.yaml
with:
image: "${{ env.REGISTRY }}/${{ env.AUTH_IMAGE_NAME }}:sha-${{ github.sha }}"
fail-build: true
severity-cutoff: high
only-fixed: "true"
output-format: sarif
- name: Push Docker image
if: github.event_name == 'push'
uses: docker/build-push-action@v6
with:
context: ./auth
file: ./auth/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
APT_CACHE_BUST=${{ github.run_id }}
cache-from: type=gha
deploy-dev: deploy-dev:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
needs: [build-and-push, build-and-push-receiptwitness, build-and-push-api] needs: [build-and-push, build-and-push-receiptwitness, build-and-push-api, build-and-push-auth]
if: always() && !cancelled() && github.event_name == 'push' && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main') if: always() && !cancelled() && github.event_name == 'push' && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main')
steps: steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v1
with:
app-id: ${{ secrets.CARTSNITCH_APP_ID }}
private-key: ${{ secrets.CARTSNITCH_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: infra
- name: Checkout infra repo - name: Checkout infra repo
uses: actions/checkout@v4 uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with: with:
repository: cartsnitch/infra repository: cartsnitch/infra
token: ${{ steps.app-token.outputs.token }} token: ${{ secrets.REGISTRY_TOKEN }}
ref: main ref: main
path: infra path: infra
@@ -422,7 +490,7 @@ jobs:
if: needs.build-and-push.result == 'success' if: needs.build-and-push.result == 'success'
run: | run: |
cd infra/apps/overlays/dev cd infra/apps/overlays/dev
kustomize edit set image ghcr.io/cartsnitch/cartsnitch:${{ steps.frontend_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/cartsnitch=git.farh.net/cartsnitch/cartsnitch:${{ steps.frontend_tag.outputs.tag }}
- name: Determine image tag for receiptwitness - name: Determine image tag for receiptwitness
id: receiptwitness_tag id: receiptwitness_tag
@@ -437,7 +505,7 @@ jobs:
if: needs.build-and-push-receiptwitness.result == 'success' if: needs.build-and-push-receiptwitness.result == 'success'
run: | run: |
cd infra/apps/overlays/dev cd infra/apps/overlays/dev
kustomize edit set image ghcr.io/cartsnitch/receiptwitness:${{ steps.receiptwitness_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/receiptwitness=git.farh.net/cartsnitch/receiptwitness:${{ steps.receiptwitness_tag.outputs.tag }}
- name: Determine image tag for api - name: Determine image tag for api
id: api_tag id: api_tag
@@ -452,38 +520,44 @@ jobs:
if: needs.build-and-push-api.result == 'success' if: needs.build-and-push-api.result == 'success'
run: | run: |
cd infra/apps/overlays/dev cd infra/apps/overlays/dev
kustomize edit set image ghcr.io/cartsnitch/api:${{ steps.api_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/api=git.farh.net/cartsnitch/api:${{ steps.api_tag.outputs.tag }}
- name: Determine image tag for auth
id: auth_tag
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
echo "tag=${{ needs.build-and-push-auth.outputs.calver_tag }}" >> "$GITHUB_OUTPUT"
else
echo "tag=${{ needs.build-and-push-auth.outputs.sha_tag }}" >> "$GITHUB_OUTPUT"
fi
- name: Update auth image tag
if: needs.build-and-push-auth.result == 'success'
run: |
cd infra/apps/overlays/dev
kustomize edit set image ghcr.io/cartsnitch/auth=git.farh.net/cartsnitch/auth:${{ steps.auth_tag.outputs.tag }}
- name: Commit and push to infra - name: Commit and push to infra
run: | run: |
cd infra cd infra
git config user.name "cartsnitch-ci[bot]" git config user.name "cartsnitch-ci[bot]"
git config user.email "cartsnitch-ci[bot]@users.noreply.github.com" git config user.email "cartsnitch-ci[bot]@users.noreply.git.farh.net"
git add apps/overlays/dev/kustomization.yaml git add apps/overlays/dev/kustomization.yaml
git diff --cached --quiet && echo "No image changes to deploy" && exit 0 git diff --cached --quiet && echo "No image changes to deploy" && exit 0
git commit -m "ci(dev): update cartsnitch, receiptwitness, and api images" git commit -m "ci(dev): update cartsnitch, receiptwitness, api, and auth images"
git pull --rebase origin main git pull --rebase origin main
git push origin main git push origin main
deploy-uat: deploy-uat:
runs-on: runners-cartsnitch runs-on: ubuntu-latest
needs: [build-and-push, build-and-push-receiptwitness, build-and-push-api] needs: [build-and-push, build-and-push-receiptwitness, build-and-push-api, build-and-push-auth]
if: always() && !cancelled() && github.event_name == 'push' && (github.ref == 'refs/heads/uat' || github.ref == 'refs/heads/main') if: always() && !cancelled() && github.event_name == 'push' && (github.ref == 'refs/heads/uat' || github.ref == 'refs/heads/main')
steps: steps:
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v1
with:
app-id: ${{ secrets.CARTSNITCH_APP_ID }}
private-key: ${{ secrets.CARTSNITCH_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: infra
- name: Checkout infra repo - name: Checkout infra repo
uses: actions/checkout@v4 uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with: with:
repository: cartsnitch/infra repository: cartsnitch/infra
token: ${{ steps.app-token.outputs.token }} token: ${{ secrets.REGISTRY_TOKEN }}
ref: main ref: main
path: infra path: infra
@@ -506,7 +580,7 @@ jobs:
if: needs.build-and-push.result == 'success' if: needs.build-and-push.result == 'success'
run: | run: |
cd infra/apps/overlays/uat cd infra/apps/overlays/uat
kustomize edit set image ghcr.io/cartsnitch/cartsnitch:${{ steps.frontend_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/cartsnitch=git.farh.net/cartsnitch/cartsnitch:${{ steps.frontend_tag.outputs.tag }}
- name: Determine image tag for receiptwitness - name: Determine image tag for receiptwitness
id: receiptwitness_tag id: receiptwitness_tag
@@ -521,7 +595,7 @@ jobs:
if: needs.build-and-push-receiptwitness.result == 'success' if: needs.build-and-push-receiptwitness.result == 'success'
run: | run: |
cd infra/apps/overlays/uat cd infra/apps/overlays/uat
kustomize edit set image ghcr.io/cartsnitch/receiptwitness:${{ steps.receiptwitness_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/receiptwitness=git.farh.net/cartsnitch/receiptwitness:${{ steps.receiptwitness_tag.outputs.tag }}
- name: Determine image tag for api - name: Determine image tag for api
id: api_tag id: api_tag
@@ -536,15 +610,30 @@ jobs:
if: needs.build-and-push-api.result == 'success' if: needs.build-and-push-api.result == 'success'
run: | run: |
cd infra/apps/overlays/uat cd infra/apps/overlays/uat
kustomize edit set image ghcr.io/cartsnitch/api:${{ steps.api_tag.outputs.tag }} kustomize edit set image ghcr.io/cartsnitch/api=git.farh.net/cartsnitch/api:${{ steps.api_tag.outputs.tag }}
- name: Determine image tag for auth
id: auth_tag
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
echo "tag=${{ needs.build-and-push-auth.outputs.calver_tag }}" >> "$GITHUB_OUTPUT"
else
echo "tag=${{ needs.build-and-push-auth.outputs.sha_tag }}" >> "$GITHUB_OUTPUT"
fi
- name: Update auth image tag
if: needs.build-and-push-auth.result == 'success'
run: |
cd infra/apps/overlays/uat
kustomize edit set image ghcr.io/cartsnitch/auth=git.farh.net/cartsnitch/auth:${{ steps.auth_tag.outputs.tag }}
- name: Commit and push to infra - name: Commit and push to infra
run: | run: |
cd infra cd infra
git config user.name "cartsnitch-ci[bot]" git config user.name "cartsnitch-ci[bot]"
git config user.email "cartsnitch-ci[bot]@users.noreply.github.com" git config user.email "cartsnitch-ci[bot]@users.noreply.git.farh.net"
git add apps/overlays/uat/kustomization.yaml git add apps/overlays/uat/kustomization.yaml
git diff --cached --quiet && echo "No image changes to deploy" && exit 0 git diff --cached --quiet && echo "No image changes to deploy" && exit 0
git commit -m "ci(uat): update cartsnitch, receiptwitness, and api images" git commit -m "ci(uat): update cartsnitch, receiptwitness, api, and auth images"
git pull --rebase origin main git pull --rebase origin main
git push origin main git push origin main
@@ -15,7 +15,7 @@ permissions:
packages: write packages: write
env: env:
REGISTRY: ghcr.io REGISTRY: git.farh.net
IMAGE_NAME: cartsnitch/api IMAGE_NAME: cartsnitch/api
jobs: jobs:
@@ -130,13 +130,13 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR - name: Log in to Gitea registry
if: github.event_name == 'push' && github.ref == 'refs/heads/main' if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: docker/login-action@v3 uses: docker/login-action@v3
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.REGISTRY_TOKEN }}
- name: Extract metadata - name: Extract metadata
id: meta id: meta
-1
View File
@@ -6,7 +6,6 @@ from fastapi import APIRouter, FastAPI
from cartsnitch_api.auth.routes import router as auth_router from cartsnitch_api.auth.routes import router as auth_router
from cartsnitch_api.cache import cache_client from cartsnitch_api.cache import cache_client
from cartsnitch_api.database import dispose_engine
from cartsnitch_api.middleware.cors import add_cors_middleware from cartsnitch_api.middleware.cors import add_cors_middleware
from cartsnitch_api.middleware.error_handler import add_error_handlers, add_error_monitor_middleware from cartsnitch_api.middleware.error_handler import add_error_handlers, add_error_monitor_middleware
from cartsnitch_api.middleware.rate_limit import add_rate_limit_middleware from cartsnitch_api.middleware.rate_limit import add_rate_limit_middleware
+1 -1
View File
@@ -12,5 +12,5 @@
"resolveJsonModule": true "resolveJsonModule": true
}, },
"include": ["src"], "include": ["src"],
"exclude": ["node_modules", "dist"] "exclude": ["node_modules", "dist", "src/__tests__"]
} }
+23 -4
View File
@@ -1,17 +1,36 @@
"""Database engine and session factories for sync and async usage.""" """Database engine and session factories for sync and async usage."""
from collections.abc import AsyncGenerator, Generator from collections.abc import AsyncGenerator, Generator
from typing import TYPE_CHECKING
from sqlalchemy import create_engine from sqlalchemy import create_engine
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker, create_async_engine
from sqlalchemy.orm import Session, sessionmaker from sqlalchemy.orm import Session, sessionmaker
from cartsnitch_common.config import settings from cartsnitch_common.config import settings
if TYPE_CHECKING:
from sqlalchemy.engine import Engine
def get_async_engine(url: str | None = None): # Module-level async engine cache — one engine per unique URL, shared across all callers.
"""Create an async SQLAlchemy engine.""" # This prevents pool exhaustion in high-throughput workers (e.g. email-worker hitting
return create_async_engine(url or settings.database_url, echo=settings.debug) # DragonflyDB/Postgres repeatedly per message). pool_size=10, max_overflow=20 gives
# headroom for bursts while capping max connections at 30 per URL.
_async_engine_cache: dict[str, "AsyncEngine"] = {}
def get_async_engine(url: str | None = None) -> "AsyncEngine":
"""Get or create a cached async engine for the given URL."""
target = url or settings.database_url
if target not in _async_engine_cache:
_async_engine_cache[target] = create_async_engine(
target,
echo=settings.debug,
pool_size=10,
max_overflow=20,
pool_pre_ping=True,
)
return _async_engine_cache[target]
def get_sync_engine(url: str | None = None): def get_sync_engine(url: str | None = None):
+290 -384
View File
File diff suppressed because it is too large Load Diff
+5 -3
View File
@@ -45,14 +45,16 @@
"typescript-eslint": "^8.56.1", "typescript-eslint": "^8.56.1",
"vite": "^6.4.2", "vite": "^6.4.2",
"vite-plugin-pwa": "^0.21.2", "vite-plugin-pwa": "^0.21.2",
"vitest": "^3.2.4" "vitest": "^4.1.8"
}, },
"overrides": { "overrides": {
"@rollup/pluginutils": "5.3.0", "@rollup/pluginutils": "5.3.0",
"flatted": "^3.4.2", "flatted": "^3.4.2",
"serialize-javascript": "7.0.5", "serialize-javascript": "7.0.5",
"brace-expansion": ">=1.1.13", "brace-expansion": ">=5.0.6",
"lodash": ">=4.17.24", "lodash": ">=4.17.24",
"minimatch": "^10.2.4" "minimatch": "^10.2.4",
"@babel/plugin-transform-modules-systemjs": "^7.29.4",
"fast-uri": "^3.1.2"
} }
} }
@@ -16,6 +16,29 @@ logger = logging.getLogger(__name__)
STREAM_KEY = "email:receipts" STREAM_KEY = "email:receipts"
CONSUMER_GROUP = "email-workers" CONSUMER_GROUP = "email-workers"
# Module-level Redis/DragonflyDB connection pool — shared across all worker calls.
# Without pooling, each call to get_redis() opens a new TCP connection. In a tight
# consumer loop this causes ConnectionResetError when DragonflyDB's connection limit
# is hit under load. max_connections=30 (10 base + 20 overflow) mirrors the engine pool.
_redis_pool: aioredis.ConnectionPool | None = None
def _get_redis_pool() -> aioredis.ConnectionPool:
"""Get or create the shared DragonflyDB connection pool."""
global _redis_pool
if _redis_pool is None:
_redis_pool = aioredis.ConnectionPool.from_url(
settings.redis_url,
decode_responses=True,
max_connections=30,
)
return _redis_pool
async def get_redis() -> aioredis.Redis:
"""Get async Redis/DragonflyDB client backed by a shared connection pool."""
return aioredis.Redis(connection_pool=_get_redis_pool())
@dataclass @dataclass
class EmailJob: class EmailJob:
@@ -31,11 +54,6 @@ class EmailJob:
message_id: str # from email provider, for dedup message_id: str # from email provider, for dedup
async def get_redis() -> aioredis.Redis:
"""Get async Redis/DragonflyDB client."""
return cast(aioredis.Redis, aioredis.from_url(settings.redis_url, decode_responses=True))
async def ensure_consumer_group(client: aioredis.Redis) -> None: async def ensure_consumer_group(client: aioredis.Redis) -> None:
"""Create consumer group if it does not exist.""" """Create consumer group if it does not exist."""
try: try: