Commit Graph

63 Commits

Author SHA1 Message Date
Dotta 778e775c35 Add secrets provider vaults and remote import (#5429)
## Thinking Path

> - Paperclip orchestrates AI-agent companies and needs secrets handling
to work across local development, hosted operators, and governed agent
execution.
> - The affected subsystem is the company-scoped secrets control plane:
database schema, server services/routes, CLI workflows, and the Secrets
settings UI.
> - The gap was that secrets were local-only and operators could not
manage provider vaults or import existing remote references without
exposing plaintext.
> - This branch adds provider vault configuration plus an AWS Secrets
Manager remote-import path while preserving company boundaries, binding
context, and audit trails.
> - I kept the PR to a single branch PR, removed unrelated
lockfile/package drift, rebased the full branch onto the current
`public-gh/master`, and addressed fresh Greptile findings.
> - The benefit is a reviewable implementation of provider-backed
secrets with focused tests covering provider selection, import
conflicts, deleted secret reuse, rotation guards, and AWS signing
behavior.

## What Changed

- Added provider vault support for company secrets, including provider
config storage, default vault handling, health checks, binding usage,
access events, and remote import preview/commit.
- Added an AWS Secrets Manager provider using SigV4 request signing,
bounded request timeouts, namespace guardrails, cached runtime
credential resolution, and external-reference linking without plaintext
reads.
- Added Secrets UI surfaces for vault management and remote import, plus
CLI/API documentation for setup and operations.
- Stabilized routine webhook secret binding paths and SSH
environment-driver fixture bindings discovered during verification.
- Addressed Greptile and CI findings: no lockfile/package drift,
monotonic migration metadata, disabled-vault default races, soft-deleted
secret hiding/recreate behavior, remove behavior with disabled vaults,
soft-deleted external-reference re-import, non-active rotation guards,
managed-secret soft deletion through PATCH, and per-call AWS SDK
credential client churn.
- Rebased this branch onto `public-gh/master` at `0e1a5828` and
force-pushed with lease to keep this as the single PR for the branch.

## Verification

- `git fetch public-gh master`
- `git rebase public-gh/master`
- `git diff --name-only public-gh/master...HEAD | grep
'^pnpm-lock\.yaml$' || true` confirmed `pnpm-lock.yaml` is not in the PR
diff.
- Confirmed migration ordering: master ends at `0081_optimal_dormammu`;
this PR adds `0082_dry_vision` and
`0083_company_secret_provider_configs`.
- Inspected migrations for repeat safety: new tables/indexes use `IF NOT
EXISTS`; foreign keys are guarded by `DO $$ ... IF NOT EXISTS`; column
additions use `ADD COLUMN IF NOT EXISTS`.
- `pnpm -r typecheck` passed before the Greptile follow-up commits.
- `pnpm test:run` ran the full stable Vitest path before the Greptile
follow-up commits; it completed with 3 timing-related failures under
parallel load: `codex-local-execute.test.ts`,
`cursor-local-execute.test.ts`, and `environment-service.test.ts`.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/codex-local-execute.test.ts
src/__tests__/cursor-local-execute.test.ts
src/__tests__/environment-service.test.ts` passed on targeted rerun
(`24/24`).
- `pnpm build` passed before the Greptile follow-up commits. Vite
reported existing chunk-size/dynamic-import warnings.
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/secrets-service.test.ts` passed (`26/26`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/aws-secrets-manager-provider.test.ts
src/__tests__/secrets-service.test.ts` passed (`39/39`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
typecheck` passed.
- Captured Storybook screenshots from `ui/storybook-static` for visual
review.
- Latest PR checks on `5ca3a5cf`: `policy`, serialized server suites
1/4-4/4, `Canary Dry Run`, `e2e`, `security/snyk`, and `Greptile Review`
pass; aggregate `verify` is still registering the completed child
checks.
- Greptile review loop continued through the latest requested pass; all
Greptile review threads are resolved and the latest `Greptile Review`
check on `5ca3a5cf` passed with 0 comments added.

## Screenshots

Before: the provider-vault and remote-import surfaces did not exist on
`master`; these are after-state screenshots from the Storybook fixtures.

![Secrets
inventory](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secrets-inventory.png)

![Secret binding
picker](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secret-binding-picker.png)

![Environment editor with
secrets](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/env-editor-with-secrets.png)

## Risks

- Migration risk: this adds new secret provider tables and extends
existing secret rows. The migrations were checked for monotonic ordering
and idempotent guards, but reviewers should still inspect upgrade
behavior carefully.
- Provider risk: AWS support uses direct SigV4 requests. Automated tests
cover signing, request timeouts, vault-config selection, namespace
guardrails, pending-version archival, sanitized provider errors, and
service-level cleanup paths. A real-vault AWS smoke test remains
deployment validation for an operator with AWS credentials rather than
an unverified merge blocker in this local branch.
- UI risk: the Secrets page and import dialog are large new surfaces;
screenshots are included above for reviewer inspection.
- Verification risk: the full local stable test command hit
parallel-load timing failures, although the exact failed files passed
when rerun directly.
- Operational risk: remote import intentionally avoids plaintext reads;
operators must understand that imported external references resolve at
runtime and may fail if AWS permissions change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 coding agent with local shell/tool use in the
Paperclip worktree. Exact context-window size was not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 18:22:17 -05:00
Dotta 7f893ac4ec [codex] Harden execution reliability and heartbeat tooling (#3679)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Reliable execution depends on heartbeat routing, issue lifecycle
semantics, telemetry, and a fast enough local verification loop to keep
regressions visible
> - The remaining commits on this branch were mostly server/runtime
correctness fixes plus test and documentation follow-ups in that area
> - Those changes are logically separate from the UI-focused
issue-detail and workspace/navigation branches even when they touch
overlapping issue APIs
> - This pull request groups the execution reliability, heartbeat,
telemetry, and tooling changes into one standalone branch
> - The benefit is a focused review of the control-plane correctness
work, including the follow-up fix that restored the implicit
comment-reopen helpers after branch splitting

## What Changed

- Hardened issue/heartbeat execution behavior, including self-review
stage skipping, deferred mention wakes during active execution, stranded
execution recovery, active-run scoping, assignee resolution, and
blocked-to-todo wake resumption
- Reduced noisy polling/logging overhead by trimming issue run payloads,
compacting persisted run logs, silencing high-volume request logs, and
capping heartbeat-run queries in dashboard/inbox surfaces
- Expanded telemetry and status semantics with adapter/model fields on
task completion plus clearer status guidance in docs/onboarding material
- Updated test infrastructure and verification defaults with faster
route-test module isolation, cheaper default `pnpm test`, e2e isolation
from local state, and repo verification follow-ups
- Included docs/release housekeeping from the branch and added a small
follow-up commit restoring the implicit comment-reopen helpers that were
dropped during branch reconstruction

## Verification

- `pnpm vitest run
server/src/__tests__/issue-comment-reopen-routes.test.ts
server/src/__tests__/issue-telemetry-routes.test.ts`
- `pnpm vitest run server/src/__tests__/http-log-policy.test.ts
server/src/__tests__/heartbeat-run-log.test.ts
server/src/__tests__/health.test.ts`
- `server/src/__tests__/activity-service.test.ts`,
`server/src/__tests__/heartbeat-comment-wake-batching.test.ts`, and
`server/src/__tests__/heartbeat-process-recovery.test.ts` were attempted
on this host but the embedded Postgres harness reported
init-script/data-dir problems and skipped or failed to start, so they
are noted as environment-limited

## Risks

- Medium: this branch changes core issue/heartbeat routing and
reopen/wakeup behavior, so regressions would affect agent execution flow
rather than isolated UI polish
- Because it also updates verification infrastructure, reviewers should
pay attention to whether the new tests are asserting the right failure
modes and not just reshaping harness behavior

## Model Used

- OpenAI Codex coding agent (GPT-5-class runtime in Codex CLI; exact
deployed model ID is not exposed in this environment), reasoning
enabled, tool use and local code execution enabled

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-14 13:34:52 -05:00
Dotta 8bdf4081ee chore: improve worktree tooling and security docs 2026-04-10 22:26:30 -05:00
dotta b1e9215375 docs: add browser process cleanup plan 2026-04-09 06:14:12 -05:00
dotta 5758aba91e docs: add agent-os follow-up plan 2026-04-09 06:14:12 -05:00
dotta 482dac7097 docs: add agent-os technical report 2026-04-09 06:14:12 -05:00
dotta 0937f07c79 Remove standalone issue recovery plan doc 2026-04-09 06:14:12 -05:00
dotta d00860b12a Add in-progress issue recovery plan 2026-04-09 06:14:12 -05:00
dotta 56ee63bfd0 docs: add issue detail speed inventory plan 2026-04-09 06:14:12 -05:00
dotta 87db949d3f docs: survey pi and pi-mono hook surfaces 2026-04-09 06:14:12 -05:00
dotta 2c2e13eac2 merge master into pap-1167-app-ui-bundle 2026-04-07 07:10:14 -05:00
dotta 6b4f3b56e4 docs: add sub-issue issue detail plan 2026-04-06 21:24:22 -05:00
dotta 1a3aee9ee1 docs: add smart model routing plan
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-04-06 21:23:33 -05:00
dotta 5a73556871 Use positional source arg for company import
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-23 08:14:51 -05:00
dotta 9e19f1d005 Merge public-gh/master into paperclip-company-import-export 2026-03-18 09:57:26 -05:00
dotta 19f4a78f4a feat: add release smoke workflow 2026-03-18 07:59:32 -05:00
dotta 3e0e15394a chore: switch release calver to mdd patch 2026-03-18 07:57:36 -05:00
Dotta 4d8c988dab fix: use one workflow for npm trusted publishing 2026-03-17 14:18:42 -05:00
Dotta 21c1235277 chore: automate canary and stable releases 2026-03-17 14:08:55 -05:00
Dotta 7b9718cbaa docs: plan memory service surface API
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-17 12:07:14 -05:00
Dotta b5aeae7e22 Merge public-gh/master into paperclip-company-import-export 2026-03-17 10:45:14 -05:00
Dotta bb7d1b2c71 Merge remote-tracking branch 'public-gh/master' into paperclip-subissues
* public-gh/master:
  Fix budget incident resolution edge cases
  Fix agent budget tab routing
  Fix budget auth and monthly spend rollups
  Harden budget enforcement and migration startup
  Add budget tabs and sidebar budget indicators
  feat(costs): add billing, quota, and budget control plane
  refactor(quota): move provider quota logic into adapter layer, add unit tests
  fix(costs): replace non-null map assertions with nullish coalescing, clarify weekData guard
  fix(costs): guard byProject against duplicate null keys, memoize ProviderQuotaCard row aggregations
  fix(costs): align byAgent run filter to startedAt, tighten providerTabItems memo deps, stabilize byProject row keys
  feat(costs): add agent model breakdown, harden date validation, sync CostByProject type, fix quota threshold and tab-gated queries
  fix(costs): harden company auth check, fix frozen date memo, hide empty quota rows
  fix(costs): guard routes, fix DST ranges, sync provider state, wire live updates
  feat(costs): consolidate /usage into /costs with Spend + Providers tabs
  feat(usage): add subscription quota windows per provider on /usage page
  address greptile review: per-provider deficit notch, startedAt filter, weekRange refresh, deduplicate providerDisplayName
  feat(ui): add resource and usage dashboard (/usage route)

# Conflicts:
#	packages/db/src/migration-runtime.ts
#	packages/db/src/migrations/meta/0031_snapshot.json
#	packages/db/src/migrations/meta/_journal.json
2026-03-16 17:19:55 -05:00
Dotta cca086b863 Merge public-gh/master into paperclip-company-import-export 2026-03-16 17:02:39 -05:00
Dotta 76e6cc08a6 feat(costs): add billing, quota, and budget control plane 2026-03-16 15:11:01 -05:00
Dotta 2c35be0212 Merge public-gh/master into paperclip-company-import-export 2026-03-14 21:45:54 -05:00
Dotta b2c0f3f9a5 Refine portability export behavior and skill plans 2026-03-14 18:59:26 -05:00
Dotta 6e6d67372c Merge remote-tracking branch 'public-gh/master' into paperclip-subissues
* public-gh/master:
  Drop lockfile from watcher change
  Tighten plugin dev file watching
  Fix plugin smoke example typecheck
  Fix plugin dev watcher and migration snapshot
  Clarify plugin authoring and external dev workflow
  Expand kitchen sink plugin demos
  fix: set AGENT_HOME env var for agent processes
  Add kitchen sink plugin example
  Simplify plugin runtime and cleanup lifecycle
  Add plugin framework and settings UI

# Conflicts:
#	packages/db/src/migrations/meta/0029_snapshot.json
#	packages/db/src/migrations/meta/_journal.json
2026-03-14 13:56:09 -05:00
Dotta 8cf85a5a50 Merge remote-tracking branch 'public-gh/master' into paperclip-subissues
* public-gh/master: (55 commits)
  fix(issue-documents): address greptile review
  Update packages/shared/src/validators/issue.ts
  feat(ui): add issue document copy and download actions
  fix(ui): unify new issue upload action
  feat(ui): stage issue files before create
  feat(ui): handle issue document edit conflicts
  fix(ui): refresh issue documents from live events
  feat(ui): deep link issue documents
  fix(ui): streamline issue document chrome
  fix(ui): collapse empty document and attachment states
  fix(ui): simplify document card body layout
  fix(issues): address document review comments
  feat(issues): add issue documents and inline editing
  docs: add agent evals framework plan
  fix(cli): quote env values with special characters
  Fix worktree seed source selection
  fix: address greptile follow-up
  docs: add paperclip skill tightening plan
  fix: isolate codex home in worktrees
  Add worktree UI branding
  ...

# Conflicts:
#	packages/db/src/migrations/meta/0028_snapshot.json
#	packages/db/src/migrations/meta/_journal.json
#	packages/shared/src/index.ts
#	server/src/routes/issues.ts
#	ui/src/api/issues.ts
#	ui/src/components/NewIssueDialog.tsx
#	ui/src/pages/IssueDetail.tsx
2026-03-14 12:24:40 -05:00
Dotta eafb5b8fd9 Merge public-gh/master into feature/plugin-runtime-instance-cleanup 2026-03-14 10:46:19 -05:00
Dotta 2137c2f715 Expand skills UI product plan 2026-03-14 10:15:04 -05:00
Dotta 58a9259a2e Update skill package docs and plans 2026-03-14 10:13:20 -05:00
Dotta 1d8f514d10 Refine company package export format 2026-03-14 09:46:16 -05:00
Dotta 6fa1dd2197 Add kitchen sink plugin example 2026-03-13 23:03:51 -05:00
Dotta 2975aa950b Refine company package spec and rollout plan 2026-03-13 21:36:19 -05:00
Dotta 29b70e0c36 Add company import export v2 plan 2026-03-13 21:10:45 -05:00
Dotta 9da5358bb3 Add workspace technical implementation spec 2026-03-13 16:37:40 -05:00
Dotta db81a06386 docs: add agent evals framework plan 2026-03-13 15:07:56 -05:00
Dotta c1430e7b06 docs: add paperclip skill tightening plan
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 14:37:44 -05:00
Dotta 528505a04a fix: isolate codex home in worktrees 2026-03-13 11:53:56 -05:00
Dotta 25d3bf2c64 Incorporate Worktrunk patterns into workspace plan 2026-03-13 09:41:12 -05:00
Dotta 7d1748b3a7 feat: optimize heartbeat token usage
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 09:40:43 -05:00
Dotta e2f0241533 docs: add dated plan naming rule and align workspace plan 2026-03-13 09:16:28 -05:00
Dotta 89e247b410 Expand workspace plan for migration and cloud execution 2026-03-13 09:15:36 -05:00
Dotta 216cb3fb28 Add workspace product model plan 2026-03-13 09:15:36 -05:00
Dotta 84fc6d4a87 docs: add token optimization plan
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 09:15:36 -05:00
Dotta 9c7d9ded1e docs: organize plans into doc/plans with date prefixes
Move plans from doc/plan/ into doc/plans/ and add YYYY-MM-DD date
prefixes to all undated plan files based on document headers or
earliest git commit dates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 09:11:56 -05:00
Dotta 752a53e38e Expand workspace plan for migration and cloud execution 2026-03-13 09:06:49 -05:00
Dotta 284bd733b9 Add workspace product model plan 2026-03-13 08:41:01 -05:00
Dotta 2c5e48993d docs: update PRODUCT.md and add 2026-03-13 features plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 07:25:23 -05:00
Dotta c3cf4279fa Add agent chat architecture plan
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-12 08:39:41 -05:00