Files
paperclip/docs/api/secrets.md
T
Dotta 778e775c35 Add secrets provider vaults and remote import (#5429)
## Thinking Path

> - Paperclip orchestrates AI-agent companies and needs secrets handling
to work across local development, hosted operators, and governed agent
execution.
> - The affected subsystem is the company-scoped secrets control plane:
database schema, server services/routes, CLI workflows, and the Secrets
settings UI.
> - The gap was that secrets were local-only and operators could not
manage provider vaults or import existing remote references without
exposing plaintext.
> - This branch adds provider vault configuration plus an AWS Secrets
Manager remote-import path while preserving company boundaries, binding
context, and audit trails.
> - I kept the PR to a single branch PR, removed unrelated
lockfile/package drift, rebased the full branch onto the current
`public-gh/master`, and addressed fresh Greptile findings.
> - The benefit is a reviewable implementation of provider-backed
secrets with focused tests covering provider selection, import
conflicts, deleted secret reuse, rotation guards, and AWS signing
behavior.

## What Changed

- Added provider vault support for company secrets, including provider
config storage, default vault handling, health checks, binding usage,
access events, and remote import preview/commit.
- Added an AWS Secrets Manager provider using SigV4 request signing,
bounded request timeouts, namespace guardrails, cached runtime
credential resolution, and external-reference linking without plaintext
reads.
- Added Secrets UI surfaces for vault management and remote import, plus
CLI/API documentation for setup and operations.
- Stabilized routine webhook secret binding paths and SSH
environment-driver fixture bindings discovered during verification.
- Addressed Greptile and CI findings: no lockfile/package drift,
monotonic migration metadata, disabled-vault default races, soft-deleted
secret hiding/recreate behavior, remove behavior with disabled vaults,
soft-deleted external-reference re-import, non-active rotation guards,
managed-secret soft deletion through PATCH, and per-call AWS SDK
credential client churn.
- Rebased this branch onto `public-gh/master` at `0e1a5828` and
force-pushed with lease to keep this as the single PR for the branch.

## Verification

- `git fetch public-gh master`
- `git rebase public-gh/master`
- `git diff --name-only public-gh/master...HEAD | grep
'^pnpm-lock\.yaml$' || true` confirmed `pnpm-lock.yaml` is not in the PR
diff.
- Confirmed migration ordering: master ends at `0081_optimal_dormammu`;
this PR adds `0082_dry_vision` and
`0083_company_secret_provider_configs`.
- Inspected migrations for repeat safety: new tables/indexes use `IF NOT
EXISTS`; foreign keys are guarded by `DO $$ ... IF NOT EXISTS`; column
additions use `ADD COLUMN IF NOT EXISTS`.
- `pnpm -r typecheck` passed before the Greptile follow-up commits.
- `pnpm test:run` ran the full stable Vitest path before the Greptile
follow-up commits; it completed with 3 timing-related failures under
parallel load: `codex-local-execute.test.ts`,
`cursor-local-execute.test.ts`, and `environment-service.test.ts`.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/codex-local-execute.test.ts
src/__tests__/cursor-local-execute.test.ts
src/__tests__/environment-service.test.ts` passed on targeted rerun
(`24/24`).
- `pnpm build` passed before the Greptile follow-up commits. Vite
reported existing chunk-size/dynamic-import warnings.
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/secrets-service.test.ts` passed (`26/26`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/aws-secrets-manager-provider.test.ts
src/__tests__/secrets-service.test.ts` passed (`39/39`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
typecheck` passed.
- Captured Storybook screenshots from `ui/storybook-static` for visual
review.
- Latest PR checks on `5ca3a5cf`: `policy`, serialized server suites
1/4-4/4, `Canary Dry Run`, `e2e`, `security/snyk`, and `Greptile Review`
pass; aggregate `verify` is still registering the completed child
checks.
- Greptile review loop continued through the latest requested pass; all
Greptile review threads are resolved and the latest `Greptile Review`
check on `5ca3a5cf` passed with 0 comments added.

## Screenshots

Before: the provider-vault and remote-import surfaces did not exist on
`master`; these are after-state screenshots from the Storybook fixtures.

![Secrets
inventory](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secrets-inventory.png)

![Secret binding
picker](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secret-binding-picker.png)

![Environment editor with
secrets](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/env-editor-with-secrets.png)

## Risks

- Migration risk: this adds new secret provider tables and extends
existing secret rows. The migrations were checked for monotonic ordering
and idempotent guards, but reviewers should still inspect upgrade
behavior carefully.
- Provider risk: AWS support uses direct SigV4 requests. Automated tests
cover signing, request timeouts, vault-config selection, namespace
guardrails, pending-version archival, sanitized provider errors, and
service-level cleanup paths. A real-vault AWS smoke test remains
deployment validation for an operator with AWS credentials rather than
an unverified merge blocker in this local branch.
- UI risk: the Secrets page and import dialog are large new surfaces;
screenshots are included above for reviewer inspection.
- Verification risk: the full local stable test command hit
parallel-load timing failures, although the exact failed files passed
when rerun directly.
- Operational risk: remote import intentionally avoids plaintext reads;
operators must understand that imported external references resolve at
runtime and may fail if AWS permissions change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 coding agent with local shell/tool use in the
Paperclip worktree. Exact context-window size was not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 18:22:17 -05:00

413 lines
13 KiB
Markdown

---
title: Secrets
summary: Secrets CRUD
---
Manage encrypted secrets that agents reference in their environment configuration.
## List Secrets
```
GET /api/companies/{companyId}/secrets
```
Returns secret metadata (not decrypted values).
## Create Secret
```
POST /api/companies/{companyId}/secrets
{
"name": "anthropic-api-key",
"value": "sk-ant-..."
}
```
The value is encrypted at rest. Only the secret ID and metadata are returned.
To link a provider-owned secret without copying the value into Paperclip, create
an external-reference secret:
```json
{
"name": "prod-stripe-key",
"provider": "aws_secrets_manager",
"managedMode": "external_reference",
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:paperclip/prod/stripe",
"providerVersionRef": "version-id-or-label"
}
```
Paperclip stores the provider reference and a non-sensitive fingerprint only.
The value is resolved, when the provider is configured, through the server
runtime path that enforces binding context and records access events.
## Provider Health
```
GET /api/companies/{companyId}/secret-providers/health
```
Returns provider setup diagnostics, warnings, and local backup guidance. Health
responses must not include secret values or provider credentials.
For `aws_secrets_manager`, an unready health response names the missing
non-secret provider environment variables, the AWS SDK default credential source
expected by the server runtime, and the custody rule that AWS bootstrap
credentials must not be stored in Paperclip `company_secrets`.
The equivalent CLI check is:
```sh
pnpm paperclipai secrets doctor --company-id {companyId}
```
## Provider Vaults
Provider vaults are named, company-scoped configurations that route secret
material to one of the supported provider backends. See the
[secrets deploy guide](/deploy/secrets#provider-vaults) for the operator model
and custody rules.
All routes below require board auth and company access. Mutating routes emit
`secret_provider_config.*` activity-log entries. No route in this surface
returns provider credential values; submitting credential-shaped fields in
`config` is rejected at validation time.
### List Vaults
```
GET /api/companies/{companyId}/secret-provider-configs
```
Returns every vault for the company (including disabled rows for audit), each
with id, provider, displayName, status, isDefault, non-sensitive `config`,
latest health snapshot (`healthStatus`, `healthCheckedAt`, `healthMessage`,
`healthDetails`), `disabledAt`, and audit columns.
### Create Vault
```
POST /api/companies/{companyId}/secret-provider-configs
{
"provider": "aws_secrets_manager",
"displayName": "Prod US-East",
"isDefault": true,
"config": {
"region": "us-east-1",
"namespace": "paperclip",
"secretNamePrefix": "paperclip",
"kmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcd-...",
"environmentTag": "production"
}
}
```
Per-provider `config` shapes:
- `local_encrypted`: optional `backupReminderAcknowledged: boolean`.
- `aws_secrets_manager`: required `region`; optional `namespace`,
`secretNamePrefix`, `kmsKeyId`, `ownerTag`, `environmentTag`.
- `gcp_secret_manager` (coming soon): optional `projectId`, `location`,
`namespace`, `secretNamePrefix`.
- `vault` (coming soon): optional origin-only HTTPS `address`, `namespace`,
`mountPath`, `secretPathPrefix`. `address` values with embedded credentials,
paths, query strings, or fragments are rejected.
`status` defaults to `ready` for `local_encrypted` and `aws_secrets_manager`,
and to `coming_soon` for `gcp_secret_manager` and `vault`. Coming-soon and
disabled vaults cannot be marked `isDefault`. Setting `isDefault: true` clears
the previous default for the same provider in the same transaction.
### Get Vault
```
GET /api/secret-provider-configs/{id}
```
### Update Vault
```
PATCH /api/secret-provider-configs/{id}
{
"displayName": "Prod US-East-2",
"config": {
"region": "us-east-2",
"kmsKeyId": "arn:aws:kms:us-east-2:123456789012:key/abcd-..."
}
}
```
`config` is replaced wholesale on update — pass the full provider config
payload, not a partial diff. Status transitions for `gcp_secret_manager` and
`vault` are constrained to `coming_soon` and `disabled` until their runtime
modules ship.
### Disable Vault
```
DELETE /api/secret-provider-configs/{id}
```
Soft-deletes the vault: status flips to `disabled`, `isDefault` clears, and
`disabledAt` is stamped. Disabled vaults remain in `GET` results for audit
purposes but are no longer offered in the secret create/rotate flow.
### Set Default
```
POST /api/secret-provider-configs/{id}/default
```
Marks the target vault as the default for its provider family and clears the
previous default. Returns 422 when the target is `coming_soon` or `disabled`.
### Run Health Check
```
POST /api/secret-provider-configs/{id}/health
```
Runs a provider-specific health probe and persists the result on the vault.
Response shape:
```json
{
"configId": "<uuid>",
"provider": "aws_secrets_manager",
"status": "ready" | "warning" | "error" | "coming_soon" | "disabled",
"message": "Provider vault is ready to handle managed writes",
"details": {
"code": "provider_ready",
"message": "...",
"guidance": ["..."]
},
"checkedAt": "2026-05-06T14:00:00.000Z"
}
```
Health responses never include provider credentials or secret values. For AWS
vaults, `details.guidance` may include missing non-secret env names and the
expected AWS SDK credential source; coming-soon vaults always return
`status: "coming_soon"` with `code: "runtime_locked"` and never call into
provider modules.
### Selecting A Vault When Creating Or Rotating Secrets
`POST /api/companies/{companyId}/secrets` and
`POST /api/secrets/{secretId}/rotate` both accept an optional
`providerConfigId` field that pins the secret to a specific vault. When
omitted (or null), the operation runs through the deployment-level provider
configuration — the same path existing installs already use. The board UI
preselects the company's default vault for the chosen provider before
submitting, so callers should usually send an explicit `providerConfigId`.
Coming-soon and disabled vaults are rejected with a 422; a vault that does not
match the secret's provider is rejected the same way.
```json
POST /api/companies/{companyId}/secrets
{
"name": "prod-stripe-key",
"provider": "aws_secrets_manager",
"providerConfigId": "<vault-uuid>",
"managedMode": "external_reference",
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:paperclip/prod/stripe"
}
```
### Response Redaction Rules
Every route in this surface enforces the same redaction contract:
- Secret values are never returned. The board UI never has a "reveal value"
affordance; resolution happens server-side at runtime under a binding.
- Provider credential values are never accepted, stored, returned, logged, or
echoed in error messages. Submitting credential-shaped fields fails
validation with a non-leaking error.
- Activity log entries record vault id, provider, displayName, status, and
isDefault transitions — never `config` payloads or health detail bodies.
## Remote Import From AWS Secrets Manager
Remote import links existing AWS Secrets Manager entries into Paperclip as
`external_reference` secrets. Import stores provider reference metadata only; it
does not copy the remote secret plaintext into Paperclip.
The routes are board-only and company-scoped. `providerConfigId` must point to
a same-company AWS provider vault with status `ready` or `warning`. Disabled,
coming-soon, non-AWS, and cross-company vaults are rejected. Imported secrets
resolve later through the selected vault, so runtime reads still need
`secretsmanager:GetSecretValue` and any required KMS decrypt permission on the
selected external secret.
### Preview Remote Import Candidates
```
POST /api/companies/{companyId}/secrets/remote-import/preview
{
"providerConfigId": "<aws-vault-uuid>",
"query": "stripe",
"nextToken": "opaque-provider-token",
"pageSize": 50
}
```
`query` is optional and is passed to AWS Secrets Manager inventory filtering.
Treat it as non-secret metadata because AWS may record list request parameters
in CloudTrail. `nextToken` is an opaque AWS cursor; callers must pass it back
unchanged and must not synthesize offsets. `pageSize` is optional, defaults to
50 in the UI, and is capped at 100.
Preview uses AWS `ListSecrets` only. It must not call `GetSecretValue` or
`BatchGetSecretValue`, must not request `SecretString`, and must not require KMS
decrypt. The response contains sanitized metadata for display and conflict
decisions:
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"nextToken": null,
"candidates": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"remoteName": "prod/stripe",
"name": "prod/stripe",
"key": "prod-stripe",
"providerVersionRef": null,
"providerMetadata": {
"createdDate": "2026-05-06T00:00:00.000Z",
"lastChangedDate": "2026-05-06T00:00:00.000Z",
"hasDescription": true,
"hasKmsKey": true,
"tagCount": 3
},
"status": "ready",
"importable": true,
"conflicts": []
}
]
}
```
Candidate statuses:
- `ready`: the row can be selected for import.
- `duplicate`: a Paperclip secret already links the same canonical provider
reference for the same provider vault.
- `conflict`: the row has a name/key collision or provider guardrail failure.
Conflict types are `exact_reference`, `name`, `key`, and
`provider_guardrail`. AWS refs under Paperclip's own managed namespace are
blocked as external references; use the Paperclip-managed secret flow for those
resources instead.
### Import Selected Remote References
```
POST /api/companies/{companyId}/secrets/remote-import
{
"providerConfigId": "<aws-vault-uuid>",
"secrets": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"description": "Stripe key used by production checkout",
"providerVersionRef": null,
"providerMetadata": {
"createdDate": "2026-05-06T00:00:00.000Z"
}
}
]
}
```
The `secrets` array accepts 1-100 rows. Each row may override the suggested
Paperclip `name`, `key`, optional Paperclip `description`,
`providerVersionRef`, and sanitized `providerMetadata`. Blank descriptions are
stored as `null`; AWS provider descriptions are not copied into Paperclip
descriptions. The backend re-checks duplicate refs and name/key conflicts at
submit time; a stale preview does not bypass those checks.
The import response is row-level:
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"importedCount": 1,
"skippedCount": 1,
"errorCount": 0,
"results": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"status": "imported",
"reason": null,
"secretId": "<paperclip-secret-id>",
"conflicts": []
}
]
}
```
Row statuses:
- `imported`: Paperclip created an active `external_reference` secret and one
metadata-only version row.
- `skipped`: the row had an exact-reference duplicate or name/key conflict.
- `error`: the provider rejected the reference or the row failed validation.
Activity logs for preview/import store aggregate counts, provider id, and vault
id only. They must not store remote secret names, ARNs, descriptions, tags,
plaintext values, provider credentials, or raw AWS error blobs.
## Rotate Secret
```
POST /api/secrets/{secretId}/rotate
{
"value": "sk-ant-new-value..."
}
```
Creates a new version of the secret. Agents referencing `"version": "latest"`
automatically get the new value on next heartbeat. Pin to a specific version
when a bad `latest` rollout would affect many agents at once.
## Using Secrets in Agent Config
Reference secrets in agent adapter config instead of inline values:
```json
{
"env": {
"ANTHROPIC_API_KEY": {
"type": "secret_ref",
"secretId": "{secretId}",
"version": "latest"
}
}
}
```
The server resolves and decrypts secret references at runtime, injecting the
real value into the agent process environment. Paperclip's custody guarantees
end at injection: the agent process can read, log, or forward the value, so
treat any secret bound to an agent as exposed to that agent. See the custody
boundaries note in the [secrets deploy guide](/deploy/secrets#custody-boundaries).
## Portability
Company export/import APIs represent agent and project environment requirements
as declarations in the package manifest. Exports omit secret values, secret IDs,
provider references, and encrypted provider material. Use:
```sh
pnpm paperclipai secrets declarations --company-id {companyId}
```
to inspect the declarations that an export would emit before moving a package.