gb_flea/paperclip
1
0
Fork 0
forked from farhoodlabs/paperclip
Code Pull Requests Activity

[codex] Add routine env secrets support (#6212)

Browse Source 
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - Scheduled routines are the control-plane path for recurring agent
work.
> - Routines already had dispatch/history, but their runtime environment
did not carry routine-owned secret bindings through execution.
> - Operators need routine-specific secrets that can override
project/agent env without exposing secret values in history, logs, or
access events.
> - This pull request adds the routine env runtime contract, wires it
into execution, and makes the routine UI/history surfaces show safe
secret metadata.
> - The benefit is that routine executions can use scoped secret refs
predictably while preserving company boundaries and auditability.

## What Changed

- Added routine env persistence/runtime support, including
`routines.env`, `routine_runs.routine_revision_id`, revision snapshots,
and idempotent migration `0086_routine_env_runtime_contract`.
- Resolved routine env during heartbeat adapter config assembly with
precedence `agent < project < routine` and secret access events recorded
against the routine consumer.
- Added secret binding synchronization for routine create/update/restore
flows and guarded cross-company, missing, disabled, and deleted secret
cases.
- Added a Secrets tab to routine detail, env/secret history diff
rendering, and Storybook coverage for the new UI states.
- Added server/UI regression tests, including an embedded-Postgres QA
path for routine secret execution and restore behavior.
- Updated implementation/database docs for routine env and
secret-binding behavior.

## Verification

- `pnpm install --frozen-lockfile` after rebasing onto
`public-gh/master` to refresh workspace links for the newly-added
upstream Grok adapter package.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-project-env.test.ts
server/src/__tests__/routines-service.test.ts
server/src/__tests__/secrets-service.test.ts
server/src/__tests__/qa-routine-secrets-e2e.test.ts
ui/src/components/RoutineHistoryTab.test.tsx` passed: 5 files, 92 tests.
- `pnpm -r typecheck` passed across the workspace.
- `pnpm build` passed. Vite emitted the existing
large-chunk/dynamic-import warnings.
- UI screenshots were captured locally during QA in
`artifacts/pap-9521/` and `artifacts/pap-9522/`; generated screenshots
are not committed to avoid adding binary artifacts to the repo.

## Risks

- Migration risk is limited by `IF NOT EXISTS` guards for the new
columns, FK, and index, and the migration is ordered as `0086`
immediately after upstream `0085`.
- Runtime behavior changes env precedence for routine executions by
adding routine env as the highest-precedence layer; tests cover
agent/project/routine precedence.
- Secret handling is security-sensitive; tests cover value-free
manifests/events/errors, disabled/missing/deleted secrets, and
cross-company rejection.
- UI history now renders routine env/secret diffs; tests and Storybook
stories cover the main rendering paths.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex coding agent based on GPT-5, with shell/tool use and
medium reasoning effort.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
2026-05-17 16:30:34 -05:00
committed by GitHub
parent 3e6610fb93
commit 705c1b8d81
20 changed files with 1736 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/src/__tests__/heartbeat-project-env.test.ts
+112 -20
View File
@@ -7,7 +7,7 @@ import {
} from "../services/heartbeat.ts";
describe("resolveExecutionRunAdapterConfig", () => {
it("overlays project env on top of agent env and unions secret keys", async () => {
it("overlays project and routine env on top of agent env and unions secret keys", async () => {
const resolveAdapterConfigForRuntime = vi.fn().mockResolvedValue({
config: {
env: {
@@ -29,29 +29,51 @@ describe("resolveExecutionRunAdapterConfig", () => {
},
],
});
const resolveEnvBindings = vi.fn().mockResolvedValue({
env: {
SHARED_KEY: "project",
PROJECT_ONLY: "project-only",
},
secretKeys: new Set(["PROJECT_SECRET"]),
manifest: [
{
configPath: "env.PROJECT_SECRET",
envKey: "PROJECT_SECRET",
secretId: "secret-project",
secretKey: "project-secret",
version: 1,
provider: "local_encrypted",
outcome: "success",
const resolveEnvBindings = vi
.fn()
.mockResolvedValueOnce({
env: {
SHARED_KEY: "project",
PROJECT_ONLY: "project-only",
},
],
});
secretKeys: new Set(["PROJECT_SECRET"]),
manifest: [
{
configPath: "env.PROJECT_SECRET",
envKey: "PROJECT_SECRET",
secretId: "secret-project",
secretKey: "project-secret",
version: 1,
provider: "local_encrypted",
outcome: "success",
},
],
})
.mockResolvedValueOnce({
env: {
SHARED_KEY: "routine",
ROUTINE_ONLY: "routine-only",
},
secretKeys: new Set(["ROUTINE_SECRET"]),
manifest: [
{
configPath: "env.ROUTINE_SECRET",
envKey: "ROUTINE_SECRET",
secretId: "secret-routine",
secretKey: "routine-secret",
version: 1,
provider: "local_encrypted",
outcome: "success",
},
],
});
const result = await resolveExecutionRunAdapterConfig({
companyId: "company-1",
executionRunConfig: { env: { SHARED_KEY: "agent" } },
projectEnv: { SHARED_KEY: "project" },
routineEnv: { SHARED_KEY: "routine" },
routineId: "routine-1",
secretsSvc: {
resolveAdapterConfigForRuntime,
resolveEnvBindings,
@@ -61,18 +83,88 @@ describe("resolveExecutionRunAdapterConfig", () => {
expect(result.resolvedConfig).toMatchObject({
other: "value",
env: {
SHARED_KEY: "project",
SHARED_KEY: "routine",
AGENT_ONLY: "agent-only",
PROJECT_ONLY: "project-only",
ROUTINE_ONLY: "routine-only",
},
});
expect(Array.from(result.secretKeys).sort()).toEqual(["AGENT_SECRET", "PROJECT_SECRET"]);
expect(Array.from(result.secretKeys).sort()).toEqual(["AGENT_SECRET", "PROJECT_SECRET", "ROUTINE_SECRET"]);
expect(result.secretManifest.map((entry) => entry.secretId).sort()).toEqual([
"secret-agent",
"secret-project",
"secret-routine",
]);
expect(JSON.stringify(result.secretManifest)).not.toContain("agent-only");
expect(JSON.stringify(result.secretManifest)).not.toContain("project-only");
expect(JSON.stringify(result.secretManifest)).not.toContain("routine-only");
expect(resolveEnvBindings.mock.calls[1]?.[2]).toMatchObject({
consumerType: "routine",
consumerId: "routine-1",
});
});
it("drops Paperclip runtime-owned env before resolving agent, project, and routine overlays", async () => {
const resolveAdapterConfigForRuntime = vi.fn(async (_companyId, config: Record<string, unknown>) => ({
config: {
...config,
env: { ...(config.env as Record<string, unknown>) },
},
secretKeys: new Set<string>(),
manifest: [],
}));
const resolveEnvBindings = vi.fn(async (_companyId, env: Record<string, unknown>) => ({
env: Object.fromEntries(
Object.entries(env).filter((entry): entry is [string, string] => typeof entry[1] === "string"),
),
secretKeys: new Set<string>(),
manifest: [],
}));
const result = await resolveExecutionRunAdapterConfig({
companyId: "company-1",
agentId: "agent-1",
executionRunConfig: {
env: {
PAPERCLIP_API_KEY: { type: "secret_ref", secretId: "secret-api-key", version: "latest" },
PAPERCLIP_AGENT_ID: "spoofed-agent",
AGENT_ONLY: "agent-only",
},
},
projectEnv: {
PAPERCLIP_API_KEY: "project-api-key",
PAPERCLIP_COMPANY_ID: "spoofed-company",
PROJECT_ONLY: "project-only",
},
routineEnv: {
PAPERCLIP_API_KEY: "routine-api-key",
PAPERCLIP_RUN_ID: "spoofed-run",
ROUTINE_ONLY: "routine-only",
},
routineId: "routine-1",
secretsSvc: {
resolveAdapterConfigForRuntime,
resolveEnvBindings,
} as any,
});
expect(resolveAdapterConfigForRuntime.mock.calls[0]?.[1]).toEqual({
env: {
AGENT_ONLY: "agent-only",
},
});
expect(resolveEnvBindings.mock.calls[0]?.[1]).toEqual({
PROJECT_ONLY: "project-only",
});
expect(resolveEnvBindings.mock.calls[1]?.[1]).toEqual({
ROUTINE_ONLY: "routine-only",
});
expect(result.resolvedConfig.env).toEqual({
AGENT_ONLY: "agent-only",
PROJECT_ONLY: "project-only",
ROUTINE_ONLY: "routine-only",
});
expect(JSON.stringify(result.resolvedConfig.env)).not.toContain("PAPERCLIP_");
});
it("skips project env resolution when the project has no bindings", async () => {
server/src/__tests__/qa-routine-secrets-e2e.test.ts
+458
View File
@@ -0,0 +1,458 @@
// QA validation for [PAP-9522](/PAP/issues/PAP-9522). Drives the routine-secret
// chain end-to-end against a real embedded Postgres:
//
// 1. Routine env reaches the heartbeat runtime via `resolveExecutionRunAdapterConfig`
// using `secretsSvc.resolveEnvBindings` with a `consumerType: "routine"` context,
// even when the executing agent has zero direct bindings for that secret.
// 2. Precedence: agent < project < routine for a shared key.
// 3. `secret_access_events` records routine consumption but NEVER the resolved value.
// 4. Restoring an older revision re-syncs `company_secret_bindings` to the snapshot env.
// 5. Legacy fallback: a routine_run with null `routine_revision_id` still resolves
// the routine's current env (matches the explicit acceptance criterion).
// 6. Disabled / missing / cross-company secret bindings fail clearly without
// echoing the value.
import { randomUUID } from "node:crypto";
import { mkdirSync, rmSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { eq, and } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
agents,
companies,
companySecretBindings,
companySecrets,
companySecretVersions,
createDb,
projects,
routineRuns,
routines,
secretAccessEvents,
} from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { routineService } from "../services/routines.ts";
import { secretService } from "../services/secrets.ts";
import { resolveExecutionRunAdapterConfig } from "../services/heartbeat.ts";
const support = await getEmbeddedPostgresTestSupport();
const describeEmbedded = support.supported ? describe : describe.skip;
if (!support.supported) {
console.warn(`Skipping QA e2e on this host: ${support.reason ?? "embedded pg unsupported"}`);
}
describeEmbedded("PAP-9522 QA: routine secrets end-to-end", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
const secretsTmpDir = path.join(os.tmpdir(), `paperclip-qa-routine-secrets-${randomUUID()}`);
const previousKeyFile = process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
beforeAll(async () => {
mkdirSync(secretsTmpDir, { recursive: true });
process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = path.join(secretsTmpDir, "master.key");
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-qa-routine-secrets-");
db = createDb(tempDb.connectionString);
}, 30_000);
afterEach(async () => {
await db.delete(secretAccessEvents);
await db.delete(companySecretBindings);
await db.delete(routineRuns);
await db.delete(routines);
await db.delete(companySecretVersions);
await db.delete(companySecrets);
await db.delete(projects);
await db.delete(agents);
await db.delete(companies);
});
afterAll(async () => {
await tempDb?.cleanup();
if (previousKeyFile === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE;
else process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = previousKeyFile;
rmSync(secretsTmpDir, { recursive: true, force: true });
});
async function seed() {
const companyId = randomUUID();
const executorAgentId = randomUUID();
const issuePrefix = `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`;
await db.insert(companies).values({
id: companyId,
name: "QA Co",
issuePrefix,
requireBoardApprovalForNewAgents: false,
});
// Note: executor agent has NO secret bindings of its own — this is the
// whole point of routine env (the secret rides with the routine, not the agent).
await db.insert(agents).values({
id: executorAgentId,
companyId,
name: "Executor",
role: "engineer",
status: "active",
adapterType: "codex_local",
adapterConfig: { env: {} },
runtimeConfig: {},
permissions: {},
});
return { companyId, executorAgentId };
}
const ROUTINE_VALUE = "super-sekret-routine-value";
const PROJECT_VALUE = "project-overlay-value";
const AGENT_VALUE = "agent-base-value";
it("resolves routine env for an executing agent that has no direct binding, with routine winning precedence and zero value in access events", async () => {
const { companyId, executorAgentId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const secret = await secrets.create(companyId, {
name: `routine-api-${randomUUID()}`,
provider: "local_encrypted",
value: ROUTINE_VALUE,
});
const routine = await routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "qa routine",
description: null,
assigneeAgentId: executorAgentId,
priority: "medium",
status: "active",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
env: {
SHARED: { type: "plain", value: "routine-overrides" },
ROUTINE_API_KEY: { type: "secret_ref", secretId: secret.id, version: "latest" },
},
},
{},
);
// Verify binding is owned by the routine, not the executing agent.
const bindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(bindings).toMatchObject([
{ targetType: "routine", secretId: secret.id, configPath: "env.ROUTINE_API_KEY" },
]);
// Drive the real heartbeat resolution path with the routine env.
// issueId/heartbeatRunId left null because secret_access_events has FK
// constraints on both — populating them would require seeding issue and
// heartbeat_run rows just for FK validity. The routine consumer fields are
// what this test cares about.
const result = await resolveExecutionRunAdapterConfig({
companyId,
agentId: executorAgentId,
issueId: null,
heartbeatRunId: null,
projectId: null,
routineId: routine.id,
executionRunConfig: { env: { SHARED: AGENT_VALUE, AGENT_ONLY: AGENT_VALUE } },
projectEnv: { SHARED: { type: "plain", value: PROJECT_VALUE } },
routineEnv: routine.env,
secretsSvc: secrets,
});
expect(result.resolvedConfig.env).toMatchObject({
AGENT_ONLY: AGENT_VALUE,
SHARED: "routine-overrides", // routine beats project beats agent
ROUTINE_API_KEY: ROUTINE_VALUE,
});
expect(result.secretKeys.has("ROUTINE_API_KEY")).toBe(true);
expect(result.secretManifest.some((m) => m.envKey === "ROUTINE_API_KEY")).toBe(true);
// Manifest must not echo the resolved value.
expect(JSON.stringify(result.secretManifest)).not.toContain(ROUTINE_VALUE);
const events = await db
.select()
.from(secretAccessEvents)
.where(eq(secretAccessEvents.secretId, secret.id));
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({
consumerType: "routine",
consumerId: routine.id,
actorType: "agent",
actorId: executorAgentId,
configPath: "env.ROUTINE_API_KEY",
outcome: "success",
});
// No serialized field of the access event row can contain the secret value.
expect(JSON.stringify(events[0])).not.toContain(ROUTINE_VALUE);
});
it("rejects routine env that references a secret from a different company", async () => {
const { companyId } = await seed();
const { companyId: otherCompanyId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const foreignSecret = await secrets.create(otherCompanyId, {
name: `foreign-${randomUUID()}`,
provider: "local_encrypted",
value: "cross-company-leak-bait",
});
await expect(
routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "cross company",
description: null,
assigneeAgentId: null,
priority: "medium",
status: "paused",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
env: {
BAD: { type: "secret_ref", secretId: foreignSecret.id, version: "latest" },
},
},
{},
),
).rejects.toThrow(/same company/i);
});
it("surfaces a clear, value-free error when a routine secret is missing/deleted at resolution time", async () => {
const { companyId, executorAgentId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const secret = await secrets.create(companyId, {
name: `to-be-deleted-${randomUUID()}`,
provider: "local_encrypted",
value: "doomed-secret-value",
});
const routine = await routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "doomed routine",
description: null,
assigneeAgentId: executorAgentId,
priority: "medium",
status: "active",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
env: {
DOOMED: { type: "secret_ref", secretId: secret.id, version: "latest" },
},
},
{},
);
// Hard delete the secret out from under the routine; the routine env now
// points at a vanished id.
await secrets.remove(secret.id);
let caught: unknown = null;
try {
await resolveExecutionRunAdapterConfig({
companyId,
agentId: executorAgentId,
issueId: null,
heartbeatRunId: null,
projectId: null,
routineId: routine.id,
executionRunConfig: { env: {} },
projectEnv: null,
routineEnv: routine.env,
secretsSvc: secrets,
});
} catch (error) {
caught = error;
}
expect(caught).toBeTruthy();
const message = String((caught as Error)?.message ?? caught);
expect(message).not.toContain("doomed-secret-value");
});
it("restoring an older revision re-syncs company_secret_bindings to the snapshot env", async () => {
const { companyId, executorAgentId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const secretA = await secrets.create(companyId, {
name: `a-${randomUUID()}`,
provider: "local_encrypted",
value: "val-a",
});
const secretB = await secrets.create(companyId, {
name: `b-${randomUUID()}`,
provider: "local_encrypted",
value: "val-b",
});
const routine = await routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "restore routine",
description: null,
assigneeAgentId: executorAgentId,
priority: "medium",
status: "active",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
env: {
ALPHA: { type: "secret_ref", secretId: secretA.id, version: "latest" },
},
},
{},
);
const rev1Id = routine.latestRevisionId!;
await routines.update(
routine.id,
{
env: {
ALPHA: { type: "secret_ref", secretId: secretA.id, version: "latest" },
BETA: { type: "secret_ref", secretId: secretB.id, version: "latest" },
},
},
{},
);
let bindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(bindings.map((b) => b.configPath).sort()).toEqual(["env.ALPHA", "env.BETA"]);
await routines.restoreRevision(routine.id, rev1Id, {});
bindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(bindings.map((b) => b.configPath)).toEqual(["env.ALPHA"]);
expect(bindings[0]?.secretId).toBe(secretA.id);
});
it("legacy run with null routine_revision_id falls back to the routine's current env (still resolves)", async () => {
const { companyId, executorAgentId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const secret = await secrets.create(companyId, {
name: `legacy-${randomUUID()}`,
provider: "local_encrypted",
value: "legacy-value",
});
const routine = await routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "legacy routine",
description: null,
assigneeAgentId: executorAgentId,
priority: "medium",
status: "active",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
env: {
LEGACY: { type: "secret_ref", secretId: secret.id, version: "latest" },
},
},
{},
);
// Simulate an old routine_run row (predating the migration) with no
// routine_revision_id. The fallback path in `getRoutineEnvForExecutionIssue`
// should still resolve to the routine's current env. Here we exercise the
// resolution layer directly with routine.env to mirror that behavior.
await db.insert(routineRuns).values({
id: randomUUID(),
companyId,
routineId: routine.id,
triggerId: null,
source: "manual",
status: "issue_created",
triggeredAt: new Date(),
completedAt: new Date(),
routineRevisionId: null,
});
const result = await resolveExecutionRunAdapterConfig({
companyId,
agentId: executorAgentId,
issueId: null,
heartbeatRunId: null,
projectId: null,
routineId: routine.id,
executionRunConfig: { env: {} },
projectEnv: null,
routineEnv: routine.env,
secretsSvc: secrets,
});
expect(result.resolvedConfig.env).toMatchObject({ LEGACY: "legacy-value" });
});
it("routines created with null env (no Secrets tab interaction) still resolve normally with empty env", async () => {
const { companyId, executorAgentId } = await seed();
const secrets = secretService(db);
const routines = routineService(db, { heartbeat: { wakeup: async () => null } });
const routine = await routines.create(
companyId,
{
projectId: null,
goalId: null,
parentIssueId: null,
title: "null env routine",
description: null,
assigneeAgentId: executorAgentId,
priority: "medium",
status: "active",
concurrencyPolicy: "coalesce_if_active",
catchUpPolicy: "skip_missed",
},
{},
);
expect(routine.env ?? null).toBeNull();
const bindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(bindings).toHaveLength(0);
const result = await resolveExecutionRunAdapterConfig({
companyId,
agentId: executorAgentId,
issueId: null,
heartbeatRunId: null,
projectId: null,
routineId: routine.id,
executionRunConfig: { env: { AGENT_ONLY: "agent" } },
projectEnv: null,
routineEnv: null,
secretsSvc: secrets,
});
expect(result.resolvedConfig.env).toEqual({ AGENT_ONLY: "agent" });
expect(result.secretKeys.size).toBe(0);
});
});
server/src/__tests__/routines-service.test.ts
+88
View File
@@ -5,6 +5,7 @@ import {
activityLog,
agents,
companies,
companySecretBindings,
companySecrets,
companySecretVersions,
createDb,
@@ -19,6 +20,7 @@ import {
routineRuns,
routines,
routineTriggers,
secretAccessEvents,
} from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
@@ -28,6 +30,7 @@ import { issueService } from "../services/issues.ts";
import { instanceSettingsService } from "../services/instance-settings.ts";
import * as providerRegistry from "../secrets/provider-registry.ts";
import { routineService } from "../services/routines.ts";
import { secretService } from "../services/secrets.ts";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
@@ -57,6 +60,8 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => {
await db.delete(activityLog);
await db.delete(issueInboxArchives);
await db.delete(issueReadStates);
await db.delete(secretAccessEvents);
await db.delete(companySecretBindings);
await db.delete(routineRuns);
await db.delete(routineTriggers);
await db.delete(routines);
@@ -331,6 +336,89 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => {
expect(revisions[1]?.snapshot.routine.description).toBe("Run the frog routine");
});
it("stores routine env in revisions, syncs routine secret bindings, and stamps runs with the dispatch revision", async () => {
const { agentId, companyId, projectId, svc } = await seedFixture();
const secrets = secretService(db);
const secret = await secrets.create(companyId, {
name: `routine-api-${randomUUID()}`,
provider: "local_encrypted",
value: "secret-value",
});
const routine = await svc.create(
companyId,
{
projectId,
goalId: null,
parentIssueId: null,
title: "secret routine",
description: null,
assigneeAgentId: agentId,
priority: "medium",
status: "active",
concurrencyPolicy: "always_enqueue",
catchUpPolicy: "skip_missed",
env: {
ROUTINE_API_KEY: { type: "secret_ref", secretId: secret.id, version: "latest" },
ROUTINE_PLAIN: { type: "plain", value: "plain-value" },
},
},
{},
);
const bindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(bindings).toMatchObject([
{
companyId,
secretId: secret.id,
targetType: "routine",
configPath: "env.ROUTINE_API_KEY",
},
]);
const [initialRevision] = await svc.listRevisions(routine.id);
expect(initialRevision?.snapshot.routine.env).toEqual(routine.env);
await db.delete(companySecretBindings).where(eq(companySecretBindings.targetId, routine.id));
const repaired = await svc.update(routine.id, { env: routine.env }, {});
expect(repaired).not.toBeNull();
const repairedBindings = await db
.select()
.from(companySecretBindings)
.where(eq(companySecretBindings.targetId, routine.id));
expect(repairedBindings).toMatchObject([
{
companyId,
secretId: secret.id,
targetType: "routine",
configPath: "env.ROUTINE_API_KEY",
},
]);
const currentRoutine = repaired ?? routine;
const runBefore = await svc.runRoutine(routine.id, { source: "manual" });
expect(runBefore.routineRevisionId).toBe(currentRoutine.latestRevisionId);
const updated = await svc.update(
routine.id,
{
env: {
ROUTINE_API_KEY: { type: "secret_ref", secretId: secret.id, version: "latest" },
ROUTINE_PLAIN: { type: "plain", value: "changed" },
},
},
{},
);
expect(updated?.latestRevisionNumber).toBe(currentRoutine.latestRevisionNumber + 1);
const runAfter = await svc.runRoutine(routine.id, { source: "manual" });
expect(runAfter.routineRevisionId).toBe(updated?.latestRevisionId);
expect(runAfter.dispatchFingerprint).not.toBe(runBefore.dispatchFingerprint);
});
it("rejects stale routine baseRevisionId updates", async () => {
const { routine, svc } = await seedFixture();
const updated = await svc.update(routine.id, { description: "new description" }, {});
server/src/__tests__/secrets-service.test.ts
+110
View File
@@ -205,6 +205,116 @@ describeEmbeddedPostgres("secretService", () => {
expect(JSON.stringify(events)).not.toContain("runtime-secret");
});
it("resolves routine env secret refs through routine bindings and records value-free access metadata", async () => {
const companyId = await seedCompany();
const svc = secretService(db);
const secret = await svc.create(companyId, {
name: `routine-secret-${randomUUID()}`,
provider: "local_encrypted",
value: "routine-super-secret",
});
const env = {
ROUTINE_API_KEY: { type: "secret_ref" as const, secretId: secret.id, version: "latest" as const },
};
await svc.syncEnvBindingsForTarget(companyId, { targetType: "routine", targetId: "routine-1" }, env);
const resolved = await svc.resolveEnvBindings(companyId, env, {
consumerType: "routine",
consumerId: "routine-1",
actorType: "agent",
actorId: "agent-1",
});
expect(resolved.env.ROUTINE_API_KEY).toBe("routine-super-secret");
expect(resolved.manifest).toEqual([
expect.objectContaining({
configPath: "env.ROUTINE_API_KEY",
envKey: "ROUTINE_API_KEY",
secretId: secret.id,
outcome: "success",
}),
]);
const events = await svc.listAccessEvents(companyId, secret.id);
expect(events).toHaveLength(1);
expect(events[0]).toMatchObject({
companyId,
secretId: secret.id,
consumerType: "routine",
consumerId: "routine-1",
configPath: "env.ROUTINE_API_KEY",
actorType: "agent",
actorId: "agent-1",
outcome: "success",
});
expect(JSON.stringify(events)).not.toContain("routine-super-secret");
});
it("records stable redacted failure codes for routine env secret resolution", async () => {
const companyId = await seedCompany();
const svc = secretService(db);
const secret = await svc.create(companyId, {
name: `routine-failure-codes-${randomUUID()}`,
provider: "local_encrypted",
value: "routine-super-secret",
});
const env = {
ROUTINE_API_KEY: { type: "secret_ref" as const, secretId: secret.id, version: "latest" as const },
};
const context = {
consumerType: "routine" as const,
consumerId: "routine-1",
actorType: "agent" as const,
actorId: "agent-1",
};
await svc.syncEnvBindingsForTarget(companyId, { targetType: "routine", targetId: "routine-1" }, env);
await expect(
svc.resolveEnvBindings(companyId, env, { ...context, consumerId: "routine-2" }),
).rejects.toThrow(/not bound/i);
await db.update(companySecrets).set({ status: "disabled" }).where(eq(companySecrets.id, secret.id));
await expect(svc.resolveEnvBindings(companyId, env, context)).rejects.toThrow(/not active/i);
await db.update(companySecrets).set({ status: "active" }).where(eq(companySecrets.id, secret.id));
await expect(
svc.resolveSecretValue(companyId, secret.id, 999, {
...context,
configPath: "env.ROUTINE_API_KEY",
}),
).rejects.toThrow(/version not found/i);
await db
.update(companySecretVersions)
.set({ status: "disabled" })
.where(eq(companySecretVersions.secretId, secret.id));
await expect(svc.resolveEnvBindings(companyId, env, context)).rejects.toThrow(/version is not active/i);
await db
.update(companySecretVersions)
.set({ status: "current" })
.where(eq(companySecretVersions.secretId, secret.id));
vi.spyOn(localEncryptedProvider, "resolveVersion").mockRejectedValueOnce(
new Error("provider leaked value routine-super-secret"),
);
await expect(svc.resolveEnvBindings(companyId, env, context)).rejects.toThrow(/provider leaked value/i);
await db.update(companySecrets).set({ status: "deleted" }).where(eq(companySecrets.id, secret.id));
await expect(svc.resolveEnvBindings(companyId, env, context)).rejects.toThrow(/not found/i);
const events = await svc.listAccessEvents(companyId, secret.id);
expect(events.map((event) => event.errorCode).sort()).toEqual([
"binding_missing",
"provider_error",
"secret_deleted",
"secret_inactive",
"version_inactive",
"version_missing",
]);
expect(JSON.stringify(events)).not.toContain("routine-super-secret");
expect(JSON.stringify(events)).not.toContain("provider leaked value");
});
it("scopes env binding sync deletes to the env path prefix", async () => {
const companyId = await seedCompany();
const svc = secretService(db);
server/src/services/heartbeat.ts
+120 -4
View File
@@ -18,6 +18,7 @@ import {
type IssueExecutionMonitorPolicy,
type IssueExecutionMonitorRecoveryPolicy,
type ModelProfileKey,
type RoutineRevisionSnapshotV1,
type RunLivenessState,
} from "@paperclipai/shared";
import {
@@ -40,6 +41,9 @@ import {
issueWorkProducts,
projects,
projectWorkspaces,
routineRevisions,
routineRuns,
routines,
workspaceOperations,
} from "@paperclipai/db";
import { conflict, HttpError, notFound } from "../errors.js";
@@ -327,19 +331,44 @@ type RuntimeConfigSecretResolver = Pick<
"resolveAdapterConfigForRuntime" | "resolveEnvBindings"
>;
function isPaperclipRuntimeEnvKey(key: string) {
return key.startsWith("PAPERCLIP_");
}
function stripPaperclipRuntimeEnvBindings(envValue: unknown): Record<string, unknown> | null {
const record = parseObject(envValue);
const filtered = Object.fromEntries(
Object.entries(record).filter(([key]) => !isPaperclipRuntimeEnvKey(key)),
);
return Object.keys(filtered).length > 0 ? filtered : null;
}
function stripPaperclipRuntimeEnvFromAdapterConfig(config: Record<string, unknown>): Record<string, unknown> {
if (!Object.prototype.hasOwnProperty.call(config, "env")) return config;
return {
...config,
env: stripPaperclipRuntimeEnvBindings(config.env) ?? {},
};
}
export async function resolveExecutionRunAdapterConfig(input: {
companyId: string;
agentId?: string | null;
issueId?: string | null;
heartbeatRunId?: string | null;
projectId?: string | null;
routineId?: string | null;
executionRunConfig: Record<string, unknown>;
projectEnv: unknown;
routineEnv?: unknown;
secretsSvc: RuntimeConfigSecretResolver;
}) {
const executionRunConfig = stripPaperclipRuntimeEnvFromAdapterConfig(input.executionRunConfig);
const projectEnv = stripPaperclipRuntimeEnvBindings(input.projectEnv);
const routineEnv = stripPaperclipRuntimeEnvBindings(input.routineEnv);
const { config: resolvedConfig, secretKeys, manifest } = await input.secretsSvc.resolveAdapterConfigForRuntime(
input.companyId,
input.executionRunConfig,
executionRunConfig,
input.agentId
? {
consumerType: "agent",
@@ -351,10 +380,10 @@ export async function resolveExecutionRunAdapterConfig(input: {
}
: undefined,
);
const projectEnvResolution = input.projectEnv
const projectEnvResolution = projectEnv
? await input.secretsSvc.resolveEnvBindings(
input.companyId,
input.projectEnv,
projectEnv,
input.projectId
? {
consumerType: "project",
@@ -376,10 +405,39 @@ export async function resolveExecutionRunAdapterConfig(input: {
secretKeys.add(key);
}
}
const routineEnvResolution = routineEnv
? await input.secretsSvc.resolveEnvBindings(
input.companyId,
routineEnv,
input.routineId
? {
consumerType: "routine",
consumerId: input.routineId,
actorType: "agent",
actorId: input.agentId ?? null,
issueId: input.issueId ?? null,
heartbeatRunId: input.heartbeatRunId ?? null,
}
: undefined,
)
: { env: {}, secretKeys: new Set<string>(), manifest: [] };
if (Object.keys(routineEnvResolution.env).length > 0) {
resolvedConfig.env = {
...parseObject(resolvedConfig.env),
...routineEnvResolution.env,
};
for (const key of routineEnvResolution.secretKeys) {
secretKeys.add(key);
}
}
return {
resolvedConfig,
secretKeys,
secretManifest: [...(manifest ?? []), ...(projectEnvResolution.manifest ?? [])],
secretManifest: [
...(manifest ?? []),
...(projectEnvResolution.manifest ?? []),
...(routineEnvResolution.manifest ?? []),
],
};
}
@@ -2433,12 +2491,67 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
assigneeAgentId: issues.assigneeAgentId,
assigneeAdapterOverrides: issues.assigneeAdapterOverrides,
executionWorkspaceSettings: issues.executionWorkspaceSettings,
originKind: issues.originKind,
originId: issues.originId,
originRunId: issues.originRunId,
})
.from(issues)
.where(and(eq(issues.id, issueId), eq(issues.companyId, companyId)))
.then((rows) => rows[0] ?? null);
}
async function getRoutineEnvForExecutionIssue(
companyId: string,
issueContext: Awaited<ReturnType<typeof getIssueExecutionContext>> | null,
) {
if (!issueContext || issueContext.originKind !== "routine_execution" || !issueContext.originId) {
return { routineId: null, env: null };
}
const routineRun = issueContext.originRunId
? await db
.select({
routineRevisionId: routineRuns.routineRevisionId,
})
.from(routineRuns)
.where(
and(
eq(routineRuns.id, issueContext.originRunId),
eq(routineRuns.companyId, companyId),
eq(routineRuns.routineId, issueContext.originId),
),
)
.then((rows) => rows[0] ?? null)
: null;
if (routineRun?.routineRevisionId) {
const revision = await db
.select({
snapshot: routineRevisions.snapshot,
})
.from(routineRevisions)
.where(
and(
eq(routineRevisions.id, routineRun.routineRevisionId),
eq(routineRevisions.companyId, companyId),
eq(routineRevisions.routineId, issueContext.originId),
),
)
.then((rows) => rows[0] ?? null);
const snapshot = revision?.snapshot as RoutineRevisionSnapshotV1 | undefined;
if (snapshot?.version === 1) {
return { routineId: issueContext.originId, env: snapshot.routine.env ?? null };
}
}
const routine = await db
.select({ env: routines.env })
.from(routines)
.where(and(eq(routines.id, issueContext.originId), eq(routines.companyId, companyId)))
.then((rows) => rows[0] ?? null);
return { routineId: issueContext.originId, env: routine?.env ?? null };
}
async function getRuntimeState(agentId: string) {
return db
.select()
@@ -6870,6 +6983,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
.where(and(eq(projects.id, executionProjectId), eq(projects.companyId, agent.companyId)))
.then((rows) => rows[0] ?? null)
: null;
const routineEnvContext = await getRoutineEnvForExecutionIssue(agent.companyId, issueContext);
const projectExecutionWorkspacePolicy = gateProjectExecutionWorkspacePolicy(
parseProjectExecutionWorkspacePolicy(projectContext?.executionWorkspacePolicy),
isolatedWorkspacesEnabled,
@@ -7074,8 +7188,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
issueId,
heartbeatRunId: run.id,
projectId: projectContext?.id ?? null,
routineId: routineEnvContext.routineId,
executionRunConfig,
projectEnv: projectContext?.env ?? null,
routineEnv: routineEnvContext.env,
secretsSvc,
});
if (secretManifest.length > 0) {
server/src/services/routines.ts
+75 -5
View File
@@ -366,6 +366,8 @@ function createRoutineDispatchFingerprint(input: {
payload: Record<string, unknown> | null;
projectId: string | null;
assigneeAgentId: string | null;
routineRevisionId: string | null;
routineEnvFingerprint: string | null;
executionWorkspaceId?: string | null;
executionWorkspacePreference?: string | null;
executionWorkspaceSettings?: Record<string, unknown> | null;
@@ -376,6 +378,11 @@ function createRoutineDispatchFingerprint(input: {
return crypto.createHash("sha256").update(canonical).digest("hex");
}
function createRoutineEnvFingerprint(env: unknown) {
const canonical = JSON.stringify(normalizeRoutineDispatchFingerprintValue(env ?? null));
return crypto.createHash("sha256").update(canonical).digest("hex");
}
function readManagedRoutineIssueTemplate(defaultsJson: Record<string, unknown> | null | undefined) {
const value = defaultsJson?.issueTemplate;
if (!isPlainRecord(value)) return null;
@@ -406,6 +413,7 @@ function routineRevisionSnapshotRoutine(routine: RoutineRow): RoutineRevisionSna
concurrencyPolicy: routine.concurrencyPolicy as RoutineRevisionSnapshotV1["routine"]["concurrencyPolicy"],
catchUpPolicy: routine.catchUpPolicy as RoutineRevisionSnapshotV1["routine"]["catchUpPolicy"],
variables: routine.variables ?? [],
env: routine.env ?? null,
};
}
@@ -686,6 +694,7 @@ export function routineService(
idempotencyKey: routineRuns.idempotencyKey,
triggerPayload: routineRuns.triggerPayload,
dispatchFingerprint: routineRuns.dispatchFingerprint,
routineRevisionId: routineRuns.routineRevisionId,
linkedIssueId: routineRuns.linkedIssueId,
coalescedIntoRunId: routineRuns.coalescedIntoRunId,
failureReason: routineRuns.failureReason,
@@ -719,6 +728,7 @@ export function routineService(
idempotencyKey: row.idempotencyKey,
triggerPayload: row.triggerPayload as Record<string, unknown> | null,
dispatchFingerprint: row.dispatchFingerprint,
routineRevisionId: row.routineRevisionId,
linkedIssueId: row.linkedIssueId,
coalescedIntoRunId: row.coalescedIntoRunId,
failureReason: row.failureReason,
@@ -1138,6 +1148,8 @@ export function routineService(
payload: triggerPayload,
projectId,
assigneeAgentId,
routineRevisionId: input.routine.latestRevisionId,
routineEnvFingerprint: createRoutineEnvFingerprint(input.routine.env),
executionWorkspaceId: input.executionWorkspaceId ?? null,
executionWorkspacePreference: input.executionWorkspacePreference ?? null,
executionWorkspaceSettings: input.executionWorkspaceSettings ?? null,
@@ -1183,6 +1195,7 @@ export function routineService(
idempotencyKey: input.idempotencyKey ?? null,
triggerPayload,
dispatchFingerprint,
routineRevisionId: input.routine.latestRevisionId,
})
.returning();
@@ -1430,6 +1443,7 @@ export function routineService(
idempotencyKey: routineRuns.idempotencyKey,
triggerPayload: routineRuns.triggerPayload,
dispatchFingerprint: routineRuns.dispatchFingerprint,
routineRevisionId: routineRuns.routineRevisionId,
linkedIssueId: routineRuns.linkedIssueId,
coalescedIntoRunId: routineRuns.coalescedIntoRunId,
failureReason: routineRuns.failureReason,
@@ -1462,6 +1476,7 @@ export function routineService(
idempotencyKey: run.idempotencyKey,
triggerPayload: run.triggerPayload as Record<string, unknown> | null,
dispatchFingerprint: run.dispatchFingerprint,
routineRevisionId: run.routineRevisionId,
linkedIssueId: run.linkedIssueId,
coalescedIntoRunId: run.coalescedIntoRunId,
failureReason: run.failureReason,
@@ -1508,13 +1523,19 @@ export function routineService(
await assertAssignableAgent(companyId, input.assigneeAgentId ?? null);
if (input.goalId) await assertGoal(companyId, input.goalId);
if (input.parentIssueId) await assertParentIssue(companyId, input.parentIssueId);
const env = input.env === undefined || input.env === null
? null
: await secretsSvc.normalizeEnvBindingsForPersistence(companyId, input.env, {
strictMode: process.env.PAPERCLIP_SECRETS_STRICT_MODE === "true",
fieldPath: "env",
});
const variables = syncRoutineVariablesWithTemplate(
[input.title, input.description],
sanitizeRoutineVariableInputs(input.variables),
);
assertRoutineVariableDefinitions(variables);
const status = normalizeDraftRoutineStatus(input.status, input.assigneeAgentId);
return db.transaction(async (tx) => {
const createdRoutine = await db.transaction(async (tx) => {
const txDb = tx as unknown as Db;
const [created] = await txDb
.insert(routines)
@@ -1531,6 +1552,7 @@ export function routineService(
concurrencyPolicy: input.concurrencyPolicy,
catchUpPolicy: input.catchUpPolicy,
variables,
env,
createdByAgentId: actor.agentId ?? null,
createdByUserId: actor.userId ?? null,
updatedByAgentId: actor.agentId ?? null,
@@ -1540,8 +1562,17 @@ export function routineService(
const { routine } = await appendRoutineRevision(txDb, created, actor, {
changeSummary: "Created routine",
});
if (env) {
await secretsSvc.syncEnvBindingsForTarget(
companyId,
{ targetType: "routine", targetId: routine.id },
env,
{ db: tx },
);
}
return routine;
});
return createdRoutine;
},
update: async (id: string, patch: UpdateRoutine, actor: Actor): Promise<Routine | null> => {
@@ -1551,6 +1582,14 @@ export function routineService(
const nextAssigneeAgentId = patch.assigneeAgentId === undefined ? existing.assigneeAgentId : patch.assigneeAgentId;
const nextTitle = patch.title ?? existing.title;
const nextDescription = patch.description === undefined ? existing.description : patch.description;
const nextEnv = patch.env === undefined
? existing.env
: patch.env === null
? null
: await secretsSvc.normalizeEnvBindingsForPersistence(existing.companyId, patch.env, {
strictMode: process.env.PAPERCLIP_SECRETS_STRICT_MODE === "true",
fieldPath: "env",
});
const requestedStatus = patch.status ?? existing.status;
if (patch.status === "active") {
assertRoutineCanEnable(patch.status, nextAssigneeAgentId);
@@ -1582,7 +1621,7 @@ export function routineService(
if (enabledScheduleTriggers) {
assertScheduleCompatibleVariables(nextVariables);
}
return db.transaction(async (tx) => {
const updatedRoutine = await db.transaction(async (tx) => {
const txDb = tx as unknown as Db;
await tx.execute(sql`select id from ${routines} where ${routines.id} = ${id} for update`);
const locked = await txDb
@@ -1611,6 +1650,7 @@ export function routineService(
concurrencyPolicy: patch.concurrencyPolicy ?? locked.concurrencyPolicy,
catchUpPolicy: patch.catchUpPolicy ?? locked.catchUpPolicy,
variables: nextVariables,
env: nextEnv,
updatedByAgentId: actor.agentId ?? null,
updatedByUserId: actor.userId ?? null,
};
@@ -1633,6 +1673,14 @@ export function routineService(
)
.then((rows) => rows[0] ?? null);
if (latestRevision && snapshotsMatch(nextSnapshot, latestRevision.snapshot as RoutineRevisionSnapshotV1)) {
if (patch.env !== undefined) {
await secretsSvc.syncEnvBindingsForTarget(
locked.companyId,
{ targetType: "routine", targetId: locked.id },
candidate.env,
{ db: tx },
);
}
return locked;
}
}
@@ -1651,6 +1699,7 @@ export function routineService(
concurrencyPolicy: candidate.concurrencyPolicy,
catchUpPolicy: candidate.catchUpPolicy,
variables: candidate.variables,
env: candidate.env,
updatedByAgentId: actor.agentId ?? null,
updatedByUserId: actor.userId ?? null,
updatedAt: new Date(),
@@ -1661,8 +1710,17 @@ export function routineService(
const { routine } = await appendRoutineRevision(txDb, updated, actor, {
changeSummary: "Updated routine",
});
if (patch.env !== undefined) {
await secretsSvc.syncEnvBindingsForTarget(
routine.companyId,
{ targetType: "routine", targetId: routine.id },
routine.env,
{ db: tx },
);
}
return routine;
});
return updatedRoutine;
},
createTrigger: async (
@@ -1770,7 +1828,7 @@ export function routineService(
}
}
return db.transaction(async (tx) => {
const result = await db.transaction(async (tx) => {
const txDb = tx as unknown as Db;
await tx.execute(sql`select id from ${routines} where ${routines.id} = ${existing.routineId} for update`);
const [updated] = await txDb
@@ -1801,12 +1859,13 @@ export function routineService(
});
return { trigger: updated as RoutineTrigger, revision: appended.revision };
});
return result;
},
deleteTrigger: async (id: string, actor: Actor = {}): Promise<{ deleted: boolean; revision: RoutineRevision | null }> => {
const existing = await getTriggerById(id);
if (!existing) return { deleted: false, revision: null };
return db.transaction(async (tx) => {
const result = await db.transaction(async (tx) => {
const txDb = tx as unknown as Db;
await tx.execute(sql`select id from ${routines} where ${routines.id} = ${existing.routineId} for update`);
await txDb.delete(routineTriggers).where(eq(routineTriggers.id, id));
@@ -1821,6 +1880,7 @@ export function routineService(
});
return { deleted: true, revision: appended.revision };
});
return result;
},
rotateTriggerSecret: async (
@@ -1912,7 +1972,7 @@ export function routineService(
const routineSnapshot = snapshot.routine;
await assertRestorableAssignee(existingRoutine.companyId, routineSnapshot.assigneeAgentId, actor);
return db.transaction(async (tx) => {
const result = await db.transaction(async (tx) => {
const txDb = tx as unknown as Db;
await tx.execute(sql`select id from ${routines} where ${routines.id} = ${existingRoutine.id} for update`);
const locked = await txDb
@@ -1964,6 +2024,7 @@ export function routineService(
concurrencyPolicy: routineSnapshot.concurrencyPolicy,
catchUpPolicy: routineSnapshot.catchUpPolicy,
variables: routineSnapshot.variables,
env: routineSnapshot.env,
updatedByAgentId: actor.agentId ?? null,
updatedByUserId: actor.userId ?? null,
updatedAt: now,
@@ -2033,6 +2094,12 @@ export function routineService(
changeSummary: `Restored from revision ${targetRevision.revisionNumber}`,
restoredFromRevisionId: targetRevision.id,
});
await secretsSvc.syncEnvBindingsForTarget(
locked.companyId,
{ targetType: "routine", targetId: locked.id },
routineSnapshot.env,
{ db: tx },
);
return {
routine: appended.routine,
revision: appended.revision,
@@ -2041,6 +2108,7 @@ export function routineService(
secretMaterials: [...recreatedWebhookSecrets.values()].map((entry) => entry.secretMaterial),
};
});
return result;
},
runRoutine: async (id: string, input: RunRoutine, actor?: Actor) => {
@@ -2172,6 +2240,7 @@ export function routineService(
idempotencyKey: routineRuns.idempotencyKey,
triggerPayload: routineRuns.triggerPayload,
dispatchFingerprint: routineRuns.dispatchFingerprint,
routineRevisionId: routineRuns.routineRevisionId,
linkedIssueId: routineRuns.linkedIssueId,
coalescedIntoRunId: routineRuns.coalescedIntoRunId,
failureReason: routineRuns.failureReason,
@@ -2204,6 +2273,7 @@ export function routineService(
idempotencyKey: row.idempotencyKey,
triggerPayload: row.triggerPayload as Record<string, unknown> | null,
dispatchFingerprint: row.dispatchFingerprint,
routineRevisionId: row.routineRevisionId,
linkedIssueId: row.linkedIssueId,
coalescedIntoRunId: row.coalescedIntoRunId,
failureReason: row.failureReason,
server/src/services/secrets.ts
+70 -15
View File
@@ -61,6 +61,8 @@ const COMING_SOON_SECRET_PROVIDERS: ReadonlySet<SecretProvider> = new Set([
"gcp_secret_manager",
"vault",
]);
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
type SecretBindingDb = Pick<Db | DbTransaction, "select" | "delete" | "insert">;
function remoteProviderHttpError(error: unknown, context: {
companyId: string;
@@ -195,6 +197,14 @@ type RuntimeSecretResolution = {
manifestEntry: RuntimeSecretManifestEntry;
};
type SecretResolutionErrorCode =
| "binding_missing"
| "secret_deleted"
| "secret_inactive"
| "version_missing"
| "version_inactive"
| "provider_error";
function asRecord(value: unknown): Record<string, unknown> | null {
if (typeof value !== "object" || value === null || Array.isArray(value)) return null;
return value as Record<string, unknown>;
@@ -238,6 +248,33 @@ function defaultProviderConfigStatus(provider: SecretProvider): SecretProviderCo
return COMING_SOON_SECRET_PROVIDERS.has(provider) ? "coming_soon" : "ready";
}
function secretResolutionErrorCode(error: unknown): SecretResolutionErrorCode {
if (isSecretProviderClientError(error)) return "provider_error";
if (error instanceof HttpError) {
const details = asRecord(error.details);
switch (details?.code) {
case "binding_missing":
case "secret_deleted":
case "secret_inactive":
case "version_missing":
case "version_inactive":
case "provider_error":
return details.code;
}
if (error.message === "Secret is not active") return "secret_inactive";
if (error.message === "Secret version not found") return "version_missing";
if (error.message === "Secret version is not active") return "version_inactive";
if (
error.message === "Secret resolution requires a binding config path" ||
error.message.startsWith("Secret is not bound to ")
) {
return "binding_missing";
}
if (error.status >= 500) return "provider_error";
}
return "provider_error";
}
function assertSelectableProviderConfig(config: {
provider: string;
status: string;
@@ -259,8 +296,8 @@ export function secretService(db: Db) {
fieldPath?: string;
};
async function getById(id: string) {
return db
async function getById(id: string, source: Pick<Db | DbTransaction, "select"> = db) {
return source
.select()
.from(companySecrets)
.where(eq(companySecrets.id, id))
@@ -321,7 +358,7 @@ export function secretService(db: Db) {
) {
if (!context) return;
if (!context.configPath) {
throw unprocessable("Secret resolution requires a binding config path");
throw unprocessable("Secret resolution requires a binding config path", { code: "binding_missing" });
}
const binding = await getBinding({
companyId,
@@ -333,6 +370,7 @@ export function secretService(db: Db) {
if (!binding) {
throw unprocessable(
`Secret is not bound to ${context.consumerType}:${context.consumerId} at ${context.configPath}`,
{ code: "binding_missing" },
);
}
}
@@ -365,8 +403,12 @@ export function secretService(db: Db) {
});
}
async function assertSecretInCompany(companyId: string, secretId: string) {
const secret = await getById(secretId);
async function assertSecretInCompany(
companyId: string,
secretId: string,
source: Pick<Db | DbTransaction, "select"> = db,
) {
const secret = await getById(secretId, source);
if (!secret) throw notFound("Secret not found");
if (secret.status === "deleted") throw notFound("Secret not found");
if (secret.companyId !== companyId) throw unprocessable("Secret must belong to same company");
@@ -495,19 +537,24 @@ export function secretService(db: Db) {
version: number | "latest",
context?: SecretConsumerContext,
): Promise<RuntimeSecretResolution> {
const secret = await assertSecretInCompany(companyId, secretId);
const secret = await getById(secretId);
if (!secret) throw notFound("Secret not found");
if (secret.companyId !== companyId) throw unprocessable("Secret must belong to same company");
const resolvedVersion = version === "latest" ? secret.latestVersion : version;
const providerId = secret.provider as SecretProvider;
const configPath = context?.configPath ?? null;
try {
if (secret.status === "deleted") {
throw new HttpError(404, "Secret not found", { code: "secret_deleted" });
}
if (secret.status !== "active") {
throw unprocessable("Secret is not active");
throw unprocessable("Secret is not active", { code: "secret_inactive" });
}
await assertBindingContext(companyId, secret.id, context);
const versionRow = await getSecretVersion(secret.id, resolvedVersion);
if (!versionRow) throw notFound("Secret version not found");
if (!versionRow) throw new HttpError(404, "Secret version not found", { code: "version_missing" });
if (versionRow.status === "disabled" || versionRow.status === "destroyed" || versionRow.revokedAt) {
throw unprocessable("Secret version is not active");
throw unprocessable("Secret version is not active", { code: "version_inactive" });
}
const provider = getSecretProvider(providerId);
const providerConfig = await getSelectableRuntimeProviderConfig({
@@ -555,7 +602,7 @@ export function secretService(db: Db) {
},
};
} catch (err) {
const errorCode = err instanceof Error ? err.message.slice(0, 120) : "resolution_failed";
const errorCode = secretResolutionErrorCode(err);
await recordAccessEvent({
companyId,
secretId: secret.id,
@@ -1984,6 +2031,7 @@ export function secretService(db: Db) {
companyId: string,
target: { targetType: SecretBindingTargetType; targetId: string; pathPrefix?: string },
envValue: unknown,
options?: { db?: SecretBindingDb },
) => {
const record = asRecord(envValue) ?? {};
const refs: Array<{
@@ -1992,12 +2040,13 @@ export function secretService(db: Db) {
versionSelector: SecretVersionSelector;
}> = [];
const pathPrefix = target.pathPrefix ?? "env";
const bindingDb = options?.db ?? db;
for (const [key, rawBinding] of Object.entries(record)) {
const parsed = envBindingSchema.safeParse(rawBinding);
if (!parsed.success) continue;
const binding = canonicalizeBinding(parsed.data as EnvBinding);
if (binding.type !== "secret_ref") continue;
await assertSecretInCompany(companyId, binding.secretId);
await assertSecretInCompany(companyId, binding.secretId, bindingDb);
refs.push({
secretId: binding.secretId,
configPath: `${pathPrefix}.${key}`,
@@ -2005,8 +2054,8 @@ export function secretService(db: Db) {
});
}
await db.transaction(async (tx) => {
await tx
const writeBindings = async (targetDb: SecretBindingDb) => {
await targetDb
.delete(companySecretBindings)
.where(
and(
@@ -2017,7 +2066,7 @@ export function secretService(db: Db) {
),
);
if (refs.length === 0) return;
await tx.insert(companySecretBindings).values(
await targetDb.insert(companySecretBindings).values(
refs.map((ref) => ({
companyId,
secretId: ref.secretId,
@@ -2028,7 +2077,13 @@ export function secretService(db: Db) {
required: true,
})),
);
});
};
if (options?.db) {
await writeBindings(options.db);
} else {
await db.transaction(async (tx) => writeBindings(tx));
}
return refs;
},