forked from farhoodlabs/paperclip
486fb88a15
> _Stacked on top of #5685 → #5686. Diff against master includes commits from earlier PRs in the stack — review focuses on the two new commits (`Extend sandbox callback bridge for Worker-hosted plugins` + `Add Cloudflare sandbox provider plugin`)._ ## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - Each agent runs in a sandbox environment, and operators choose which provider backs that sandbox — today E2B and Daytona are bundled with the platform > - Cloudflare Workers + Durable Objects + the Sandbox SDK offer a credible new option: globally distributed, cheap idle, and operator-deployable as a single Worker > - To plug it in, Paperclip needs (a) a provider plugin that speaks the `PaperclipPluginManifestV1` lifecycle and (b) a small operator-deployed Worker — the **bridge** — that adapts Paperclip's runtime RPCs to the Cloudflare Sandbox SDK > - The plugin extends the existing sandbox-callback-bridge with a `bridge.transport: "worker"` discriminator so the platform routes runtime RPCs through the Worker bridge instead of the in-process runner > - This pull request adds the plugin, the bridge Worker template, and the supporting adapter-utils + server hooks the new transport needs > - The benefit is that operators can run sandboxes on Cloudflare's edge with no new platform code beyond installing the plugin and deploying the Worker ## What Changed **Shared support (`Extend sandbox callback bridge for Worker-hosted plugins`):** - `packages/adapter-utils/src/sandbox-callback-bridge.{ts,test.ts}`: expose `expectedHostHeader` so plugin-side bridge clients can verify the canonical request envelope before forwarding. - `packages/adapter-utils/src/command-managed-runtime.{ts,test.ts}`: relax the always-fresh runner construction so callers can re-use a runner across exec calls (Worker-hosted bridges hold the runner inside a Durable Object). - `server/src/services/environment-runtime.ts` + `environment-runtime.test.ts`: route Worker-hosted bridges through the same env-shaping path as E2B and pin the `requestEnv` contract. - `server/src/services/plugin-environment-driver.ts`: thread an optional `issueId` through the runtime descriptor so bridges can scope leases to the originating issue (used by Cloudflare to map a sandbox to the issue/workflow for billing and audit). - `packages/plugins/sdk/src/protocol.ts`: add `issueId?` to `PluginEnvironmentDriverBaseParams` and the new `bridge.transport: "worker"` discriminator that the new plugin declares. - `server/__tests__/heartbeat-plugin-environment.test.ts`: pin the heartbeat path against the new runtime descriptor. **The Cloudflare plugin itself (`Add Cloudflare sandbox provider plugin`):** - `packages/plugins/sandbox-providers/cloudflare/`: plugin entry, manifest, plugin runtime (lifecycle + bridge client), config parsing, and Vitest coverage. Manifest declares `bridge.transport: "worker"` so the platform routes runtime RPCs through the bridge client. - `bridge-template/`: a Worker template the operator deploys with `wrangler`. Owns Durable Object-backed sessions (`sessions.ts`), exec/stream routes (`exec.ts`, `routes.ts`), and an HMAC auth layer (`auth.ts`) that pins the `Host` header surface. Includes the SDK-contract-correct exec implementation, lease recovery, and chunked stdout/stderr streaming. - Tests cover lease/session handoff (`bridge-template/src/exec.test.ts`, `routes.test.ts`), bridge client request shaping (`src/bridge-client.test.ts`), and end-to-end plugin behavior (`src/plugin.test.ts`) including streamed exec output. 27 tests in total. - `README.md` walks the operator through deploying the bridge Worker, registering the plugin, and configuring the runtime. ## Verification - `pnpm typecheck` - `pnpm exec vitest run --no-coverage packages/adapter-utils/src/sandbox-callback-bridge.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/environment-runtime.test.ts server/src/__tests__/heartbeat-plugin-environment.test.ts` - `(cd packages/plugins/sandbox-providers/cloudflare && pnpm test)` — 27 passing For an operator-side smoke test: 1. Deploy the bridge: `cd packages/plugins/sandbox-providers/cloudflare/bridge-template && wrangler deploy` 2. Register the plugin in your Paperclip instance, point its bridge URL at the deployed Worker, set the HMAC shared secret. 3. Create a sandbox environment whose provider is `cloudflare`, then run a Codex or Claude job against it. ## Risks - Adds a new `bridge.transport: "worker"` code path, but the existing E2B / Daytona transports go through the same shaped helpers and have explicit test coverage that pins their behavior unchanged. - The Worker bridge stores session state in a Durable Object; operator instances must be aware of the corresponding Cloudflare costs (DO requests, storage). Documented in the README. - The `issueId` plumbing is optional throughout — existing plugins that don't supply it continue to work. ## Model Used - Provider: Anthropic - Model: Claude Opus 4.7 (1M context) - Capabilities used: extended reasoning, tool use (Read/Edit/Bash/Grep) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots — N/A, no UI change - [x] I have updated relevant documentation to reflect my changes (plugin README, bridge-template README) - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
221 lines
7.8 KiB
TypeScript
221 lines
7.8 KiB
TypeScript
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { execFile as execFileCallback } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
|
|
import { prepareCommandManagedRuntime } from "./command-managed-runtime.js";
|
|
import type { RunProcessResult } from "./server-utils.js";
|
|
|
|
const execFile = promisify(execFileCallback);
|
|
|
|
describe("command managed runtime", () => {
|
|
const cleanupDirs: string[] = [];
|
|
|
|
afterEach(async () => {
|
|
while (cleanupDirs.length > 0) {
|
|
const dir = cleanupDirs.pop();
|
|
if (!dir) continue;
|
|
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
|
}
|
|
});
|
|
|
|
it("keeps the runtime overlay out of sandbox workspace sync by default", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-"));
|
|
cleanupDirs.push(rootDir);
|
|
|
|
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
|
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
|
await mkdir(path.join(localWorkspaceDir, ".paperclip-runtime"), { recursive: true });
|
|
await mkdir(remoteWorkspaceDir, { recursive: true });
|
|
await writeFile(path.join(localWorkspaceDir, "README.md"), "local workspace\n", "utf8");
|
|
await writeFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "{\"keep\":true}\n", "utf8");
|
|
|
|
const calls: Array<{
|
|
command: string;
|
|
args?: string[];
|
|
cwd?: string;
|
|
env?: Record<string, string>;
|
|
stdin?: string;
|
|
timeoutMs?: number;
|
|
}> = [];
|
|
const runner = {
|
|
execute: async (input: {
|
|
command: string;
|
|
args?: string[];
|
|
cwd?: string;
|
|
env?: Record<string, string>;
|
|
stdin?: string;
|
|
timeoutMs?: number;
|
|
}): Promise<RunProcessResult> => {
|
|
calls.push({ ...input });
|
|
const startedAt = new Date().toISOString();
|
|
const env = {
|
|
...process.env,
|
|
...input.env,
|
|
};
|
|
const command =
|
|
input.command === "sh" ? "/bin/sh" : input.command === "bash" ? "/bin/bash" : input.command;
|
|
const args = [...(input.args ?? [])];
|
|
if (
|
|
input.stdin != null &&
|
|
(input.command === "sh" || input.command === "bash") &&
|
|
(args[0] === "-c" || args[0] === "-lc") &&
|
|
typeof args[1] === "string"
|
|
) {
|
|
env.PAPERCLIP_TEST_STDIN = input.stdin;
|
|
args[1] = `printf '%s' \"$PAPERCLIP_TEST_STDIN\" | (${args[1]})`;
|
|
}
|
|
try {
|
|
const result = await execFile(command, args, {
|
|
cwd: input.cwd,
|
|
env,
|
|
maxBuffer: 32 * 1024 * 1024,
|
|
timeout: input.timeoutMs,
|
|
});
|
|
return {
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: result.stdout,
|
|
stderr: result.stderr,
|
|
pid: null,
|
|
startedAt,
|
|
};
|
|
} catch (error) {
|
|
const err = error as NodeJS.ErrnoException & {
|
|
stdout?: string;
|
|
stderr?: string;
|
|
code?: string | number | null;
|
|
signal?: NodeJS.Signals | null;
|
|
killed?: boolean;
|
|
};
|
|
return {
|
|
exitCode: typeof err.code === "number" ? err.code : null,
|
|
signal: err.signal ?? null,
|
|
timedOut: Boolean(err.killed && input.timeoutMs),
|
|
stdout: err.stdout ?? "",
|
|
stderr: err.stderr ?? "",
|
|
pid: null,
|
|
startedAt,
|
|
};
|
|
}
|
|
},
|
|
};
|
|
|
|
const prepared = await prepareCommandManagedRuntime({
|
|
runner,
|
|
spec: {
|
|
remoteCwd: remoteWorkspaceDir,
|
|
timeoutMs: 30_000,
|
|
},
|
|
adapterKey: "claude",
|
|
workspaceLocalDir: localWorkspaceDir,
|
|
});
|
|
|
|
await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("local workspace\n");
|
|
await expect(readFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).rejects
|
|
.toMatchObject({ code: "ENOENT" });
|
|
expect(calls.every((call) => call.stdin == null)).toBe(true);
|
|
|
|
await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true });
|
|
await writeFile(path.join(remoteWorkspaceDir, "README.md"), "remote workspace\n", "utf8");
|
|
await writeFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "remote-state.json"), "{\"remote\":true}\n", "utf8");
|
|
await prepared.restoreWorkspace();
|
|
|
|
await expect(readFile(path.join(localWorkspaceDir, "README.md"), "utf8")).resolves.toBe("remote workspace\n");
|
|
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).resolves
|
|
.toBe("{\"keep\":true}\n");
|
|
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "remote-state.json"), "utf8")).rejects
|
|
.toMatchObject({ code: "ENOENT" });
|
|
expect(calls.every((call) => call.stdin == null)).toBe(true);
|
|
});
|
|
|
|
it("runs setup commands from a stable root cwd when staging into a nested remote workspace dir", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-nested-"));
|
|
cleanupDirs.push(rootDir);
|
|
|
|
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
|
const remoteBaseDir = path.join(rootDir, "remote-base");
|
|
const remoteWorkspaceDir = path.join(remoteBaseDir, ".paperclip-runtime", "runs", "test", "workspace");
|
|
await mkdir(localWorkspaceDir, { recursive: true });
|
|
await mkdir(remoteBaseDir, { recursive: true });
|
|
await writeFile(path.join(localWorkspaceDir, "README.md"), "local workspace\n", "utf8");
|
|
|
|
const calls: Array<{
|
|
command: string;
|
|
args?: string[];
|
|
cwd?: string;
|
|
env?: Record<string, string>;
|
|
stdin?: string;
|
|
timeoutMs?: number;
|
|
}> = [];
|
|
const runner = {
|
|
execute: async (input: {
|
|
command: string;
|
|
args?: string[];
|
|
cwd?: string;
|
|
env?: Record<string, string>;
|
|
stdin?: string;
|
|
timeoutMs?: number;
|
|
}): Promise<RunProcessResult> => {
|
|
calls.push({ ...input });
|
|
const startedAt = new Date().toISOString();
|
|
try {
|
|
const result = await execFile(input.command === "sh" ? "/bin/sh" : input.command, input.args ?? [], {
|
|
cwd: input.cwd,
|
|
env: {
|
|
...process.env,
|
|
...input.env,
|
|
},
|
|
maxBuffer: 32 * 1024 * 1024,
|
|
timeout: input.timeoutMs,
|
|
});
|
|
return {
|
|
exitCode: 0,
|
|
signal: null,
|
|
timedOut: false,
|
|
stdout: result.stdout,
|
|
stderr: result.stderr,
|
|
pid: null,
|
|
startedAt,
|
|
};
|
|
} catch (error) {
|
|
const err = error as NodeJS.ErrnoException & {
|
|
stdout?: string;
|
|
stderr?: string;
|
|
code?: string | number | null;
|
|
signal?: NodeJS.Signals | null;
|
|
killed?: boolean;
|
|
};
|
|
return {
|
|
exitCode: typeof err.code === "number" ? err.code : null,
|
|
signal: err.signal ?? null,
|
|
timedOut: Boolean(err.killed && input.timeoutMs),
|
|
stdout: err.stdout ?? "",
|
|
stderr: err.stderr ?? "",
|
|
pid: null,
|
|
startedAt,
|
|
};
|
|
}
|
|
},
|
|
};
|
|
|
|
await prepareCommandManagedRuntime({
|
|
runner,
|
|
spec: {
|
|
remoteCwd: remoteBaseDir,
|
|
timeoutMs: 30_000,
|
|
},
|
|
adapterKey: "codex",
|
|
workspaceLocalDir: localWorkspaceDir,
|
|
workspaceRemoteDir: remoteWorkspaceDir,
|
|
});
|
|
|
|
expect(calls.length).toBeGreaterThan(0);
|
|
expect(calls.every((call) => call.cwd === "/")).toBe(true);
|
|
await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("local workspace\n");
|
|
});
|
|
});
|