[codex] Add plugin orchestration host APIs (#4114)

## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - The plugin system is the extension path for optional capabilities that should not require core product changes for every integration. > - Plugins need scoped host APIs for issue orchestration, documents, wakeups, summaries, activity attribution, and isolated database state. > - Without those host APIs, richer plugins either cannot coordinate Paperclip work safely or need privileged core-side special cases. > - This pull request adds the plugin orchestration host surface, scoped route dispatch, a database namespace layer, and a smoke plugin that exercises the contract. > - The benefit is a broader plugin API that remains company-scoped, auditable, and covered by tests. ## What Changed - Added plugin orchestration host APIs for issue creation, document access, wakeups, summaries, plugin-origin activity, and scoped API route dispatch. - Added plugin database namespace tables, schema exports, migration checks, and idempotent replay coverage under migration `0059_plugin_database_namespaces`. - Added shared plugin route/API types and validators used by server and SDK boundaries. - Expanded plugin SDK types, protocol helpers, worker RPC host behavior, and testing utilities for orchestration flows. - Added the `plugin-orchestration-smoke-example` package to exercise scoped routes, restricted database namespaces, issue orchestration, documents, wakeups, summaries, and UI status surfaces. - Kept the new orchestration smoke fixture out of the root pnpm workspace importer so this PR preserves the repository policy of not committing `pnpm-lock.yaml`. - Updated plugin docs and database docs for the new orchestration and database namespace surfaces. - Rebased the branch onto `public-gh/master`, resolved conflicts, and removed `pnpm-lock.yaml` from the final PR diff. ## Verification - `pnpm install --frozen-lockfile` - `pnpm --filter @paperclipai/db typecheck` - `pnpm exec vitest run packages/db/src/client.test.ts` - `pnpm exec vitest run server/src/__tests__/plugin-database.test.ts server/src/__tests__/plugin-orchestration-apis.test.ts server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/plugin-scoped-api-routes.test.ts server/src/__tests__/plugin-sdk-orchestration-contract.test.ts` - From `packages/plugins/examples/plugin-orchestration-smoke-example`: `pnpm exec vitest run --config ./vitest.config.ts` - `pnpm --dir packages/plugins/examples/plugin-orchestration-smoke-example run typecheck` - `pnpm --filter @paperclipai/server typecheck` - PR CI on latest head `293fc67c`: `policy`, `verify`, `e2e`, and `security/snyk` all passed. ## Risks - Medium risk: this expands plugin host authority, so route auth, company scoping, and plugin-origin activity attribution need careful review. - Medium risk: database namespace migration behavior must remain idempotent for environments that may have seen earlier branch versions. - Medium risk: the orchestration smoke fixture is intentionally excluded from the root workspace importer to avoid a `pnpm-lock.yaml` PR diff; direct fixture verification remains listed above. - Low operational risk from the PR setup itself: the branch is rebased onto current `master`, the migration is ordered after upstream `0057`/`0058`, and `pnpm-lock.yaml` is not in the final diff. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. Roadmap checked: this work aligns with the completed Plugin system milestone and extends the plugin surface rather than duplicating an unrelated planned core feature. ## Model Used - OpenAI Codex, GPT-5-based coding agent in a tool-enabled CLI environment. Exact hosted model build and context-window size are not exposed by the runtime; reasoning/tool use were enabled for repository inspection, editing, testing, git operations, and PR creation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A: no core UI screen change; example plugin UI contract is covered by tests) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-20 08:52:51 -05:00
parent 16b2b84d84
commit 9c6f551595
53 changed files with 5584 additions and 53 deletions
@@ -0,0 +1,269 @@
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { and, eq, sql } from "drizzle-orm";
+import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
+import {
+  companies,
+  createDb,
+  issueRelations,
+  issues,
+  pluginDatabaseNamespaces,
+  pluginMigrations,
+  plugins,
+} from "@paperclipai/db";
+import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
+import {
+  getEmbeddedPostgresTestSupport,
+  startEmbeddedPostgresTestDatabase,
+} from "./helpers/embedded-postgres.js";
+import {
+  derivePluginDatabaseNamespace,
+  pluginDatabaseService,
+  validatePluginMigrationStatement,
+  validatePluginRuntimeExecute,
+  validatePluginRuntimeQuery,
+} from "../services/plugin-database.js";
+
+const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
+
+if (!embeddedPostgresSupport.supported) {
+  console.warn(
+    `Skipping embedded Postgres plugin database tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
+  );
+}
+
+describe("plugin database SQL validation", () => {
+  it("allows namespace migrations with whitelisted public foreign keys", () => {
+    expect(() =>
+      validatePluginMigrationStatement(
+        "CREATE TABLE plugin_test.rows (id uuid PRIMARY KEY, issue_id uuid REFERENCES public.issues(id))",
+        "plugin_test",
+        ["issues"],
+      )
+    ).not.toThrow();
+  });
+
+  it("rejects migrations that create public objects", () => {
+    expect(() =>
+      validatePluginMigrationStatement(
+        "CREATE TABLE public.rows (id uuid PRIMARY KEY)",
+        "plugin_test",
+        ["issues"],
+      )
+    ).toThrow(/public/i);
+  });
+
+  it("allows whitelisted runtime reads but rejects public writes", () => {
+    expect(() =>
+      validatePluginRuntimeQuery(
+        "SELECT r.id FROM plugin_test.rows r JOIN public.issues i ON i.id = r.issue_id",
+        "plugin_test",
+        ["issues"],
+      )
+    ).not.toThrow();
+    expect(() =>
+      validatePluginRuntimeExecute("UPDATE public.issues SET title = $1", "plugin_test")
+    ).toThrow(/namespace/i);
+  });
+
+  it("targets anonymous DO blocks without rejecting do-prefixed aliases", () => {
+    expect(() =>
+      validatePluginRuntimeQuery(
+        "SELECT EXTRACT(DOW FROM created_at) AS do_flag FROM plugin_test.rows",
+        "plugin_test",
+      )
+    ).not.toThrow();
+    expect(() =>
+      validatePluginMigrationStatement("DO $$ BEGIN END $$;", "plugin_test")
+    ).toThrow(/disallowed/i);
+  });
+});
+
+describeEmbeddedPostgres("plugin database namespaces", () => {
+  let db!: ReturnType<typeof createDb>;
+  let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
+  let packageRoots: string[] = [];
+
+  beforeAll(async () => {
+    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-plugin-db-");
+    db = createDb(tempDb.connectionString);
+  }, 20_000);
+
+  afterEach(async () => {
+    for (const pluginKey of ["paperclip.dbtest", "paperclip.escape"]) {
+      const namespace = derivePluginDatabaseNamespace(pluginKey);
+      await db.execute(sql.raw(`DROP SCHEMA IF EXISTS "${namespace}" CASCADE`));
+    }
+    await db.delete(pluginMigrations);
+    await db.delete(pluginDatabaseNamespaces);
+    await db.delete(plugins);
+    await db.delete(issueRelations);
+    await db.delete(issues);
+    await db.delete(companies);
+    await Promise.all(packageRoots.map((root) => rm(root, { recursive: true, force: true })));
+    packageRoots = [];
+  });
+
+  afterAll(async () => {
+    await tempDb?.cleanup();
+  });
+
+  async function createPluginPackage(manifest: PaperclipPluginManifestV1, migrationSql: string) {
+    const packageRoot = await mkdtemp(path.join(os.tmpdir(), "paperclip-plugin-package-"));
+    packageRoots.push(packageRoot);
+    const migrationsDir = path.join(packageRoot, manifest.database!.migrationsDir);
+    await mkdir(migrationsDir, { recursive: true });
+    await writeFile(path.join(migrationsDir, "001_init.sql"), migrationSql, "utf8");
+    return packageRoot;
+  }
+
+  async function installPluginRecord(manifest: PaperclipPluginManifestV1) {
+    const pluginId = randomUUID();
+    await db.insert(plugins).values({
+      id: pluginId,
+      pluginKey: manifest.id,
+      packageName: manifest.id,
+      version: manifest.version,
+      apiVersion: manifest.apiVersion,
+      categories: manifest.categories,
+      manifestJson: manifest,
+      status: "installed",
+      installOrder: 1,
+    });
+    return pluginId;
+  }
+
+  function manifest(pluginKey = "paperclip.dbtest"): PaperclipPluginManifestV1 {
+    return {
+      id: pluginKey,
+      apiVersion: 1,
+      version: "1.0.0",
+      displayName: "DB Test",
+      description: "Exercises restricted plugin database access.",
+      author: "Paperclip",
+      categories: ["automation"],
+      capabilities: [
+        "database.namespace.migrate",
+        "database.namespace.read",
+        "database.namespace.write",
+      ],
+      entrypoints: { worker: "./dist/worker.js" },
+      database: {
+        migrationsDir: "migrations",
+        coreReadTables: ["issues"],
+      },
+    };
+  }
+
+  it("applies migrations once and allows whitelisted core joins at runtime", async () => {
+    const pluginManifest = manifest();
+    const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
+    const packageRoot = await createPluginPackage(
+      pluginManifest,
+      `
+      CREATE TABLE ${namespace}.mission_rows (
+        id uuid PRIMARY KEY,
+        issue_id uuid NOT NULL REFERENCES public.issues(id),
+        label text NOT NULL
+      );
+      `,
+    );
+    const pluginId = await installPluginRecord(pluginManifest);
+    const companyId = randomUUID();
+    const issueId = randomUUID();
+    await db.insert(companies).values({
+      id: companyId,
+      name: "Paperclip",
+      issuePrefix: "TST",
+      requireBoardApprovalForNewAgents: false,
+    });
+    await db.insert(issues).values({
+      id: issueId,
+      companyId,
+      title: "Joined issue",
+      status: "todo",
+      priority: "medium",
+      identifier: "TST-1",
+    });
+
+    const pluginDb = pluginDatabaseService(db);
+    await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
+    await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
+
+    await pluginDb.execute(
+      pluginId,
+      `INSERT INTO ${namespace}.mission_rows (id, issue_id, label) VALUES ($1, $2, $3)`,
+      [randomUUID(), issueId, "alpha"],
+    );
+    const rows = await pluginDb.query<{ label: string; title: string }>(
+      pluginId,
+      `SELECT m.label, i.title FROM ${namespace}.mission_rows m JOIN public.issues i ON i.id = m.issue_id`,
+    );
+    expect(rows).toEqual([{ label: "alpha", title: "Joined issue" }]);
+
+    const migrations = await db
+      .select()
+      .from(pluginMigrations)
+      .where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.status, "applied")));
+    expect(migrations).toHaveLength(1);
+  });
+
+  it("rejects runtime writes to public core tables", async () => {
+    const pluginManifest = manifest();
+    const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
+    const packageRoot = await createPluginPackage(
+      pluginManifest,
+      `CREATE TABLE ${namespace}.notes (id uuid PRIMARY KEY, body text NOT NULL);`,
+    );
+    const pluginId = await installPluginRecord(pluginManifest);
+    const pluginDb = pluginDatabaseService(db);
+    await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
+
+    await expect(
+      pluginDb.execute(pluginId, "UPDATE public.issues SET title = $1", ["bad"]),
+    ).rejects.toThrow(/plugin namespace/i);
+  });
+
+  it("records a failed migration when SQL escapes the plugin namespace", async () => {
+    const pluginManifest = manifest("paperclip.escape");
+    const packageRoot = await createPluginPackage(
+      pluginManifest,
+      "CREATE TABLE public.plugin_escape (id uuid PRIMARY KEY);",
+    );
+    const pluginId = await installPluginRecord(pluginManifest);
+
+    await expect(
+      pluginDatabaseService(db).applyMigrations(pluginId, pluginManifest, packageRoot),
+    ).rejects.toThrow(/public\.plugin_escape|public/i);
+
+    const [migration] = await db
+      .select()
+      .from(pluginMigrations)
+      .where(eq(pluginMigrations.pluginId, pluginId));
+    expect(migration?.status).toBe("failed");
+  });
+
+  it("rejects checksum changes for already applied migrations", async () => {
+    const pluginManifest = manifest();
+    const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
+    const packageRoot = await createPluginPackage(
+      pluginManifest,
+      `CREATE TABLE ${namespace}.checksum_rows (id uuid PRIMARY KEY);`,
+    );
+    const pluginId = await installPluginRecord(pluginManifest);
+    const pluginDb = pluginDatabaseService(db);
+    await pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot);
+
+    await writeFile(
+      path.join(packageRoot, "migrations", "001_init.sql"),
+      `CREATE TABLE ${namespace}.checksum_rows (id uuid PRIMARY KEY, note text);`,
+      "utf8",
+    );
+
+    await expect(pluginDb.applyMigrations(pluginId, pluginManifest, packageRoot))
+      .rejects.toThrow(/checksum mismatch/i);
+  });
+});