[codex] Add plugin orchestration host APIs (#4114)

## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - The plugin system is the extension path for optional capabilities that should not require core product changes for every integration. > - Plugins need scoped host APIs for issue orchestration, documents, wakeups, summaries, activity attribution, and isolated database state. > - Without those host APIs, richer plugins either cannot coordinate Paperclip work safely or need privileged core-side special cases. > - This pull request adds the plugin orchestration host surface, scoped route dispatch, a database namespace layer, and a smoke plugin that exercises the contract. > - The benefit is a broader plugin API that remains company-scoped, auditable, and covered by tests. ## What Changed - Added plugin orchestration host APIs for issue creation, document access, wakeups, summaries, plugin-origin activity, and scoped API route dispatch. - Added plugin database namespace tables, schema exports, migration checks, and idempotent replay coverage under migration `0059_plugin_database_namespaces`. - Added shared plugin route/API types and validators used by server and SDK boundaries. - Expanded plugin SDK types, protocol helpers, worker RPC host behavior, and testing utilities for orchestration flows. - Added the `plugin-orchestration-smoke-example` package to exercise scoped routes, restricted database namespaces, issue orchestration, documents, wakeups, summaries, and UI status surfaces. - Kept the new orchestration smoke fixture out of the root pnpm workspace importer so this PR preserves the repository policy of not committing `pnpm-lock.yaml`. - Updated plugin docs and database docs for the new orchestration and database namespace surfaces. - Rebased the branch onto `public-gh/master`, resolved conflicts, and removed `pnpm-lock.yaml` from the final PR diff. ## Verification - `pnpm install --frozen-lockfile` - `pnpm --filter @paperclipai/db typecheck` - `pnpm exec vitest run packages/db/src/client.test.ts` - `pnpm exec vitest run server/src/__tests__/plugin-database.test.ts server/src/__tests__/plugin-orchestration-apis.test.ts server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/plugin-scoped-api-routes.test.ts server/src/__tests__/plugin-sdk-orchestration-contract.test.ts` - From `packages/plugins/examples/plugin-orchestration-smoke-example`: `pnpm exec vitest run --config ./vitest.config.ts` - `pnpm --dir packages/plugins/examples/plugin-orchestration-smoke-example run typecheck` - `pnpm --filter @paperclipai/server typecheck` - PR CI on latest head `293fc67c`: `policy`, `verify`, `e2e`, and `security/snyk` all passed. ## Risks - Medium risk: this expands plugin host authority, so route auth, company scoping, and plugin-origin activity attribution need careful review. - Medium risk: database namespace migration behavior must remain idempotent for environments that may have seen earlier branch versions. - Medium risk: the orchestration smoke fixture is intentionally excluded from the root workspace importer to avoid a `pnpm-lock.yaml` PR diff; direct fixture verification remains listed above. - Low operational risk from the PR setup itself: the branch is rebased onto current `master`, the migration is ordered after upstream `0057`/`0058`, and `pnpm-lock.yaml` is not in the final diff. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. Roadmap checked: this work aligns with the completed Plugin system milestone and extends the plugin surface rather than duplicating an unrelated planned core feature. ## Model Used - OpenAI Codex, GPT-5-based coding agent in a tool-enabled CLI environment. Exact hosted model build and context-window size are not exposed by the runtime; reasoning/tool use were enabled for repository inspection, editing, testing, git operations, and PR creation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A: no core UI screen change; example plugin UI contract is covered by tests) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-20 08:52:51 -05:00
parent 16b2b84d84
commit 9c6f551595
53 changed files with 5584 additions and 53 deletions
@@ -11,6 +11,10 @@ import {
  PLUGIN_LAUNCHER_BOUNDS,
  PLUGIN_LAUNCHER_RENDER_ENVIRONMENTS,
  PLUGIN_STATE_SCOPE_KINDS,
+  PLUGIN_DATABASE_CORE_READ_TABLES,
+  PLUGIN_API_ROUTE_AUTH_MODES,
+  PLUGIN_API_ROUTE_CHECKOUT_POLICIES,
+  PLUGIN_API_ROUTE_METHODS,
 } from "../constants.js";

 // ---------------------------------------------------------------------------
@@ -336,6 +340,48 @@ export const pluginLauncherDeclarationSchema = z.object({

 export type PluginLauncherDeclarationInput = z.infer<typeof pluginLauncherDeclarationSchema>;

+export const pluginDatabaseDeclarationSchema = z.object({
+  namespaceSlug: z.string().regex(/^[a-z0-9][a-z0-9_]*$/, {
+    message: "namespaceSlug must be lowercase letters, digits, or underscores and start with a letter or digit",
+  }).max(40).optional(),
+  migrationsDir: z.string().min(1).refine(
+    (value) => !value.startsWith("/") && !value.includes("..") && !/[\\]/.test(value),
+    { message: "migrationsDir must be a relative package path without '..' or backslashes" },
+  ),
+  coreReadTables: z.array(z.enum(PLUGIN_DATABASE_CORE_READ_TABLES)).optional(),
+});
+
+export type PluginDatabaseDeclarationInput = z.infer<typeof pluginDatabaseDeclarationSchema>;
+
+export const pluginApiRouteDeclarationSchema = z.object({
+  routeKey: z.string().min(1).max(100).regex(/^[a-z0-9][a-z0-9._:-]*$/, {
+    message: "routeKey must be lowercase letters, digits, dots, colons, underscores, or hyphens",
+  }),
+  method: z.enum(PLUGIN_API_ROUTE_METHODS),
+  path: z.string().min(1).regex(/^\/[a-zA-Z0-9:_./-]*$/, {
+    message: "path must start with / and contain only path-safe literal or :param segments",
+  }).refine(
+    (value) =>
+      !value.includes("..") &&
+      !value.includes("//") &&
+      value !== "/api" &&
+      !value.startsWith("/api/") &&
+      value !== "/plugins" &&
+      !value.startsWith("/plugins/"),
+    { message: "path must stay inside the plugin api namespace" },
+  ),
+  auth: z.enum(PLUGIN_API_ROUTE_AUTH_MODES),
+  capability: z.literal("api.routes.register"),
+  checkoutPolicy: z.enum(PLUGIN_API_ROUTE_CHECKOUT_POLICIES).optional(),
+  companyResolution: z.discriminatedUnion("from", [
+    z.object({ from: z.literal("body"), key: z.string().min(1) }),
+    z.object({ from: z.literal("query"), key: z.string().min(1) }),
+    z.object({ from: z.literal("issue"), param: z.string().min(1) }),
+  ]).optional(),
+});
+
+export type PluginApiRouteDeclarationInput = z.infer<typeof pluginApiRouteDeclarationSchema>;
+
 // ---------------------------------------------------------------------------
 // Plugin Manifest V1 schema
 // ---------------------------------------------------------------------------
@@ -405,6 +451,8 @@ export const pluginManifestV1Schema = z.object({
  jobs: z.array(pluginJobDeclarationSchema).optional(),
  webhooks: z.array(pluginWebhookDeclarationSchema).optional(),
  tools: z.array(pluginToolDeclarationSchema).optional(),
+  database: pluginDatabaseDeclarationSchema.optional(),
+  apiRoutes: z.array(pluginApiRouteDeclarationSchema).optional(),
  launchers: z.array(pluginLauncherDeclarationSchema).optional(),
  ui: z.object({
    slots: z.array(pluginUiSlotDeclarationSchema).min(1).optional(),
@@ -474,6 +522,42 @@ export const pluginManifestV1Schema = z.object({
    }
  }

+  if (manifest.apiRoutes && manifest.apiRoutes.length > 0) {
+    if (!manifest.capabilities.includes("api.routes.register")) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: "Capability 'api.routes.register' is required when apiRoutes are declared",
+        path: ["capabilities"],
+      });
+    }
+  }
+
+  if (manifest.database) {
+    const requiredCapabilities = [
+      "database.namespace.migrate",
+      "database.namespace.read",
+    ] as const;
+    for (const capability of requiredCapabilities) {
+      if (!manifest.capabilities.includes(capability)) {
+        ctx.addIssue({
+          code: z.ZodIssueCode.custom,
+          message: `Capability '${capability}' is required when database migrations are declared`,
+          path: ["capabilities"],
+        });
+      }
+    }
+
+    const coreReadTables = manifest.database.coreReadTables ?? [];
+    const duplicates = coreReadTables.filter((table, i) => coreReadTables.indexOf(table) !== i);
+    if (duplicates.length > 0) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: `Duplicate database coreReadTables: ${[...new Set(duplicates)].join(", ")}`,
+        path: ["database", "coreReadTables"],
+      });
+    }
+  }
+
  // ── Uniqueness checks ──────────────────────────────────────────────────
  // Duplicate keys within a plugin's own manifest are always a bug. The host
  // would not know which declaration takes precedence, so we reject early.
@@ -504,6 +588,27 @@ export const pluginManifestV1Schema = z.object({
    }
  }

+  if (manifest.apiRoutes) {
+    const routeKeys = manifest.apiRoutes.map((route) => route.routeKey);
+    const duplicateKeys = routeKeys.filter((key, i) => routeKeys.indexOf(key) !== i);
+    if (duplicateKeys.length > 0) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: `Duplicate api route keys: ${[...new Set(duplicateKeys)].join(", ")}`,
+        path: ["apiRoutes"],
+      });
+    }
+    const routeSignatures = manifest.apiRoutes.map((route) => `${route.method} ${route.path}`);
+    const duplicateRoutes = routeSignatures.filter((sig, i) => routeSignatures.indexOf(sig) !== i);
+    if (duplicateRoutes.length > 0) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: `Duplicate api routes: ${[...new Set(duplicateRoutes)].join(", ")}`,
+        path: ["apiRoutes"],
+      });
+    }
+  }
+
  // tool names must be unique within the plugin (namespaced at runtime)
  if (manifest.tools) {
    const toolNames = manifest.tools.map((t) => t.name);