forked from farhoodlabs/paperclip
Expand plugin host surface (#5205)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - The plugin system is the extension boundary for optional product capabilities > - Rich plugins need more than a worker entrypoint: they need scoped database storage, local project folders, managed agents/routines, host navigation, and reusable UI components > - The LLM Wiki work exposed those missing host surfaces while keeping plugin code outside the core control plane > - This pull request expands the core plugin host, SDK, server APIs, and UI bridge so plugins can declare and use those surfaces > - The benefit is that future plugins can integrate with Paperclip through documented, validated contracts instead of bespoke server or UI imports ## What Changed - Added plugin-managed database namespaces and migration tracking, including Drizzle schema/migration files and SQL validation for namespace isolation. - Added server support for plugin local folders, managed agents, managed routines, scoped plugin APIs, and plugin operation visibility. - Expanded shared plugin manifest/types/validators and SDK host/testing/UI exports for richer plugin surfaces. - Added reusable UI pieces for file trees, managed routines, resizable sidebars, route sidebars, and plugin bridge initialization. - Updated plugin docs and example plugins to use the expanded host and SDK surface. ## Verification - `pnpm install --frozen-lockfile` - `pnpm run preflight:workspace-links && pnpm exec vitest run packages/shared/src/validators/plugin.test.ts server/src/__tests__/plugin-database.test.ts server/src/__tests__/plugin-local-folders.test.ts server/src/__tests__/plugin-managed-agents.test.ts server/src/__tests__/plugin-managed-routines.test.ts server/src/__tests__/plugin-orchestration-apis.test.ts ui/src/api/plugins.test.ts ui/src/components/FileTree.test.tsx ui/src/components/ResizableSidebarPane.test.tsx ui/src/pages/PluginPage.test.tsx ui/src/plugins/bridge.test.ts` passed: 11 files, 67 tests. - Confirmed this PR changes 89 files and does not include `pnpm-lock.yaml` or `.github/workflows/*`. ## Risks - Medium: this expands plugin host contracts across db/shared/server/ui and includes a new core migration (`0076_useful_elektra.sql`). - The plugin database namespace validator is intentionally restrictive; plugin authors may need follow-up affordances for SQL patterns that remain blocked. - Merge this before the LLM Wiki plugin PR so the plugin can resolve the new SDK and host APIs. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5 coding agent, tool-enabled shell/git/GitHub workflow. Context window size was not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
@@ -3,7 +3,7 @@ import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
companies,
|
||||
createDb,
|
||||
@@ -25,9 +25,11 @@ import {
|
||||
validatePluginRuntimeExecute,
|
||||
validatePluginRuntimeQuery,
|
||||
} from "../services/plugin-database.js";
|
||||
import { pluginLoader } from "../services/plugin-loader.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
const multiMigrationPluginKey = "paperclip.dbfixture";
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
@@ -93,7 +95,7 @@ describeEmbeddedPostgres("plugin database namespaces", () => {
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
for (const pluginKey of ["paperclip.dbtest", "paperclip.escape"]) {
|
||||
for (const pluginKey of ["paperclip.dbtest", "paperclip.escape", "paperclip.refresh", multiMigrationPluginKey]) {
|
||||
const namespace = derivePluginDatabaseNamespace(pluginKey);
|
||||
await db.execute(sql.raw(`DROP SCHEMA IF EXISTS "${namespace}" CASCADE`));
|
||||
}
|
||||
@@ -120,6 +122,31 @@ describeEmbeddedPostgres("plugin database namespaces", () => {
|
||||
return packageRoot;
|
||||
}
|
||||
|
||||
async function createInstallablePluginPackage(
|
||||
pluginManifest: PaperclipPluginManifestV1,
|
||||
migrationSql: string,
|
||||
) {
|
||||
const packageRoot = await createPluginPackage(pluginManifest, migrationSql);
|
||||
await writeFile(
|
||||
path.join(packageRoot, "package.json"),
|
||||
JSON.stringify({
|
||||
name: pluginManifest.id,
|
||||
version: pluginManifest.version,
|
||||
type: "module",
|
||||
paperclipPlugin: { manifest: "./manifest.js" },
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
await writeFile(
|
||||
path.join(packageRoot, "manifest.js"),
|
||||
`export default ${JSON.stringify(pluginManifest, null, 2)};\n`,
|
||||
"utf8",
|
||||
);
|
||||
await mkdir(path.join(packageRoot, "dist"), { recursive: true });
|
||||
await writeFile(path.join(packageRoot, "dist", "worker.js"), "export {};\n", "utf8");
|
||||
return packageRoot;
|
||||
}
|
||||
|
||||
async function installPluginRecord(manifest: PaperclipPluginManifestV1) {
|
||||
const pluginId = randomUUID();
|
||||
await db.insert(plugins).values({
|
||||
@@ -158,6 +185,31 @@ describeEmbeddedPostgres("plugin database namespaces", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it("applies multi-file plugin migrations through the production validator", async () => {
|
||||
const pluginManifest = manifest(multiMigrationPluginKey);
|
||||
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
||||
const packageRoot = await createPluginPackage(
|
||||
pluginManifest,
|
||||
`CREATE TABLE ${namespace}.source_rows (id uuid PRIMARY KEY, label text NOT NULL);`,
|
||||
);
|
||||
await writeFile(
|
||||
path.join(packageRoot, pluginManifest.database!.migrationsDir, "002_derived.sql"),
|
||||
`CREATE TABLE ${namespace}.derived_rows (
|
||||
id uuid PRIMARY KEY,
|
||||
source_id uuid NOT NULL REFERENCES ${namespace}.source_rows(id)
|
||||
);`,
|
||||
"utf8",
|
||||
);
|
||||
const pluginId = await installPluginRecord(pluginManifest);
|
||||
await pluginDatabaseService(db).applyMigrations(pluginId, pluginManifest, packageRoot);
|
||||
|
||||
const migrations = await db
|
||||
.select()
|
||||
.from(pluginMigrations)
|
||||
.where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.status, "applied")));
|
||||
expect(migrations).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("applies migrations once and allows whitelisted core joins at runtime", async () => {
|
||||
const pluginManifest = manifest();
|
||||
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
||||
@@ -246,6 +298,131 @@ describeEmbeddedPostgres("plugin database namespaces", () => {
|
||||
expect(migration?.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("rolls back plugin install when migration validation fails", async () => {
|
||||
const pluginManifest = manifest("paperclip.escape");
|
||||
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
||||
const packageRoot = await createInstallablePluginPackage(
|
||||
pluginManifest,
|
||||
"CREATE TABLE public.plugin_escape (id uuid PRIMARY KEY);",
|
||||
);
|
||||
const loader = pluginLoader(db, {
|
||||
enableLocalFilesystem: false,
|
||||
enableNpmDiscovery: false,
|
||||
});
|
||||
|
||||
await expect(loader.installPlugin({ localPath: packageRoot }))
|
||||
.rejects.toThrow(/public\.plugin_escape|public/i);
|
||||
|
||||
const installedPlugins = await db
|
||||
.select()
|
||||
.from(plugins)
|
||||
.where(eq(plugins.pluginKey, pluginManifest.id));
|
||||
const namespaces = await db
|
||||
.select()
|
||||
.from(pluginDatabaseNamespaces)
|
||||
.where(eq(pluginDatabaseNamespaces.pluginKey, pluginManifest.id));
|
||||
const migrations = await db
|
||||
.select()
|
||||
.from(pluginMigrations)
|
||||
.where(eq(pluginMigrations.pluginKey, pluginManifest.id));
|
||||
const schemaRows = Array.from(
|
||||
await db.execute(
|
||||
sql<{ schema_name: string }>`SELECT schema_name FROM information_schema.schemata WHERE schema_name = ${namespace}`,
|
||||
) as Iterable<{ schema_name: string }>,
|
||||
);
|
||||
|
||||
expect(installedPlugins).toHaveLength(0);
|
||||
expect(namespaces).toHaveLength(0);
|
||||
expect(migrations).toHaveLength(0);
|
||||
expect(schemaRows).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("refreshes persisted manifests from disk before activation", async () => {
|
||||
const staleManifest = manifest("paperclip.refresh");
|
||||
const refreshedManifest: PaperclipPluginManifestV1 = {
|
||||
...staleManifest,
|
||||
database: {
|
||||
...staleManifest.database!,
|
||||
coreReadTables: ["companies"],
|
||||
},
|
||||
};
|
||||
const namespace = derivePluginDatabaseNamespace(refreshedManifest.id);
|
||||
const packageRoot = await createInstallablePluginPackage(
|
||||
refreshedManifest,
|
||||
`
|
||||
CREATE TABLE ${namespace}.company_refs (
|
||||
id uuid PRIMARY KEY,
|
||||
company_id uuid NOT NULL REFERENCES public.companies(id)
|
||||
);
|
||||
`,
|
||||
);
|
||||
const pluginId = await installPluginRecord(staleManifest);
|
||||
await db
|
||||
.update(plugins)
|
||||
.set({
|
||||
packagePath: packageRoot,
|
||||
status: "ready",
|
||||
})
|
||||
.where(eq(plugins.id, pluginId));
|
||||
|
||||
const workerManager = {
|
||||
startWorker: vi.fn().mockResolvedValue(undefined),
|
||||
stopAll: vi.fn().mockResolvedValue(undefined),
|
||||
};
|
||||
const loader = pluginLoader(db, {
|
||||
enableLocalFilesystem: false,
|
||||
enableNpmDiscovery: false,
|
||||
}, {
|
||||
workerManager,
|
||||
eventBus: {
|
||||
forPlugin: vi.fn(() => ({})),
|
||||
subscriptionCount: vi.fn(() => 0),
|
||||
},
|
||||
jobScheduler: {
|
||||
registerPlugin: vi.fn().mockResolvedValue(undefined),
|
||||
stop: vi.fn(),
|
||||
},
|
||||
jobStore: {
|
||||
syncJobDeclarations: vi.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
toolDispatcher: {
|
||||
registerPluginTools: vi.fn(),
|
||||
},
|
||||
lifecycleManager: {
|
||||
markError: vi.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
buildHostHandlers: vi.fn(() => ({})),
|
||||
instanceInfo: {
|
||||
instanceId: "test-instance",
|
||||
hostVersion: "1.0.0",
|
||||
deploymentMode: "authenticated",
|
||||
deploymentExposure: "public",
|
||||
},
|
||||
} as never);
|
||||
|
||||
const result = await loader.loadSingle(pluginId);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(workerManager.startWorker).toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
expect.objectContaining({
|
||||
databaseNamespace: namespace,
|
||||
env: {
|
||||
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
||||
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
||||
},
|
||||
manifest: expect.objectContaining({
|
||||
database: expect.objectContaining({ coreReadTables: ["companies"] }),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
const [plugin] = await db
|
||||
.select()
|
||||
.from(plugins)
|
||||
.where(eq(plugins.id, pluginId));
|
||||
expect(plugin?.manifestJson.database?.coreReadTables).toEqual(["companies"]);
|
||||
});
|
||||
|
||||
it("rejects checksum changes for already applied migrations", async () => {
|
||||
const pluginManifest = manifest();
|
||||
const namespace = derivePluginDatabaseNamespace(pluginManifest.id);
|
||||
|
||||
Reference in New Issue
Block a user