diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md new file mode 100644 index 00000000..f724dad2 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/README.md @@ -0,0 +1,165 @@ +# @paperclipai/plugin-kubernetes (alpha) + +First-party Paperclip sandbox-provider plugin for Kubernetes. + +**Alpha:** the default backend (`sandbox-cr`) is built on `kubernetes-sigs/agent-sandbox` v1alpha1 — expect breaking changes as that CRD evolves toward Beta. A stable fallback backend (`job`, using `batch/v1` Job) is available for clusters without agent-sandbox installed, but it does NOT support multi-command exec (paperclip-server's adapter-install pattern requires sandbox-cr). + +## Prerequisites + +### For `sandbox-cr` backend (default, recommended) + +1. A Kubernetes cluster running k8s 1.27+ +2. [`kubernetes-sigs/agent-sandbox`](https://github.com/kubernetes-sigs/agent-sandbox) controller installed in the cluster (alpha — installs the `sandboxes.agents.x-k8s.io/v1alpha1` CRD and controller) +3. Paperclip-server running with access to the cluster (in-cluster via `inCluster: true` or external via `kubeconfig`) + +### For `job` backend (stable fallback) + +1. A Kubernetes cluster running k8s 1.27+ +2. Paperclip-server with cluster access — no additional controllers or CRDs required + +## Installation + +```bash +paperclipai plugin install @paperclipai/plugin-kubernetes +``` + +Or, for local development: + +```bash +paperclipai plugin install --local /path/to/paperclip/packages/plugins/sandbox-providers/kubernetes +``` + +## Backends + +The plugin supports two backend modes, selected via the `backend` config field: + +| Backend | Default | Stability | Multi-command exec | Requires | +|---|---|---|---|---| +| `sandbox-cr` | Yes | Alpha | Yes | `kubernetes-sigs/agent-sandbox` controller | +| `job` | No | Stable | No | Nothing beyond k8s 1.27+ | + +**`sandbox-cr` (default):** Creates a `Sandbox` CR (`agents.x-k8s.io/v1alpha1`) whose controller provisions a long-lived pod running `sleep infinity`. paperclip-server execs individual commands into the running pod — this is the multi-command adapter-install pattern. When you `releaseLease`, the Sandbox CR is deleted and the controller tears down the pod. + +**`job` (stable fallback):** Creates a `batch/v1` Job. The container entrypoint runs once and exits — no multi-command exec possible. Use this when you cannot install agent-sandbox, or when you need strictly stable Kubernetes APIs. Note: paperclip-server's adapter-install pattern will not work in job mode. + +### Migrating from `job` to `sandbox-cr` + +1. Install the agent-sandbox controller: `kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/install.yaml` +2. Update your environment config to set `backend: "sandbox-cr"` (or remove `backend` since `sandbox-cr` is the default) +3. New leases will use the Sandbox CR backend. Existing leases created with `job` mode continue to use job semantics until they are released. + +## Configuration + +Create a `sandbox` environment with `driver: kubernetes`. One of these auth fields is required: + +- `inCluster: true` — use the in-pod ServiceAccount credentials (when paperclip-server runs inside the same cluster). +- `kubeconfig: ` — inline kubeconfig (stored as a company secret). +- `kubeconfigSecretRef: ` — reference to an existing Paperclip secret. + +Common optional fields: + +| Field | Default | Purpose | +|---|---|---| +| `backend` | `"sandbox-cr"` | `sandbox-cr` (alpha, requires agent-sandbox controller) or `job` (stable, one-shot entrypoint). | +| `adapterType` | `"claude_local"` | One of the supported adapter types (claude_local, codex_local, gemini_local, cursor_local, opencode_local, acpx_local, pi_local). Determines runtime image + env keys + egress allow-list. | +| `namespacePrefix` | `"paperclip-"` | Prefix for the per-company tenant namespace. | +| `companySlug` | derived from companyId | Override the auto-derived company slug. | +| `imageRegistry` | (none) | Override the default registry for agent runtime images. | +| `imageAllowList` | `[]` | Glob patterns of allowed `target.imageOverride` values. Empty = no override permitted. | +| `imagePullSecrets` | `[]` | Names of pre-created Docker image pull secrets in the tenant namespace. | +| `egressAllowFqdns` | `[]` | Additional FQDNs (beyond adapter defaults like `api.anthropic.com`). | +| `egressAllowCidrs` | `[]` | Additional CIDRs to allow egress to. | +| `egressMode` | `"standard"` | `standard` (NetworkPolicy + CIDRs, plus public HTTPS fallback when adapter FQDNs are configured) or `cilium` (CiliumNetworkPolicy + exact FQDN allow-list). | +| `runtimeClassName` | (none) | e.g. `kata-fc` for Firecracker-backed microVMs. Cluster must have the RuntimeClass installed. | +| `serviceAccountAnnotations` | `{}` | Annotations applied to per-tenant ServiceAccount (e.g. IRSA `eks.amazonaws.com/role-arn`). | +| `jobTtlSecondsAfterFinished` | `900` | Seconds after a Job completes before garbage-collection. | +| `podActivityDeadlineSec` | `3600` | Hard ceiling on a single run's wall-clock time. | + +Full JSON Schema in `src/manifest.ts`. + +## What gets created in your cluster + +For each company that runs agents (created lazily on first dispatch): + +``` +Namespace paperclip-{companySlug} (PSS: restricted enforce + audit) +ServiceAccount paperclip-tenant-sa +Role paperclip-tenant-role (only get pods/log) +RoleBinding paperclip-tenant-rb +ResourceQuota paperclip-quota (pods, requests/limits cpu+memory) +LimitRange paperclip-limits (container max/min/default/defaultRequest) +NetworkPolicy paperclip-deny-all (deny ingress + egress baseline) +NetworkPolicy paperclip-egress-allow (DNS + paperclip-server callback + user CIDRs + public HTTPS fallback for adapter FQDNs) + OR CiliumNetworkPolicy paperclip-egress-fqdn if egressMode=cilium +``` + +Standard Kubernetes NetworkPolicy cannot match FQDNs. In `egressMode: "standard"`, adapter-default FQDNs such as `api.anthropic.com` trigger a public IPv4 HTTPS fallback that excludes private and link-local ranges, so default agent runs can reach model APIs without opening intra-cluster/private-network egress. Use `egressMode: "cilium"` when you need exact FQDN enforcement. + +For each agent run (sandbox-cr backend): + +``` +Sandbox CR pc-{ulid} (agents.x-k8s.io/v1alpha1; explicit delete on release) +Pod pc-{ulid}-{podSuffix} (managed by Sandbox controller; torn down on CR delete) +Secret pc-{ulid}-env (owned by Sandbox CR; cascade-deleted) +``` + +For each agent run (job backend): + +``` +Job pc-{ulid} (backoffLimit: 0, ttlSecondsAfterFinished from config) +Pod pc-{ulid}-{podSuffix} (owned by Job; cascade-deleted) +Secret pc-{ulid}-env (owned by Job; cascade-deleted) +``` + +## Security baseline + +Every agent pod is: + +- non-root (`runAsUser: 1000`, `runAsGroup: 1000`, `runAsNonRoot: true`) +- drops ALL Linux capabilities, `allowPrivilegeEscalation: false` +- `readOnlyRootFilesystem: true` with explicit `emptyDir` mounts for `/workspace`, `/home/paperclip`, `/home/paperclip/.cache`, `/tmp` +- `seccompProfile: RuntimeDefault` +- Tini as PID 1 (reaps zombies, forwards signals) +- `fsGroupChangePolicy: OnRootMismatch` (fast PVC startup; openclaw-operator lesson) +- `automountServiceAccountToken: false` + +Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, CIDRs, and either Cilium FQDN rules or standard-mode public HTTPS fallback). + +The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Sandbox CR or Job, so releasing the lease cascades cleanly to the Pod and Secret. + +## Optional Kata-FC microVM isolation + +For stronger isolation, install [Kata Containers](https://github.com/kata-containers/kata-containers) with the Firecracker hypervisor, then set `runtimeClassName: kata-fc` in the plugin config. Each agent pod will run inside a Firecracker microVM. Requires nested-virt-capable nodes (bare-metal or specific cloud instance types). + +## Roadmap + +- **Phase A (done):** `sandbox-cr` backend — multi-command exec via agent-sandbox Sandbox CRD. +- **Phase B:** Warm pool support — pre-provisioned Sandbox CRs for sub-second cold starts. The `SandboxOrchestrator` interface reserves optional `pause?`/`resume?` extension slots. +- **Phase C:** Kata-FC + snapshots — `runtimeClassName: kata-fc` with VM snapshot for fast restore. +- **Phase D:** Contribute back to agent-sandbox upstream if their Beta model diverges from our needs. The `SandboxOrchestrator` interface (`src/sandbox-orchestrator.ts`) is the clean swap point — a new implementation can be added without touching `plugin.ts` business logic. + +## Lessons learned (from openclaw-operator) + +This plugin adopts patterns from `openclaw-rocks/openclaw-operator`: + +- Tini PID 1 (issue #471 — zombie helper processes) +- Read-only rootFS with explicit writable mounts (issue #456 — ~/.config not writable) +- Strategic merge on reconcile (issue #446 — preserve third-party annotations) +- Multi-storage-class testing (issue #448 — `local-path-provisioner` differences) +- Image version compat matrix (issue #462 — runtime deps cannot resolve after upgrade) + +## Development + +```bash +cd packages/plugins/sandbox-providers/kubernetes +pnpm install --ignore-workspace +pnpm test # unit tests only (fast) +pnpm typecheck +pnpm build +``` + +To run the kind-cluster integration test (requires `kubectl --context kind-paperclip` and a pre-loaded alpine image; see `test/integration/end-to-end-run.test.ts`): + +```bash +RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts +``` diff --git a/packages/plugins/sandbox-providers/kubernetes/SMOKE.md b/packages/plugins/sandbox-providers/kubernetes/SMOKE.md new file mode 100644 index 00000000..9b23828f --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/SMOKE.md @@ -0,0 +1,135 @@ +# Manual smoke test — `@paperclipai/plugin-kubernetes` + +Manual sanity check that the plugin works end-to-end against a real +paperclip-server instance and a real Kubernetes cluster (kind for local +dev). Future work may automate this in CI. + +## Prerequisites + +- A running kind cluster: + ```bash + kind create cluster --name paperclip + ``` +- `kubectl --context kind-paperclip get nodes` returns a node in `Ready` state. + +## Steps + +### 1. Build the plugin + +```bash +cd packages/plugins/sandbox-providers/kubernetes +pnpm install --ignore-workspace +pnpm build +``` + +Expected: `dist/` populated with compiled `.js` and `.d.ts` files. No errors. + +### 2. Start paperclip-server in dev mode + +In a separate terminal: + +```bash +cd /path/to/paperclip +export PAPERCLIP_HOME=/tmp/paperclip-smoke +export PAPERCLIP_INSTANCE_ID=smoke +export PAPERCLIP_DEPLOYMENT_MODE=local_trusted +pnpm --filter @paperclipai/server dev +``` + +Wait for `Server listening on 127.0.0.1:3100`. + +### 3. Install the plugin via the CLI + +```bash +pnpm paperclipai plugin install \ + --local /path/to/paperclip/packages/plugins/sandbox-providers/kubernetes \ + --api-base http://127.0.0.1:3100 +``` + +Expected: `✓ Installed paperclip.kubernetes-sandbox-provider v0.1.0 (ready)`. + +### 4. Create a company and a kubernetes sandbox environment + +```bash +CO_ID=$(curl -s -X POST -H "Content-Type: application/json" \ + -d '{"name":"SmokeCo"}' \ + http://127.0.0.1:3100/api/companies | jq -r '.id') + +KUBECONFIG_CONTENT=$(cat ~/.kube/config | jq -Rs .) + +curl -s -X POST -H "Content-Type: application/json" \ + -d "{ + \"name\": \"k8s-sandbox\", + \"driver\": \"sandbox\", + \"config\": { + \"provider\": \"kubernetes\", + \"kubeconfig\": $KUBECONFIG_CONTENT, + \"companySlug\": \"smoke\", + \"adapterType\": \"claude_local\", + \"imageAllowList\": [\"ghcr.io/paperclipai/agent-runtime-claude:v1\"] + } + }" \ + http://127.0.0.1:3100/api/companies/$CO_ID/environments | jq +``` + +Expected: HTTP 201 with the new environment row. + +### 5. Probe the environment + +```bash +ENV_ID=$(curl -s http://127.0.0.1:3100/api/companies/$CO_ID/environments | jq -r '.[0].id') +curl -s -X POST -d '{}' -H "Content-Type: application/json" \ + http://127.0.0.1:3100/api/environments/$ENV_ID/probe | jq +``` + +Expected: `{"ok": true, ...}` with a summary mentioning the tenant namespace +(`paperclip-smoke`). On first probe the namespace may not yet exist — +the plugin treats a 404 on `listNamespacedPod` as a successful reachability +check. + +### 6. Trigger an agent run + +Use the UI or the API to dispatch a run against the `k8s-sandbox` environment. +The plugin's `onEnvironmentAcquireLease` will: + +1. `ensureTenant` — provision the `paperclip-smoke` namespace, SA, Role, + RoleBinding, ResourceQuota, LimitRange, NetworkPolicies +2. `buildJobManifest` — render the security-hardened Job manifest +3. `createJob` — submit to `batch/v1` +4. `createPerRunSecret` — owned by the Job for cascade-delete + +### 7. Verify the tenant resources + +```bash +kubectl --context kind-paperclip get namespace paperclip-smoke +kubectl --context kind-paperclip get all,networkpolicy,resourcequota,limitrange,sa,role,rolebinding -n paperclip-smoke +``` + +Expected: + +- Namespace `paperclip-smoke` exists with PSS labels + (`pod-security.kubernetes.io/enforce=restricted`) +- ServiceAccount `paperclip-tenant-sa` +- Role `paperclip-tenant-role`, RoleBinding `paperclip-tenant-rb` +- ResourceQuota `paperclip-quota`, LimitRange `paperclip-limits` +- NetworkPolicies `paperclip-deny-all` + `paperclip-egress-allow` +- Job `pc-{ulid}` and its child Pod +- Secret `pc-{ulid}-env` with `ownerReferences` pointing at the Job + +### 8. Tear down + +```bash +kubectl --context kind-paperclip delete namespace paperclip-smoke +kill %1 # paperclip-server +``` + +### 9. Document the result + +In the PR description (or appended to this file as a dated section), +record: + +- Date + git SHA +- `kubectl version` server version +- Output of `kubectl get all -n paperclip-smoke` after step 6 +- Probe response from step 5 +- Time-to-acquire-lease (target: <30s on kind for a cold tenant) diff --git a/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml b/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml new file mode 100644 index 00000000..1cb9c96a --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml @@ -0,0 +1,22 @@ +# This plugin uses only stable Kubernetes APIs. No CRD installation is required. +# +# Minimum cluster version: Kubernetes 1.27+ +# - batch/v1 Job (GA since k8s 1.21) +# - core/v1 Pod, Secret, Namespace, ServiceAccount, ResourceQuota, LimitRange (GA since k8s 1.0) +# - rbac.authorization.k8s.io/v1 Role, RoleBinding (GA since k8s 1.8) +# - networking.k8s.io/v1 NetworkPolicy (GA since k8s 1.7) +# - Pod Security Standards namespace labels (GA in k8s 1.25) +# - fsGroupChangePolicy: OnRootMismatch (GA in k8s 1.23) +# - seccompProfile.type: RuntimeDefault (GA in k8s 1.19) +# +# Optional CNI prerequisites for FQDN-based egress (egressMode: cilium): +# - Cilium >= 1.11 with hubble + DNS proxy enabled +# - cilium.io/v2 CiliumNetworkPolicy (provided by Cilium installation) +# +# Optional runtime class for microVM isolation (runtimeClassName: kata-fc): +# - kata-containers with Firecracker hypervisor +# - nested-virt-capable nodes +# +# Future backends (not currently required): +# - kubernetes-sigs/agent-sandbox (when it reaches v1beta1) as an alternative +# backend for warm pools / templates / pause-resume. diff --git a/packages/plugins/sandbox-providers/kubernetes/package.json b/packages/plugins/sandbox-providers/kubernetes/package.json new file mode 100644 index 00000000..4f6fdef6 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/package.json @@ -0,0 +1,60 @@ +{ + "name": "@paperclipai/plugin-kubernetes", + "version": "0.1.0", + "description": "Kubernetes sandbox provider plugin for Paperclip environments", + "license": "MIT", + "homepage": "https://github.com/paperclipai/paperclip", + "bugs": { + "url": "https://github.com/paperclipai/paperclip/issues" + }, + "repository": { + "type": "git", + "url": "https://github.com/paperclipai/paperclip", + "directory": "packages/plugins/sandbox-providers/kubernetes" + }, + "type": "module", + "exports": { + ".": "./src/index.ts" + }, + "publishConfig": { + "access": "public", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + } + }, + "main": "./dist/index.js", + "types": "./dist/index.d.ts" + }, + "files": ["dist", "manifests", "README.md"], + "paperclipPlugin": { + "manifest": "./dist/manifest.js", + "worker": "./dist/worker.js" + }, + "keywords": [ + "paperclip", + "plugin", + "sandbox", + "kubernetes" + ], + "scripts": { + "postinstall": "node ../../../../scripts/link-plugin-dev-sdk.mjs", + "prebuild": "pnpm -C ../../../.. --filter @paperclipai/plugin-sdk ensure-build-deps", + "build": "rm -rf dist && tsc", + "clean": "rm -rf dist", + "typecheck": "pnpm -C ../../../.. --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit", + "test": "vitest run --config vitest.config.ts", + "prepack": "rm -f package.dev.json && cp package.json package.dev.json && node ../../../../scripts/generate-plugin-package-json.mjs", + "postpack": "if [ -f package.dev.json ]; then mv package.dev.json package.json; fi" + }, + "dependencies": { + "@kubernetes/client-node": "^1.0.0", + "zod": "^3.24.2" + }, + "devDependencies": { + "@types/node": "^24.6.0", + "typescript": "^5.7.3", + "vitest": "^3.2.4" + } +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts b/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts new file mode 100644 index 00000000..149cb0d0 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts @@ -0,0 +1,61 @@ +export interface AdapterDefaults { + runtimeImage: string; + envKeys: string[]; + allowFqdns: string[]; + probeCommand: string[]; +} + +const REGISTRY: Record = { + claude_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-claude:v1", + envKeys: ["ANTHROPIC_API_KEY"], + allowFqdns: ["api.anthropic.com"], + probeCommand: ["claude", "--version"], + }, + codex_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-codex:v1", + envKeys: ["OPENAI_API_KEY"], + allowFqdns: ["api.openai.com"], + probeCommand: ["codex", "--version"], + }, + gemini_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-gemini:v1", + envKeys: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], + allowFqdns: ["generativelanguage.googleapis.com"], + probeCommand: ["gemini", "--version"], + }, + cursor_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-cursor:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com"], + probeCommand: ["cursor-agent", "--version"], + }, + opencode_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-opencode:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com", "openrouter.ai"], + probeCommand: ["opencode", "--version"], + }, + acpx_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-acpx:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com"], + probeCommand: ["acpx", "--version"], + }, + pi_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-pi:v1", + envKeys: ["ANTHROPIC_API_KEY"], + allowFqdns: ["api.anthropic.com"], + probeCommand: ["pi", "--version"], + }, +}; + +export const KNOWN_ADAPTER_TYPES: ReadonlySet = new Set(Object.keys(REGISTRY)); + +export function getAdapterDefaults(adapterType: string): AdapterDefaults { + const defaults = REGISTRY[adapterType]; + if (!defaults) { + throw new Error(`Unknown adapter type: ${adapterType}`); + } + return defaults; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts b/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts new file mode 100644 index 00000000..5dedcf73 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts @@ -0,0 +1,68 @@ +export interface BuildCiliumNetworkPolicyInput { + namespace: string; + paperclipServerNamespace: string; + egressAllowFqdns: string[]; + egressAllowCidrs: string[]; +} + +// Design note: no ingress rules are defined here. Paperclip-server does NOT +// push to agent pods — agents make outbound (egress) callbacks to +// paperclip-server on port 3100. If server→agent push is ever needed, add a +// targeted ingress rule scoped to the paperclip-server endpoint selector. +export function buildCiliumNetworkPolicyManifest(input: BuildCiliumNetworkPolicyInput): Record { + const egress: Record[] = []; + + egress.push({ + toEndpoints: [ + { matchLabels: { "k8s:io.kubernetes.pod.namespace": "kube-system", "k8s-app": "kube-dns" } }, + ], + toPorts: [ + { + ports: [ + { port: "53", protocol: "UDP" }, + { port: "53", protocol: "TCP" }, + ], + rules: { dns: [{ matchPattern: "*" }] }, + }, + ], + }); + + if (input.egressAllowFqdns.length > 0) { + egress.push({ + toFQDNs: input.egressAllowFqdns.map((fqdn) => ({ matchName: fqdn })), + toPorts: [{ ports: [{ port: "443", protocol: "TCP" }] }], + }); + } + + egress.push({ + toEndpoints: [ + { + matchLabels: { + "k8s:io.kubernetes.pod.namespace": input.paperclipServerNamespace, + app: "paperclip-server", + }, + }, + ], + toPorts: [{ ports: [{ port: "3100", protocol: "TCP" }] }], + }); + + if (input.egressAllowCidrs.length > 0) { + egress.push({ + toCIDRSet: input.egressAllowCidrs.map((cidr) => ({ cidr })), + }); + } + + return { + apiVersion: "cilium.io/v2", + kind: "CiliumNetworkPolicy", + metadata: { + name: "paperclip-egress-fqdn", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + endpointSelector: { matchLabels: { "paperclip.io/role": "agent" } }, + egress, + }, + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts b/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts new file mode 100644 index 00000000..46cdaf61 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts @@ -0,0 +1,59 @@ +/** + * Glob matching for image references. + * - `*` matches any sequence of characters EXCEPT `/` (so a wildcard doesn't span path segments) + * - `?` matches exactly one character (excluding `/`) + */ +export function globMatch(pattern: string, value: string): boolean { + const re = new RegExp( + "^" + + pattern + .replace(/[.+^${}()|[\]\\]/g, "\\$&") + .replace(/\*/g, "[^/]*") + .replace(/\?/g, "[^/]") + + "$", + ); + return re.test(value); +} + +export interface ResolveImageInput { + imageOverride?: string | null; +} + +export interface ResolveImageDefaults { + runtimeImage: string; +} + +export interface ResolveImageConfig { + imageAllowList: string[]; + imageRegistry?: string; +} + +export function resolveImage( + target: ResolveImageInput, + defaults: ResolveImageDefaults, + config: ResolveImageConfig, +): string { + if (target.imageOverride) { + if (!config.imageAllowList.some((p) => globMatch(p, target.imageOverride!))) { + throw new Error(`Image override "${target.imageOverride}" is not in allowlist`); + } + return target.imageOverride; + } + if (config.imageRegistry) { + return rewriteRegistry(defaults.runtimeImage, config.imageRegistry); + } + return defaults.runtimeImage; +} + +function rewriteRegistry(image: string, registry: string): string { + // image is like "ghcr.io/paperclipai/agent-runtime-claude:v1" + // we want to replace the first two path segments (host + org) with `registry` + const cleanRegistry = registry.replace(/\/+$/, ""); + const colonIdx = image.lastIndexOf(":"); + const tag = colonIdx >= 0 ? image.slice(colonIdx) : ""; + const path = colonIdx >= 0 ? image.slice(0, colonIdx) : image; + const segments = path.split("/"); + // Strip the host+org (first two segments), keep the image name + const imageName = segments.slice(2).join("/") || segments[segments.length - 1]; + return `${cleanRegistry}/${imageName}${tag}`; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/index.ts b/packages/plugins/sandbox-providers/kubernetes/src/index.ts new file mode 100644 index 00000000..f7ce1cc1 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/index.ts @@ -0,0 +1,2 @@ +export { default as manifest } from "./manifest.js"; +export { default as plugin } from "./plugin.js"; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts b/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts new file mode 100644 index 00000000..498302e4 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts @@ -0,0 +1,129 @@ +import type { KubeClients } from "./kube-client.js"; +import type { SandboxOrchestrator, SandboxStatus } from "./sandbox-orchestrator.js"; + +export class JobTimeoutError extends Error { + constructor(namespace: string, name: string, timeoutMs: number) { + super(`Job ${namespace}/${name} did not complete within ${timeoutMs}ms`); + this.name = "JobTimeoutError"; + } +} + +export async function createJob( + clients: KubeClients, + namespace: string, + manifest: Record, +): Promise<{ uid: string }> { + const result = await clients.batch.createNamespacedJob({ namespace, body: manifest as never }); + const uid = (result as { metadata?: { uid?: string } }).metadata?.uid; + if (!uid) throw new Error("Job created without a UID"); + return { uid }; +} + +export type JobStatus = SandboxStatus; + +export async function getJobStatus( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + const result = await clients.batch.readNamespacedJobStatus({ namespace, name }); + const body = (result as Record) ?? {}; + const status = (body.status as Record) ?? {}; + const active = (status.active as number) ?? 0; + const succeeded = (status.succeeded as number) ?? 0; + const failed = (status.failed as number) ?? 0; + const conditions = (status.conditions as { type: string; status: string; reason?: string; message?: string }[]) ?? []; + const completed = conditions.find((c) => c.type === "Complete" && c.status === "True"); + const failedCond = conditions.find((c) => c.type === "Failed" && c.status === "True"); + if (failedCond || failed > 0) { + return { phase: "Failed", complete: false, active, succeeded, failed, reason: failedCond?.reason, message: failedCond?.message }; + } + if (completed || succeeded > 0) { + return { phase: "Succeeded", complete: true, active, succeeded, failed }; + } + if (active > 0) { + return { phase: "Running", complete: false, active, succeeded, failed }; + } + return { phase: "Pending", complete: false, active, succeeded, failed }; +} + +export async function findPodForJob( + clients: KubeClients, + namespace: string, + jobName: string, +): Promise { + const result = await clients.core.listNamespacedPod({ + namespace, + labelSelector: `job-name=${jobName}`, + }); + const items = ((result as { items?: { metadata?: { name?: string }; status?: { phase?: string } }[] }).items) ?? []; + const running = items.find((p) => p.status?.phase === "Running"); + return (running ?? items[0])?.metadata?.name ?? null; +} + +export async function streamPodLogs( + clients: KubeClients, + namespace: string, + podName: string, + onChunk: (stream: "stdout" | "stderr", text: string) => Promise, +): Promise { + // V1 limitation: the Pod log API returns the container's combined log stream. + // Kubernetes does not preserve stdout/stderr channel separation after the + // container runtime writes logs, so the Job backend reports combined logs on + // stdout. The sandbox-cr backend uses exec and keeps streams separate. + const result = await clients.core.readNamespacedPodLog({ namespace, name: podName }); + const text = readPodLogText(result); + if (text.length > 0) await onChunk("stdout", text); +} + +function readPodLogText(result: unknown): string { + if (typeof result === "string") return result; + const body = (result as { body?: unknown })?.body; + return typeof body === "string" ? body : ""; +} + +export async function deleteJob( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + await clients.batch.deleteNamespacedJob({ + namespace, + name, + propagationPolicy: "Foreground", + }); +} + +export async function waitForJobCompletion( + clients: KubeClients, + namespace: string, + name: string, + opts: { timeoutMs: number; pollMs?: number } = { timeoutMs: 120_000, pollMs: 2000 }, +): Promise { + const deadline = Date.now() + opts.timeoutMs; + const pollMs = opts.pollMs ?? 2000; + while (Date.now() < deadline) { + const status = await getJobStatus(clients, namespace, name); + if (status.phase === "Succeeded" || status.phase === "Failed") return status; + await sleep(pollMs); + } + throw new JobTimeoutError(namespace, name, opts.timeoutMs); +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** + * Job-backed conformance to SandboxOrchestrator. Plugin.ts imports THIS value + * (the swap point) — to use a different backend, swap this import for another + * module exposing a SandboxOrchestrator-shaped default export. + */ +export const jobOrchestrator: SandboxOrchestrator = { + claim: createJob, + getStatus: getJobStatus, + findPod: findPodForJob, + streamLogs: streamPodLogs, + release: deleteJob, + waitForCompletion: waitForJobCompletion, +}; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts b/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts new file mode 100644 index 00000000..d66a9fc4 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts @@ -0,0 +1,44 @@ +import { + KubeConfig, + CoreV1Api, + BatchV1Api, + CustomObjectsApi, + NetworkingV1Api, + RbacAuthorizationV1Api, +} from "@kubernetes/client-node"; + +export interface CreateKubeConfigInput { + inCluster?: boolean; + kubeconfig?: string; +} + +export function createKubeConfig(input: CreateKubeConfigInput): KubeConfig { + const kc = new KubeConfig(); + if (input.inCluster) { + kc.loadFromCluster(); + return kc; + } + if (input.kubeconfig && input.kubeconfig.trim().length > 0) { + kc.loadFromString(input.kubeconfig); + return kc; + } + throw new Error("createKubeConfig requires either inCluster=true or a kubeconfig string"); +} + +export interface KubeClients { + core: CoreV1Api; + batch: BatchV1Api; + custom: CustomObjectsApi; + networking: NetworkingV1Api; + rbac: RbacAuthorizationV1Api; +} + +export function makeKubeClients(kc: KubeConfig): KubeClients { + return { + core: kc.makeApiClient(CoreV1Api), + batch: kc.makeApiClient(BatchV1Api), + custom: kc.makeApiClient(CustomObjectsApi), + networking: kc.makeApiClient(NetworkingV1Api), + rbac: kc.makeApiClient(RbacAuthorizationV1Api), + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts b/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts new file mode 100644 index 00000000..da2ff88c --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts @@ -0,0 +1,122 @@ +import type { PaperclipPluginManifestV1 } from "@paperclipai/plugin-sdk"; + +const PLUGIN_ID = "paperclip.kubernetes-sandbox-provider"; +const PLUGIN_VERSION = "0.1.0-alpha.1"; + +const manifest: PaperclipPluginManifestV1 = { + id: PLUGIN_ID, + apiVersion: 1, + version: PLUGIN_VERSION, + displayName: "Kubernetes Sandbox (alpha)", + description: + "Built on kubernetes-sigs/agent-sandbox (v1alpha1). ALPHA — expect breaking changes as the upstream CRD evolves. Falls back to stable batch/v1 Job mode for clusters without agent-sandbox installed. First-party Paperclip sandbox-provider plugin for Kubernetes.", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { + worker: "./dist/worker.js", + }, + environmentDrivers: [ + { + driverKey: "kubernetes", + kind: "sandbox_provider", + displayName: "Kubernetes", + description: + "Dispatches agent runs in per-tenant Kubernetes namespaces. Default backend (sandbox-cr, alpha) uses kubernetes-sigs/agent-sandbox for multi-command exec; fallback backend (job) uses stable batch/v1 Job for clusters without agent-sandbox installed.", + configSchema: { + type: "object", + properties: { + inCluster: { + type: "boolean", + description: + "When true, the plugin uses the in-pod ServiceAccount credentials. Requires paperclip-server to be running inside the target cluster.", + }, + kubeconfig: { + type: "string", + format: "secret-ref", + description: + "Inline kubeconfig YAML. Paste a kubeconfig or an existing Paperclip secret reference; pasted values are stored as company secrets.", + }, + namespacePrefix: { + type: "string", + description: "Prefix for the per-company tenant namespace (default: paperclip-).", + }, + companySlug: { + type: "string", + description: "Override the auto-derived company slug used in the tenant namespace name.", + }, + imageRegistry: { + type: "string", + description: "Override the default registry for agent runtime images (default: ghcr.io/paperclipai).", + }, + imageAllowList: { + type: "array", + items: { type: "string" }, + description: + "Glob patterns of allowed `target.imageOverride` values. Empty list = no override permitted.", + }, + imagePullSecrets: { + type: "array", + items: { type: "string" }, + description: "Names of pre-created Docker image pull secrets in the tenant namespace.", + }, + egressAllowFqdns: { + type: "array", + items: { type: "string" }, + description: + "Additional FQDNs to allow egress to from agent pods. Adapter-default FQDNs (e.g. api.anthropic.com) are added automatically.", + }, + egressAllowCidrs: { + type: "array", + items: { type: "string" }, + description: "Additional CIDRs to allow egress to from agent pods.", + }, + egressMode: { + type: "string", + enum: ["standard", "cilium"], + description: + "Network policy mode. `standard` uses NetworkPolicy and allows public HTTPS when adapter FQDNs are configured; `cilium` enables exact FQDN egress filtering via CiliumNetworkPolicy.", + }, + runtimeClassName: { + type: "string", + description: + "Optional RuntimeClass for pod isolation (e.g. `kata-fc` for Firecracker-backed microVMs). Cluster must have the RuntimeClass installed.", + }, + serviceAccountAnnotations: { + type: "object", + additionalProperties: { type: "string" }, + description: + "Annotations applied to the per-tenant ServiceAccount (e.g. `eks.amazonaws.com/role-arn` for IRSA).", + }, + jobTtlSecondsAfterFinished: { + type: "integer", + minimum: 0, + description: "Seconds after a Job completes before it is garbage-collected (default: 900).", + }, + podActivityDeadlineSec: { + type: "integer", + minimum: 1, + description: "Hard ceiling on a single run's wall-clock time (default: 3600).", + }, + adapterType: { + type: "string", + description: + "The adapter type that Jobs in this environment will run (e.g. `claude_local`, `codex_local`). Defaults to `claude_local`. Each environment is bound to one adapter; create multiple environments for different adapters.", + }, + backend: { + type: "string", + enum: ["sandbox-cr", "job"], + description: + "sandbox-cr (default, alpha — requires kubernetes-sigs/agent-sandbox installed) | job (stable fallback — batch/v1 Job, one-shot entrypoint, no multi-command exec)", + }, + }, + anyOf: [ + { required: ["inCluster"] }, + { required: ["kubeconfig"] }, + ], + }, + }, + ], +}; + +export default manifest; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts b/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts new file mode 100644 index 00000000..c13652c0 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts @@ -0,0 +1,101 @@ +export interface BuildNetworkPolicyInput { + namespace: string; + paperclipServerNamespace: string; + egressAllowFqdns: string[]; + egressAllowCidrs: string[]; +} + +const PUBLIC_IPV4_EXCEPTIONS = [ + "10.0.0.0/8", + "100.64.0.0/10", + "127.0.0.0/8", + "169.254.0.0/16", + "172.16.0.0/12", + "192.168.0.0/16", +]; + +// Design note: the deny-all baseline blocks all ingress to agent pods. +// Paperclip-server does NOT push to agent pods — the agent shim makes +// outbound calls to paperclip-server via the egress allow-list (port 3100). +// This pull/callback model means no ingress rule is needed. If a future +// feature requires server→agent push (e.g. forced shutdown, live exec), +// add a targeted ingress rule here scoped to the paperclip-server pod +// selector. +// +// Standard Kubernetes NetworkPolicy cannot express FQDN allow-lists. When +// adapter defaults require FQDN egress, keep runs functional by allowing public +// IPv4 HTTPS while excluding private/link-local ranges. Operators who need +// exact FQDN enforcement should use egressMode="cilium". +export function buildNetworkPolicyManifests(input: BuildNetworkPolicyInput): Record[] { + const fqdnsRequirePublicHttpsFallback = input.egressAllowFqdns.length > 0; + const denyAll = { + apiVersion: "networking.k8s.io/v1", + kind: "NetworkPolicy", + metadata: { + name: "paperclip-deny-all", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + podSelector: {}, + policyTypes: ["Ingress", "Egress"], + }, + }; + + const egressAllow: Record = { + apiVersion: "networking.k8s.io/v1", + kind: "NetworkPolicy", + metadata: { + name: "paperclip-egress-allow", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + podSelector: { matchLabels: { "paperclip.io/role": "agent" } }, + policyTypes: ["Egress"], + egress: [ + { + to: [ + { + namespaceSelector: { matchLabels: { "kubernetes.io/metadata.name": "kube-system" } }, + podSelector: { matchLabels: { "k8s-app": "kube-dns" } }, + }, + ], + ports: [ + { protocol: "UDP", port: 53 }, + { protocol: "TCP", port: 53 }, + ], + }, + { + to: [ + { + namespaceSelector: { matchLabels: { "kubernetes.io/metadata.name": input.paperclipServerNamespace } }, + podSelector: { matchLabels: { app: "paperclip-server" } }, + }, + ], + ports: [{ protocol: "TCP", port: 3100 }], + }, + ...(fqdnsRequirePublicHttpsFallback + ? [ + { + to: [ + { + ipBlock: { + cidr: "0.0.0.0/0", + except: PUBLIC_IPV4_EXCEPTIONS, + }, + }, + ], + ports: [{ protocol: "TCP", port: 443 }], + }, + ] + : []), + ...input.egressAllowCidrs.map((cidr) => ({ + to: [{ ipBlock: { cidr } }], + })), + ], + }, + }; + + return [denyAll, egressAllow]; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts new file mode 100644 index 00000000..a529e961 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts @@ -0,0 +1,554 @@ +import { randomBytes } from "node:crypto"; +import { definePlugin } from "@paperclipai/plugin-sdk"; +import type { + PluginEnvironmentAcquireLeaseParams, + PluginEnvironmentExecuteParams, + PluginEnvironmentExecuteResult, + PluginEnvironmentLease, + PluginEnvironmentProbeParams, + PluginEnvironmentProbeResult, + PluginEnvironmentRealizeWorkspaceParams, + PluginEnvironmentRealizeWorkspaceResult, + PluginEnvironmentReleaseLeaseParams, + PluginEnvironmentValidateConfigParams, + PluginEnvironmentValidationResult, +} from "@paperclipai/plugin-sdk"; +import { + kubernetesProviderConfigSchema, + type KubernetesProviderConfig, + type KubernetesLeaseMetadata, +} from "./types.js"; +import { createKubeConfig, makeKubeClients } from "./kube-client.js"; +import { getAdapterDefaults } from "./adapter-defaults.js"; +import { resolveImage } from "./image-allowlist.js"; +import { buildJobManifest } from "./pod-spec-builder.js"; +import { buildSandboxCrManifest } from "./sandbox-cr-builder.js"; +import { ensureTenant } from "./tenant-orchestrator.js"; +import { createPerRunSecret } from "./secret-manager.js"; +import { jobOrchestrator, JobTimeoutError } from "./job-orchestrator.js"; +import { + sandboxCrOrchestrator, + SandboxCrTimeoutError, +} from "./sandbox-cr-orchestrator.js"; +import { execInPod } from "./pod-exec.js"; +import { + deriveCompanySlug, + deriveNamespaceName, + newRunUlidDns, + paperclipLabels, +} from "./utils.js"; + +// The namespace paperclip-server itself runs in. Used when building +// NetworkPolicy manifests so the tenant namespace allows inbound traffic +// from the server pod. +const PAPERCLIP_SERVER_NAMESPACE = "paperclip"; + +// Name of the ServiceAccount created inside each tenant namespace by ensureTenant. +const TENANT_SERVICE_ACCOUNT = "paperclip-tenant-sa"; + +// Resource quota defaults applied to every tenant namespace (M4b; tunable via +// config in a future milestone). +const DEFAULT_RESOURCE_QUOTA = { + pods: "20", + requestsCpu: "10", + requestsMemory: "20Gi", + limitsCpu: "20", + limitsMemory: "40Gi", +}; + +function deriveTenantNamespace(config: KubernetesProviderConfig, companyId: string): string { + // TODO: future versions could thread companyName through AcquireLeaseParams + // to get a friendlier slug (e.g. "acme-corp") instead of the UUID-derived one. + const slug = config.companySlug ?? deriveCompanySlug(companyId); + return deriveNamespaceName(config.namespacePrefix, slug); +} + +/** + * Reads adapter env keys (e.g. ANTHROPIC_API_KEY) from the current process + * environment. The plugin worker runs inside paperclip-server's pod, which has + * these vars injected at deploy time. + * + * M4b approach: env vars sourced from process.env at acquire time. + * TODO: future milestones may thread per-run secrets differently (e.g. via + * a secret store reference on the environment config). + */ +function extractAdapterEnvFromProcess(envKeys: string[]): Record { + const out: Record = {}; + for (const k of envKeys) { + const v = process.env[k]; + if (v) out[k] = v; + } + return out; +} + +function generateBootstrapToken(): string { + // TODO: paperclip-server's actual callback auth scheme is separate and is + // out of M4b scope. This per-run random token is stored in the per-run + // Secret and consumed by paperclip-agent-shim for initial registration. + return randomBytes(32).toString("hex"); +} + +const plugin = definePlugin({ + async setup(ctx) { + ctx.logger.info("Kubernetes sandbox provider plugin ready"); + }, + + async onHealth() { + return { status: "ok", message: "Kubernetes sandbox provider plugin healthy" }; + }, + + async onEnvironmentValidateConfig( + params: PluginEnvironmentValidateConfigParams, + ): Promise { + const parsed = kubernetesProviderConfigSchema.safeParse(params.config); + if (!parsed.success) { + return { + ok: false, + errors: parsed.error.issues.map((i) => i.message), + }; + } + const warnings: string[] = []; + const cfg = parsed.data; + const adapterDefaults = getAdapterDefaults(cfg.adapterType); + const totalFqdns = [...adapterDefaults.allowFqdns, ...cfg.egressAllowFqdns]; + if (cfg.egressMode === "standard" && totalFqdns.length > 0) { + warnings.push( + `egressMode=standard cannot enforce FQDN-based egress rules for ${totalFqdns.join(", ")}. Agent pods will get public IPv4 HTTPS egress with private/link-local ranges excluded. Switch egressMode to "cilium" for exact FQDN enforcement.`, + ); + } + return { ok: true, normalizedConfig: cfg as Record, warnings: warnings.length > 0 ? warnings : undefined }; + }, + + async onEnvironmentProbe( + params: PluginEnvironmentProbeParams, + ): Promise { + const parsed = kubernetesProviderConfigSchema.safeParse(params.config); + if (!parsed.success) { + return { + ok: false, + summary: "Invalid Kubernetes provider configuration.", + metadata: { + errors: parsed.error.issues.map((i) => i.message), + }, + }; + } + const config = parsed.data; + const namespace = deriveTenantNamespace(config, params.companyId); + + try { + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + // Reachability check: list pods in the tenant namespace. If the namespace + // doesn't exist yet this will throw a 404 which we treat as "reachable + // but namespace not provisioned" — still a successful probe. + try { + await clients.core.listNamespacedPod({ namespace }); + } catch (err) { + const code = (err as { code?: number; statusCode?: number }).code + ?? (err as { code?: number; statusCode?: number }).statusCode; + if (code !== 404) throw err; + // 404 means namespace doesn't exist yet — cluster is reachable. + } + return { + ok: true, + summary: `Kubernetes cluster reachable. Tenant namespace: ${namespace}.`, + metadata: { namespace, provider: "kubernetes" }, + }; + } catch (err) { + return { + ok: false, + summary: "Kubernetes cluster probe failed.", + metadata: { + namespace, + provider: "kubernetes", + error: err instanceof Error ? err.message : String(err), + }, + }; + } + }, + + async onEnvironmentAcquireLease( + params: PluginEnvironmentAcquireLeaseParams, + ): Promise { + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = deriveTenantNamespace(config, params.companyId); + + // Emit a runtime warning if FQDNs are configured but egressMode=standard + // cannot enforce them. Mirrors the validateConfig warning so operators see + // it in paperclip-server logs even if they missed the validation step. + const adapterDefaultsForWarn = getAdapterDefaults(config.adapterType); + const totalFqdnsForWarn = [...adapterDefaultsForWarn.allowFqdns, ...config.egressAllowFqdns]; + if (config.egressMode === "standard" && totalFqdnsForWarn.length > 0) { + console.warn( + `[plugin-kubernetes] egressMode=standard cannot enforce FQDN-based egress rules for ${totalFqdnsForWarn.join(", ")}. Agent pods will get public IPv4 HTTPS egress with private/link-local ranges excluded. Switch egressMode to "cilium" for exact FQDN enforcement.`, + ); + } + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + // Ensure the tenant namespace and all its RBAC / network policy resources + // exist before we try to create the Job. + const adapterDefaults = getAdapterDefaults(config.adapterType); + + await ensureTenant(clients, { + namespace, + companyId: params.companyId, + paperclipServerNamespace: PAPERCLIP_SERVER_NAMESPACE, + serviceAccountAnnotations: config.serviceAccountAnnotations, + egressMode: config.egressMode, + egressAllowFqdns: [...adapterDefaults.allowFqdns, ...config.egressAllowFqdns], + egressAllowCidrs: config.egressAllowCidrs, + resourceQuota: DEFAULT_RESOURCE_QUOTA, + }); + + const jobName = `pc-${newRunUlidDns()}`; + const secretName = `${jobName}-env`; + + // TODO: use params.runId as stand-in for agentId in labels; future + // versions will have a dedicated agentId on AcquireLeaseParams. + const labels = paperclipLabels({ + runId: params.runId, + agentId: params.runId, + companyId: params.companyId, + adapterType: config.adapterType, + }); + + const image = resolveImage( + { imageOverride: null }, + adapterDefaults, + { imageAllowList: config.imageAllowList, imageRegistry: config.imageRegistry }, + ); + + // Pick the orchestrator and build the appropriate manifest based on backend. + const isSandboxCrBackend = config.backend === "sandbox-cr"; + const orchestrator = isSandboxCrBackend ? sandboxCrOrchestrator : jobOrchestrator; + + const manifest = isSandboxCrBackend + ? buildSandboxCrManifest({ + namespace, + sandboxName: jobName, + adapterType: config.adapterType, + image, + envSecretName: secretName, + serviceAccountName: TENANT_SERVICE_ACCOUNT, + labels, + resources: config.defaultResources ?? {}, + runtimeClassName: config.runtimeClassName, + imagePullSecrets: config.imagePullSecrets, + }) + : buildJobManifest({ + namespace, + jobName, + adapterType: config.adapterType, + image, + envSecretName: secretName, + serviceAccountName: TENANT_SERVICE_ACCOUNT, + labels, + resources: config.defaultResources ?? {}, + runtimeClassName: config.runtimeClassName, + activeDeadlineSec: config.podActivityDeadlineSec, + ttlSecondsAfterFinished: config.jobTtlSecondsAfterFinished, + imagePullSecrets: config.imagePullSecrets, + }); + + const { uid: ownerUid } = await orchestrator.claim(clients, namespace, manifest); + + // M4b: adapter env vars are sourced from the plugin worker's own process + // environment (paperclip-server pod has them injected at deploy time). + const adapterEnv = extractAdapterEnvFromProcess(adapterDefaults.envKeys); + const bootstrapToken = generateBootstrapToken(); + + // Secret ownerRef: for job backend, the Job owns the Secret (cascade delete). + // For sandbox-cr backend, the Sandbox CR owns the Secret. + // NOTE: For sandbox-cr, if the Secret outlives the Sandbox due to a cluster + // quirk, the release() call will still clean it up via namespace GC or + // explicit delete in a future milestone. + await createPerRunSecret(clients, { + namespace, + secretName, + runId: params.runId, + ownerKind: isSandboxCrBackend ? "Sandbox" : "Job", + ownerApiVersion: isSandboxCrBackend ? "agents.x-k8s.io/v1alpha1" : "batch/v1", + ownerName: jobName, + ownerUid, + bootstrapToken, + adapterEnv, + }); + + const podName = await orchestrator.findPod(clients, namespace, jobName); + + const leaseMetadata: KubernetesLeaseMetadata = { + namespace, + jobName, + podName, + secretName, + phase: "Pending", + backend: config.backend, + }; + + return { + providerLeaseId: jobName, + metadata: leaseMetadata as unknown as Record, + }; + }, + + async onEnvironmentRealizeWorkspace( + params: PluginEnvironmentRealizeWorkspaceParams, + ): Promise { + // The agent pod already has /workspace mounted as an emptyDir at pod + // scheduling time (see pod-spec-builder). Nothing to provision here — + // we just hand back the cwd. Honor a caller-supplied remotePath if set. + const cwd = + params.workspace.remotePath && params.workspace.remotePath.trim().length > 0 + ? params.workspace.remotePath.trim() + : "/workspace"; + return { + cwd, + metadata: { + provider: "kubernetes", + remoteCwd: cwd, + }, + }; + }, + + async onEnvironmentReleaseLease( + params: PluginEnvironmentReleaseLeaseParams, + ): Promise { + if (!params.providerLeaseId) return; + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof params.leaseMetadata?.namespace === "string" + ? params.leaseMetadata.namespace + : deriveTenantNamespace(config, params.companyId); + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + const leaseBackend = + typeof params.leaseMetadata?.backend === "string" + ? (params.leaseMetadata.backend as "sandbox-cr" | "job") + : config.backend; + const releaseOrchestrator = + leaseBackend === "sandbox-cr" ? sandboxCrOrchestrator : jobOrchestrator; + + try { + await releaseOrchestrator.release(clients, namespace, params.providerLeaseId); + } catch (err) { + // If the resource is already gone (404), that's fine. + const code = (err as { code?: number; statusCode?: number }).code + ?? (err as { code?: number; statusCode?: number }).statusCode; + if (code !== 404) throw err; + } + }, + + async onEnvironmentExecute( + params: PluginEnvironmentExecuteParams, + ): Promise { + const { lease, timeoutMs } = params; + + if (!lease.providerLeaseId) { + return { + exitCode: 1, + timedOut: false, + stdout: "", + stderr: "No provider lease ID available for execution.", + }; + } + + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof lease.metadata?.namespace === "string" + ? lease.metadata.namespace + : deriveTenantNamespace(config, params.companyId); + + // Determine which backend this lease was created with. + const leaseBackend = + typeof lease.metadata?.backend === "string" + ? (lease.metadata.backend as "sandbox-cr" | "job") + : config.backend; + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + const effectiveTimeoutMs = + typeof timeoutMs === "number" && timeoutMs > 0 + ? timeoutMs + : config.podActivityDeadlineSec * 1000; + + if (leaseBackend === "sandbox-cr") { + // ── Sandbox-CR backend ────────────────────────────────────────────────── + // 1. Ensure the Sandbox pod is Ready (wait if needed). + // 2. Exec the command into the running pod. + // 3. Return exec result directly (no log scraping needed). + + let podName = + typeof lease.metadata?.podName === "string" && lease.metadata.podName + ? lease.metadata.podName + : null; + + // Wait for pod Ready if we don't have a pod name yet (or as a health check). + try { + await sandboxCrOrchestrator.waitForCompletion( + clients, + namespace, + lease.providerLeaseId, + { timeoutMs: effectiveTimeoutMs, pollMs: 2000 }, + ); + } catch (err) { + if (err instanceof SandboxCrTimeoutError) { + return { + exitCode: null, + timedOut: true, + stdout: "", + stderr: `Sandbox pod did not become Ready within ${effectiveTimeoutMs}ms`, + metadata: { + provider: "kubernetes", + backend: "sandbox-cr", + namespace, + sandboxName: lease.providerLeaseId, + }, + }; + } + throw err; + } + + // Resolve pod name (may now be populated in Sandbox status). + if (!podName) { + podName = await sandboxCrOrchestrator.findPod( + clients, + namespace, + lease.providerLeaseId, + ); + } + + if (!podName) { + return { + exitCode: 1, + timedOut: false, + stdout: "", + stderr: "Sandbox pod is Ready but podName could not be resolved.", + metadata: { + provider: "kubernetes", + backend: "sandbox-cr", + namespace, + sandboxName: lease.providerLeaseId, + }, + }; + } + + // Build the command to exec. If params.command is provided use it; + // otherwise wrap in a login shell so profile scripts run. + const rawCommand = + typeof params.command === "string" && params.command.trim().length > 0 + ? params.command + : params.args?.join(" ") ?? ""; + + const execCommand = rawCommand.length > 0 + ? ["/bin/sh", "-lc", rawCommand] + : ["/bin/sh", "-l"]; + + const execResult = await execInPod( + kc, + namespace, + podName, + "agent", + execCommand, + typeof params.stdin === "string" ? params.stdin : undefined, + ); + + return { + exitCode: execResult.exitCode, + timedOut: false, + stdout: execResult.stdout, + stderr: execResult.stderr, + metadata: { + provider: "kubernetes", + backend: "sandbox-cr", + namespace, + sandboxName: lease.providerLeaseId, + podName, + }, + }; + } else { + // ── Job backend (legacy / stable fallback) ────────────────────────────── + // The container entrypoint is baked into the Job spec (Tini + paperclip-agent-shim). + // We do NOT re-exec command/args — instead we wait for the Job to finish + // and collect its logs. + // + // params.command / params.args / params.stdin are intentionally ignored. + + let status; + let timedOut = false; + try { + status = await jobOrchestrator.waitForCompletion( + clients, + namespace, + lease.providerLeaseId, + { timeoutMs: effectiveTimeoutMs, pollMs: 2000 }, + ); + } catch (err) { + if (err instanceof JobTimeoutError) { + timedOut = true; + status = null; + } else { + throw err; + } + } + + // Collect logs from the pod. + const podName = + typeof lease.metadata?.podName === "string" + ? lease.metadata.podName + : await jobOrchestrator.findPod( + clients, + namespace, + lease.providerLeaseId, + ); + + const stdoutChunks: string[] = []; + const stderrChunks: string[] = []; + + if (podName) { + await jobOrchestrator.streamLogs( + clients, + namespace, + podName, + async (stream, text) => { + if (stream === "stdout") stdoutChunks.push(text); + else stderrChunks.push(text); + }, + ); + } + + return { + exitCode: timedOut ? null : status?.phase === "Succeeded" ? 0 : 1, + timedOut, + stdout: stdoutChunks.join(""), + stderr: stderrChunks.join(""), + metadata: { + provider: "kubernetes", + backend: "job", + namespace, + jobName: lease.providerLeaseId, + podName: podName ?? null, + phase: status?.phase ?? null, + }, + }; + } + }, +}); + +export default plugin; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/pod-exec.ts b/packages/plugins/sandbox-providers/kubernetes/src/pod-exec.ts new file mode 100644 index 00000000..b036780d --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/pod-exec.ts @@ -0,0 +1,79 @@ +/** + * Exec a command inside a running pod container using the Kubernetes exec API. + * + * Uses @kubernetes/client-node's Exec class, which opens a WebSocket to the + * kube-apiserver and streams stdout/stderr. The statusCallback receives a V1Status + * with status="Success" or status="Failure" + details.causes[{reason:"ExitCode"}]. + * + * NOTE: tty=false so stdout and stderr arrive on separate channels. If tty=true + * were used, they would be merged onto stdout and the exit code would not be + * reliable from the status callback on older cluster versions. + */ + +import { Exec } from "@kubernetes/client-node"; +import { PassThrough } from "node:stream"; +import type { KubeConfig } from "@kubernetes/client-node"; + +export async function execInPod( + kc: KubeConfig, + namespace: string, + podName: string, + containerName: string, + command: string[], + stdin?: string, +): Promise<{ exitCode: number; stdout: string; stderr: string }> { + const exec = new Exec(kc); + const stdoutStream = new PassThrough(); + const stderrStream = new PassThrough(); + + // If stdin is provided build a readable stream from it; the Exec API accepts + // a Readable | null for stdin. + const stdinStream: import("node:stream").Readable | null = stdin + ? PassThrough.from(stdin) + : null; + + let stdoutData = ""; + let stderrData = ""; + + stdoutStream.on("data", (chunk: Buffer) => { + stdoutData += chunk.toString("utf-8"); + }); + stderrStream.on("data", (chunk: Buffer) => { + stderrData += chunk.toString("utf-8"); + }); + + return await new Promise<{ exitCode: number; stdout: string; stderr: string }>( + (resolve, reject) => { + exec + .exec( + namespace, + podName, + containerName, + command, + stdoutStream, + stderrStream, + stdinStream, + false, // tty=false: keep stdout/stderr on separate channels + (status) => { + // status.status is "Success" | "Failure" + if (status.status === "Success") { + resolve({ exitCode: 0, stdout: stdoutData, stderr: stderrData }); + return; + } + // On failure, the exit code surfaces via + // status.details?.causes[].{reason:"ExitCode", message:""} + const causes = status.details?.causes ?? []; + const exitCodeCause = causes.find( + (c: { reason?: string; message?: string }) => + c.reason === "ExitCode", + ); + const exitCode = exitCodeCause?.message + ? Number(exitCodeCause.message) + : 1; + resolve({ exitCode, stdout: stdoutData, stderr: stderrData }); + }, + ) + .catch(reject); + }, + ); +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/pod-spec-builder.ts b/packages/plugins/sandbox-providers/kubernetes/src/pod-spec-builder.ts new file mode 100644 index 00000000..128a671c --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/pod-spec-builder.ts @@ -0,0 +1,94 @@ +export interface BuildJobManifestInput { + namespace: string; + jobName: string; + adapterType: string; + image: string; + envSecretName: string; + serviceAccountName: string; + labels: Record; + resources: { + requests?: { cpu?: string; memory?: string }; + limits?: { cpu?: string; memory?: string }; + }; + runtimeClassName?: string; + activeDeadlineSec: number; + ttlSecondsAfterFinished: number; + imagePullSecrets?: string[]; +} + +export function buildJobManifest(input: BuildJobManifestInput): Record { + const podLabels = { + ...input.labels, + "paperclip.io/role": "agent", + }; + return { + apiVersion: "batch/v1", + kind: "Job", + metadata: { + name: input.jobName, + namespace: input.namespace, + labels: { ...input.labels }, + }, + spec: { + backoffLimit: 0, + ttlSecondsAfterFinished: input.ttlSecondsAfterFinished, + activeDeadlineSeconds: input.activeDeadlineSec, + template: { + metadata: { labels: podLabels }, + spec: { + serviceAccountName: input.serviceAccountName, + // Agent containers call back to paperclip-server via HTTPS egress; + // they never call the Kubernetes API, so mounting an SA token is + // unnecessary attack surface. + automountServiceAccountToken: false, + restartPolicy: "Never", + ...(input.runtimeClassName ? { runtimeClassName: input.runtimeClassName } : {}), + ...(input.imagePullSecrets && input.imagePullSecrets.length > 0 + ? { imagePullSecrets: input.imagePullSecrets.map((name) => ({ name })) } + : {}), + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + fsGroup: 1000, + fsGroupChangePolicy: "OnRootMismatch", + seccompProfile: { type: "RuntimeDefault" }, + }, + containers: [ + { + name: "agent", + image: input.image, + imagePullPolicy: "IfNotPresent", + command: ["/usr/bin/tini", "--", "/usr/local/bin/paperclip-agent-shim"], + envFrom: [{ secretRef: { name: input.envSecretName } }], + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + readOnlyRootFilesystem: true, + allowPrivilegeEscalation: false, + capabilities: { drop: ["ALL"] }, + }, + resources: { + requests: input.resources.requests ?? { cpu: "250m", memory: "512Mi" }, + limits: input.resources.limits ?? { cpu: "2", memory: "4Gi" }, + }, + volumeMounts: [ + { name: "workspace", mountPath: "/workspace" }, + { name: "home", mountPath: "/home/paperclip" }, + { name: "cache", mountPath: "/home/paperclip/.cache" }, + { name: "tmp", mountPath: "/tmp" }, + ], + }, + ], + volumes: [ + { name: "workspace", emptyDir: { sizeLimit: "8Gi" } }, + { name: "home", emptyDir: { sizeLimit: "1Gi" } }, + { name: "cache", emptyDir: { sizeLimit: "1Gi" } }, + { name: "tmp", emptyDir: { sizeLimit: "2Gi" } }, + ], + }, + }, + }, + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-builder.ts b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-builder.ts new file mode 100644 index 00000000..fb3d7e6c --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-builder.ts @@ -0,0 +1,136 @@ +/** + * Builds a kubernetes-sigs/agent-sandbox Sandbox CR manifest. + * + * The Sandbox CR creates a long-lived pod (sleep infinity entrypoint) into + * which paperclip-server can exec arbitrary commands. This solves the + * architectural mismatch with the batch/v1 Job backend, which only supports + * a single one-shot entrypoint — not the multi-command adapter-install pattern + * used by paperclip-server. + * + * Security baseline is identical to buildJobManifest (pod-spec-builder.ts): + * non-root, drop ALL caps, read-only rootFS, Tini PID 1, seccomp + * RuntimeDefault, fsGroupChangePolicy OnRootMismatch, automountSAToken=false. + * + * NOTE: paperclip-server runs OUTSIDE the cluster, so we cannot set ownerReferences + * on the Sandbox CR (the owner would need to be an in-cluster resource). The + * release path is explicit delete via sandboxCrOrchestrator.release(). + */ + +export interface BuildSandboxCrManifestInput { + namespace: string; + sandboxName: string; + adapterType: string; + image: string; + envSecretName: string; + serviceAccountName: string; + labels: Record; + resources: { + requests?: { cpu?: string; memory?: string }; + limits?: { cpu?: string; memory?: string }; + }; + runtimeClassName?: string; + imagePullSecrets?: string[]; +} + +export function buildSandboxCrManifest( + input: BuildSandboxCrManifestInput, +): Record { + const podLabels: Record = { + ...input.labels, + "paperclip.io/role": "agent", + }; + return { + apiVersion: "agents.x-k8s.io/v1alpha1", + kind: "Sandbox", + metadata: { + name: input.sandboxName, + namespace: input.namespace, + labels: { ...input.labels }, + // No ownerReferences: paperclip-server is out-of-cluster. Release is + // explicit delete. + }, + spec: { + podTemplate: { + metadata: { + labels: podLabels, + }, + spec: { + serviceAccountName: input.serviceAccountName, + // Agent containers call back to paperclip-server via HTTPS egress; + // they never call the Kubernetes API, so mounting an SA token is + // unnecessary attack surface. + automountServiceAccountToken: false, + // Sandbox controller requires restartPolicy: Always so the pod + // stays running between exec calls. + restartPolicy: "Always", + ...(input.runtimeClassName + ? { runtimeClassName: input.runtimeClassName } + : {}), + ...(input.imagePullSecrets && input.imagePullSecrets.length > 0 + ? { + imagePullSecrets: input.imagePullSecrets.map((name) => ({ + name, + })), + } + : {}), + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + fsGroup: 1000, + fsGroupChangePolicy: "OnRootMismatch", + seccompProfile: { type: "RuntimeDefault" }, + }, + containers: [ + { + name: "agent", + image: input.image, + imagePullPolicy: "IfNotPresent", + // sleep infinity keeps the pod running; paperclip-server execs + // commands into it via Kubernetes exec API. Tini as PID 1 for + // proper signal forwarding and zombie reaping. + command: [ + "/usr/bin/tini", + "--", + "/bin/sh", + "-c", + "sleep infinity", + ], + envFrom: [{ secretRef: { name: input.envSecretName } }], + securityContext: { + runAsNonRoot: true, + runAsUser: 1000, + runAsGroup: 1000, + readOnlyRootFilesystem: true, + allowPrivilegeEscalation: false, + capabilities: { drop: ["ALL"] }, + }, + resources: { + requests: input.resources.requests ?? { + cpu: "250m", + memory: "512Mi", + }, + limits: input.resources.limits ?? { + cpu: "2", + memory: "4Gi", + }, + }, + volumeMounts: [ + { name: "workspace", mountPath: "/workspace" }, + { name: "home", mountPath: "/home/paperclip" }, + { name: "cache", mountPath: "/home/paperclip/.cache" }, + { name: "tmp", mountPath: "/tmp" }, + ], + }, + ], + volumes: [ + { name: "workspace", emptyDir: { sizeLimit: "8Gi" } }, + { name: "home", emptyDir: { sizeLimit: "1Gi" } }, + { name: "cache", emptyDir: { sizeLimit: "1Gi" } }, + { name: "tmp", emptyDir: { sizeLimit: "2Gi" } }, + ], + }, + }, + }, + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-orchestrator.ts b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-orchestrator.ts new file mode 100644 index 00000000..dc718d9b --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-orchestrator.ts @@ -0,0 +1,288 @@ +/** + * SandboxOrchestrator implementation backed by the kubernetes-sigs/agent-sandbox + * Sandbox CRD (agents.x-k8s.io/v1alpha1). + * + * The Sandbox CR creates a long-lived pod that paperclip-server can exec into + * for multi-command adapter-install workflows — the key architectural win over + * the batch/v1 Job backend. + * + * Key semantic differences from jobOrchestrator: + * - claim() creates a Sandbox CR via CustomObjectsApi instead of a batch Job + * - getStatus() maps Sandbox phase (Pending|Ready|Terminating|Failed) to SandboxStatus + * - findPod() reads status.podName from the Sandbox CR (falls back to label query) + * - waitForCompletion() means "wait until pod is Ready to exec" NOT "wait until + * workload finishes". The Sandbox pod runs sleep infinity; execution completion + * is tracked by the individual execInPod() calls. + * - release() deletes the Sandbox CR with Foreground propagation (controller + * tears down the underlying pod). + * + * NOTE: streamLogs() is provided for interface conformance but is limited — + * the sleep-infinity pod has no meaningful stdout. Callers in execute mode + * should use execInPod() and capture its stdout/stderr directly. + */ + +import type { KubeClients } from "./kube-client.js"; +import type { SandboxOrchestrator, SandboxStatus } from "./sandbox-orchestrator.js"; + +const SANDBOX_GROUP = "agents.x-k8s.io"; +const SANDBOX_VERSION = "v1alpha1"; +const SANDBOX_PLURAL = "sandboxes"; + +export class SandboxCrTimeoutError extends Error { + constructor(namespace: string, name: string, timeoutMs: number) { + super( + `Sandbox ${namespace}/${name} did not reach Ready phase within ${timeoutMs}ms`, + ); + this.name = "SandboxCrTimeoutError"; + } +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** + * Map a Sandbox CR status.phase value to our SandboxStatus shape. + * Sandbox phases: Pending | Ready | Terminating | Failed + */ +function mapSandboxPhase( + cr: Record, +): SandboxStatus { + const status = (cr.status as Record) ?? {}; + const phase = (status.phase as string) ?? "Pending"; + + switch (phase) { + case "Ready": + return { + phase: "Running", // SandboxStatus.phase uses Job semantics; "Running" = active pod + complete: false, + active: 1, + succeeded: 0, + failed: 0, + }; + case "Terminating": + return { + phase: "Running", + complete: false, + active: 0, + succeeded: 0, + failed: 0, + reason: "Terminating", + }; + case "Failed": { + const conditions = (status.conditions as { type?: string; reason?: string; message?: string }[]) ?? []; + const failedCond = conditions.find((c) => c.type === "Failed"); + return { + phase: "Failed", + complete: false, + active: 0, + succeeded: 0, + failed: 1, + reason: failedCond?.reason, + message: failedCond?.message, + }; + } + default: + // "Pending" or unknown + return { + phase: "Pending", + complete: false, + active: 0, + succeeded: 0, + failed: 0, + }; + } +} + +export async function createSandboxCr( + clients: KubeClients, + namespace: string, + manifest: Record, +): Promise<{ uid: string }> { + const result = await clients.custom.createNamespacedCustomObject({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace, + plural: SANDBOX_PLURAL, + body: manifest, + }); + const uid = (result as { metadata?: { uid?: string } }).metadata?.uid; + if (!uid) throw new Error("Sandbox CR created without a UID"); + return { uid }; +} + +export async function getSandboxCrStatus( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + const result = await clients.custom.getNamespacedCustomObject({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace, + plural: SANDBOX_PLURAL, + name, + }); + return mapSandboxPhase(result as Record); +} + +/** + * Returns the pod name backing a Sandbox CR. + * Primary: read status.podName from the CR (set by the controller once ready). + * Fallback: list pods in the namespace filtered by the paperclip.io/managed-by + * label and the sandbox name label set on the pod template. + */ +export async function findPodForSandbox( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + // Primary: read status.podName from the Sandbox CR + const cr = await clients.custom.getNamespacedCustomObject({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace, + plural: SANDBOX_PLURAL, + name, + }) as Record; + + const status = (cr.status as Record) ?? {}; + const podName = status.podName as string | undefined; + if (podName && podName.trim().length > 0) { + return podName; + } + + // Fallback: list pods with sandbox-name label (sandbox controller typically + // labels pods with the sandbox name) + const result = await clients.core.listNamespacedPod({ + namespace, + labelSelector: `paperclip.io/managed-by=paperclip-k8s-plugin`, + }); + const items = + ( + ( + result as { + items?: { + metadata?: { name?: string; labels?: Record }; + status?: { phase?: string }; + }[]; + } + ).items + ) ?? []; + + // Filter to pods that belong to this sandbox by name prefix or label + const matching = items.filter((p) => { + const podMeta = p.metadata ?? {}; + const labels = podMeta.labels ?? {}; + // The sandbox controller may label pods differently; try matching by name prefix + return ( + podMeta.name?.startsWith(name) || + labels["agents.x-k8s.io/sandbox-name"] === name + ); + }); + + const running = matching.find((p) => p.status?.phase === "Running"); + return (running ?? matching[0])?.metadata?.name ?? null; +} + +export async function streamSandboxLogs( + clients: KubeClients, + namespace: string, + podName: string, + onChunk: (stream: "stdout" | "stderr", text: string) => Promise, +): Promise { + // V1 limitation: the Pod log API returns the container's combined log stream. The + // sleep-infinity pod will have minimal output; this is provided for interface + // conformance. For actual command output, use execInPod() directly. + const result = await clients.core.readNamespacedPodLog({ + namespace, + name: podName, + }); + const text = + typeof result === "string" + ? result + : typeof (result as { body?: unknown })?.body === "string" + ? (result as { body: string }).body + : ""; + if (text.length > 0) await onChunk("stdout", text); +} + +export async function deleteSandboxCr( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + await clients.custom.deleteNamespacedCustomObject({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace, + plural: SANDBOX_PLURAL, + name, + propagationPolicy: "Foreground", + }); +} + +/** + * Wait until the Sandbox CR's pod reaches Ready phase (i.e., the pod is up and + * exec-able). This is NOT waiting for a workload to finish — the Sandbox pod + * runs sleep infinity indefinitely. Execution completion is tracked by the + * individual execInPod() calls. + * + * Throws SandboxCrTimeoutError if Ready is not reached within timeoutMs. + * Throws if the Sandbox transitions to Failed. + */ +export async function waitForSandboxReady( + clients: KubeClients, + namespace: string, + name: string, + opts: { timeoutMs: number; pollMs?: number } = { + timeoutMs: 120_000, + pollMs: 2000, + }, +): Promise { + const deadline = Date.now() + opts.timeoutMs; + const pollMs = opts.pollMs ?? 2000; + + while (Date.now() < deadline) { + const cr = await clients.custom.getNamespacedCustomObject({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace, + plural: SANDBOX_PLURAL, + name, + }) as Record; + + const status = (cr.status as Record) ?? {}; + const phase = (status.phase as string) ?? "Pending"; + + if (phase === "Ready") { + return mapSandboxPhase(cr); + } + if (phase === "Failed") { + const mapped = mapSandboxPhase(cr); + throw new Error( + `Sandbox ${namespace}/${name} failed: ${mapped.reason ?? "unknown reason"} — ${mapped.message ?? ""}`, + ); + } + // Pending or Terminating — keep polling + await sleep(pollMs); + } + + throw new SandboxCrTimeoutError(namespace, name, opts.timeoutMs); +} + +/** + * Sandbox CR-backed conformance to SandboxOrchestrator. + * + * waitForCompletion semantics change: for this backend, "completion" means + * "pod is up and Ready to exec into" — NOT "workload finished". The actual + * command execution and its completion is handled by execInPod(). + */ +export const sandboxCrOrchestrator: SandboxOrchestrator = { + claim: createSandboxCr, + getStatus: getSandboxCrStatus, + findPod: findPodForSandbox, + streamLogs: streamSandboxLogs, + release: deleteSandboxCr, + waitForCompletion: waitForSandboxReady, +}; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/sandbox-orchestrator.ts b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-orchestrator.ts new file mode 100644 index 00000000..db4dbb72 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/sandbox-orchestrator.ts @@ -0,0 +1,68 @@ +import type { KubeClients } from "./kube-client.js"; + +export interface SandboxStatus { + phase: "Pending" | "Running" | "Succeeded" | "Failed"; + complete: boolean; + active: number; + succeeded: number; + failed: number; + reason?: string; + message?: string; +} + +/** + * Abstract interface over a sandbox runtime backend. The current implementation + * is Job-backed (job-orchestrator.ts). Future backends slot in by exporting an + * object conforming to this shape — e.g. a Kata-FC warm-pool backend that + * additionally implements the optional pause/resume slots, or a CRD-backed + * backend on kubernetes-sigs/agent-sandbox once it reaches Beta. + */ +export interface SandboxOrchestrator { + /** Provision the sandbox. Returns the runtime's stable UID. */ + claim( + clients: KubeClients, + namespace: string, + manifest: Record, + ): Promise<{ uid: string }>; + + /** Read current lifecycle phase. */ + getStatus( + clients: KubeClients, + namespace: string, + name: string, + ): Promise; + + /** Locate the pod backing this sandbox (or null if none exists yet). */ + findPod( + clients: KubeClients, + namespace: string, + name: string, + ): Promise; + + /** Read logs from the sandbox's pod. V1: post-completion read. */ + streamLogs( + clients: KubeClients, + namespace: string, + podName: string, + onChunk: (stream: "stdout" | "stderr", text: string) => Promise, + ): Promise; + + /** Tear down the sandbox. Implementations MUST cascade-delete child resources. */ + release(clients: KubeClients, namespace: string, name: string): Promise; + + /** Block until phase is Succeeded or Failed, or throw on timeout. */ + waitForCompletion( + clients: KubeClients, + namespace: string, + name: string, + opts: { timeoutMs: number; pollMs?: number }, + ): Promise; + + // Optional warm-pool / Kata-FC extension slots. Job-backed implementation + // does not provide these; runtimes that do (e.g. Kata-FC microVM pause) + // implement them and acquire the warm-pool capability. + // TODO: requires custom in-cluster controller for k8s — kubelet does not + // expose pause/resume at the pod level. Add when warm-pool design lands. + pause?(clients: KubeClients, namespace: string, name: string): Promise; + resume?(clients: KubeClients, namespace: string, name: string): Promise; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/secret-manager.ts b/packages/plugins/sandbox-providers/kubernetes/src/secret-manager.ts new file mode 100644 index 00000000..cbf2150b --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/secret-manager.ts @@ -0,0 +1,52 @@ +import type { KubeClients } from "./kube-client.js"; + +export interface CreatePerRunSecretInput { + namespace: string; + secretName: string; + runId: string; + ownerKind: string; + ownerApiVersion: string; + ownerName: string; + ownerUid: string; + bootstrapToken: string; + adapterEnv: Record; +} + +export async function createPerRunSecret(clients: KubeClients, input: CreatePerRunSecretInput): Promise { + if (!input.ownerUid) { + throw new Error("createPerRunSecret requires a non-empty ownerUid"); + } + if ("BOOTSTRAP_TOKEN" in input.adapterEnv) { + throw new Error("adapterEnv must not contain BOOTSTRAP_TOKEN (reserved key)"); + } + await clients.core.createNamespacedSecret({ + namespace: input.namespace, + body: { + apiVersion: "v1", + kind: "Secret", + type: "Opaque", + metadata: { + name: input.secretName, + namespace: input.namespace, + labels: { + "paperclip.io/run-id": input.runId, + "paperclip.io/managed-by": "paperclip-k8s-plugin", + }, + ownerReferences: [ + { + apiVersion: input.ownerApiVersion, + kind: input.ownerKind, + name: input.ownerName, + uid: input.ownerUid, + controller: true, + blockOwnerDeletion: true, + }, + ], + }, + stringData: { + BOOTSTRAP_TOKEN: input.bootstrapToken, + ...input.adapterEnv, + }, + }, + }); +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/tenant-orchestrator.ts b/packages/plugins/sandbox-providers/kubernetes/src/tenant-orchestrator.ts new file mode 100644 index 00000000..e69f522e --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/tenant-orchestrator.ts @@ -0,0 +1,322 @@ +import type { KubeClients } from "./kube-client.js"; +import { buildNetworkPolicyManifests } from "./network-policy.js"; +import { buildCiliumNetworkPolicyManifest } from "./cilium-network-policy.js"; + +export interface EnsureTenantInput { + namespace: string; + companyId: string; + paperclipServerNamespace: string; + serviceAccountAnnotations: Record; + egressMode: "standard" | "cilium"; + egressAllowFqdns: string[]; + egressAllowCidrs: string[]; + resourceQuota: { + pods: string; + requestsCpu: string; + requestsMemory: string; + limitsCpu: string; + limitsMemory: string; + }; +} + +const SERVICE_ACCOUNT_NAME = "paperclip-tenant-sa"; +const ROLE_NAME = "paperclip-tenant-role"; +const ROLE_BINDING_NAME = "paperclip-tenant-rb"; +const RESOURCE_QUOTA_NAME = "paperclip-quota"; +const LIMIT_RANGE_NAME = "paperclip-limits"; + +/** + * Tenant provisioning reconciles the resources this plugin owns. Existing + * resources are replaced with the desired manifest so quota, RBAC, service + * account annotations, and egress policy changes take effect on the next run. + */ +export async function ensureTenant(clients: KubeClients, input: EnsureTenantInput): Promise { + await ensureNamespace(clients, input); + await ensureServiceAccount(clients, input); + await ensureRole(clients, input); + await ensureRoleBinding(clients, input); + await ensureResourceQuota(clients, input); + await ensureLimitRange(clients, input); + await ensureNetworkPolicies(clients, input); +} + +async function ensureNamespace(clients: KubeClients, input: EnsureTenantInput): Promise { + try { + await clients.core.readNamespace({ name: input.namespace }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.core.createNamespace({ + body: { + apiVersion: "v1", + kind: "Namespace", + metadata: { + name: input.namespace, + labels: { + "paperclip.io/company-id": input.companyId, + "paperclip.io/managed-by": "paperclip-k8s-plugin", + "pod-security.kubernetes.io/enforce": "restricted", + "pod-security.kubernetes.io/audit": "restricted", + "pod-security.kubernetes.io/warn": "restricted", + }, + }, + }, + }); +} + +async function ensureServiceAccount(clients: KubeClients, input: EnsureTenantInput): Promise { + const manifest = { + apiVersion: "v1", + kind: "ServiceAccount", + metadata: { + name: SERVICE_ACCOUNT_NAME, + namespace: input.namespace, + annotations: input.serviceAccountAnnotations, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + }; + try { + const existing = await clients.core.readNamespacedServiceAccount({ name: SERVICE_ACCOUNT_NAME, namespace: input.namespace }); + await clients.core.replaceNamespacedServiceAccount({ + name: SERVICE_ACCOUNT_NAME, + namespace: input.namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.core.createNamespacedServiceAccount({ namespace: input.namespace, body: manifest }); +} + +async function ensureRole(clients: KubeClients, input: EnsureTenantInput): Promise { + const manifest = { + apiVersion: "rbac.authorization.k8s.io/v1", + kind: "Role", + metadata: { name: ROLE_NAME, namespace: input.namespace }, + rules: [ + { apiGroups: [""], resources: ["pods/log"], verbs: ["get"] }, + ], + }; + try { + const existing = await clients.rbac.readNamespacedRole({ name: ROLE_NAME, namespace: input.namespace }); + await clients.rbac.replaceNamespacedRole({ + name: ROLE_NAME, + namespace: input.namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.rbac.createNamespacedRole({ namespace: input.namespace, body: manifest }); +} + +async function ensureRoleBinding(clients: KubeClients, input: EnsureTenantInput): Promise { + const manifest = { + apiVersion: "rbac.authorization.k8s.io/v1", + kind: "RoleBinding", + metadata: { name: ROLE_BINDING_NAME, namespace: input.namespace }, + roleRef: { apiGroup: "rbac.authorization.k8s.io", kind: "Role", name: ROLE_NAME }, + subjects: [{ kind: "ServiceAccount", name: SERVICE_ACCOUNT_NAME, namespace: input.namespace }], + }; + try { + const existing = await clients.rbac.readNamespacedRoleBinding({ name: ROLE_BINDING_NAME, namespace: input.namespace }); + await clients.rbac.replaceNamespacedRoleBinding({ + name: ROLE_BINDING_NAME, + namespace: input.namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.rbac.createNamespacedRoleBinding({ namespace: input.namespace, body: manifest }); +} + +async function ensureResourceQuota(clients: KubeClients, input: EnsureTenantInput): Promise { + const manifest = { + apiVersion: "v1", + kind: "ResourceQuota", + metadata: { name: RESOURCE_QUOTA_NAME, namespace: input.namespace }, + spec: { + hard: { + pods: input.resourceQuota.pods, + "requests.cpu": input.resourceQuota.requestsCpu, + "requests.memory": input.resourceQuota.requestsMemory, + "limits.cpu": input.resourceQuota.limitsCpu, + "limits.memory": input.resourceQuota.limitsMemory, + }, + }, + }; + try { + const existing = await clients.core.readNamespacedResourceQuota({ name: RESOURCE_QUOTA_NAME, namespace: input.namespace }); + await clients.core.replaceNamespacedResourceQuota({ + name: RESOURCE_QUOTA_NAME, + namespace: input.namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.core.createNamespacedResourceQuota({ namespace: input.namespace, body: manifest }); +} + +async function ensureLimitRange(clients: KubeClients, input: EnsureTenantInput): Promise { + const manifest = { + apiVersion: "v1", + kind: "LimitRange", + metadata: { name: LIMIT_RANGE_NAME, namespace: input.namespace }, + spec: { + limits: [ + { + type: "Container", + max: { cpu: "4", memory: "8Gi" }, + min: { cpu: "100m", memory: "128Mi" }, + // The k8s client-node type names this `_default` but the actual + // Kubernetes API field is `default`. We produce a JSON-shape + // manifest so the cast is safe. + default: { cpu: "1", memory: "2Gi" }, + defaultRequest: { cpu: "250m", memory: "512Mi" }, + }, + ], + }, + }; + try { + const existing = await clients.core.readNamespacedLimitRange({ name: LIMIT_RANGE_NAME, namespace: input.namespace }); + await clients.core.replaceNamespacedLimitRange({ + name: LIMIT_RANGE_NAME, + namespace: input.namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.core.createNamespacedLimitRange({ + namespace: input.namespace, + body: manifest as never, + }); +} + +async function ensureNetworkPolicies(clients: KubeClients, input: EnsureTenantInput): Promise { + const [denyAll, egressStd] = buildNetworkPolicyManifests({ + namespace: input.namespace, + paperclipServerNamespace: input.paperclipServerNamespace, + egressAllowFqdns: input.egressAllowFqdns, + egressAllowCidrs: input.egressAllowCidrs, + }); + + await ensureNetworkPolicy(clients, input.namespace, denyAll); + + if (input.egressMode === "cilium") { + const cnp = buildCiliumNetworkPolicyManifest({ + namespace: input.namespace, + paperclipServerNamespace: input.paperclipServerNamespace, + egressAllowFqdns: input.egressAllowFqdns, + egressAllowCidrs: input.egressAllowCidrs, + }); + await ensureCiliumNetworkPolicy(clients, input.namespace, cnp); + await deleteNetworkPolicyIfExists(clients, input.namespace, "paperclip-egress-allow"); + } else { + await ensureNetworkPolicy(clients, input.namespace, egressStd); + await deleteCiliumNetworkPolicyIfExists(clients, input.namespace, "paperclip-egress-fqdn"); + } +} + +async function ensureNetworkPolicy( + clients: KubeClients, + namespace: string, + manifest: Record, +): Promise { + const name = (manifest.metadata as { name: string }).name; + try { + const existing = await clients.networking.readNamespacedNetworkPolicy({ name, namespace }); + await clients.networking.replaceNamespacedNetworkPolicy({ + name, + namespace, + body: withResourceVersion(manifest, existing) as never, + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.networking.createNamespacedNetworkPolicy({ namespace, body: manifest as never }); +} + +async function ensureCiliumNetworkPolicy( + clients: KubeClients, + namespace: string, + manifest: Record, +): Promise { + const name = (manifest.metadata as { name: string }).name; + try { + const existing = await clients.custom.getNamespacedCustomObject({ + group: "cilium.io", + version: "v2", + namespace, + plural: "ciliumnetworkpolicies", + name, + }); + await clients.custom.replaceNamespacedCustomObject({ + group: "cilium.io", + version: "v2", + namespace, + plural: "ciliumnetworkpolicies", + name, + body: withResourceVersion(manifest, existing), + }); + return; + } catch (err) { + if (!isNotFound(err)) throw err; + } + await clients.custom.createNamespacedCustomObject({ + group: "cilium.io", + version: "v2", + namespace, + plural: "ciliumnetworkpolicies", + body: manifest, + }); +} + +async function deleteNetworkPolicyIfExists(clients: KubeClients, namespace: string, name: string): Promise { + try { + await clients.networking.deleteNamespacedNetworkPolicy({ name, namespace }); + } catch (err) { + if (!isNotFound(err)) throw err; + } +} + +async function deleteCiliumNetworkPolicyIfExists(clients: KubeClients, namespace: string, name: string): Promise { + try { + await clients.custom.deleteNamespacedCustomObject({ + group: "cilium.io", + version: "v2", + namespace, + plural: "ciliumnetworkpolicies", + name, + }); + } catch (err) { + if (!isNotFound(err)) throw err; + } +} + +function withResourceVersion>(manifest: T, existing: unknown): T { + const resourceVersion = (existing as { metadata?: { resourceVersion?: string } })?.metadata?.resourceVersion; + if (!resourceVersion) return manifest; + return { + ...manifest, + metadata: { + ...(manifest.metadata as Record), + resourceVersion, + }, + }; +} + +function isNotFound(err: unknown): boolean { + if (typeof err !== "object" || err === null) return false; + const e = err as { code?: number; statusCode?: number }; + return e.code === 404 || e.statusCode === 404; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/types.ts b/packages/plugins/sandbox-providers/kubernetes/src/types.ts new file mode 100644 index 00000000..2a4407fa --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/types.ts @@ -0,0 +1,85 @@ +import { z } from "zod"; +import { KNOWN_ADAPTER_TYPES } from "./adapter-defaults.js"; + +const cidrRegex = /^(\d{1,3}\.){3}\d{1,3}\/\d{1,2}$/; + +export const kubernetesProviderConfigSchema = z + .object({ + inCluster: z.boolean().default(false), + kubeconfig: z.string().optional(), + + namespacePrefix: z.string().regex(/^[a-z0-9-]{1,32}$/).default("paperclip-"), + companySlug: z.string().regex(/^[a-z0-9-]{1,32}$/).optional(), + + imageRegistry: z.string().url().optional(), + imageAllowList: z.array(z.string()).default([]), + imagePullSecrets: z.array(z.string()).default([]), + + egressAllowFqdns: z.array(z.string()).default([]), + egressAllowCidrs: z.array(z.string().regex(cidrRegex, "Invalid CIDR")).default([]), + egressMode: z.enum(["cilium", "standard"]).default("standard"), + + defaultResources: z + .object({ + requests: z.object({ cpu: z.string(), memory: z.string() }).partial().optional(), + limits: z.object({ cpu: z.string(), memory: z.string() }).partial().optional(), + }) + .optional(), + + runtimeClassName: z.string().optional(), + serviceAccountAnnotations: z.record(z.string()).default({}), + + jobTtlSecondsAfterFinished: z.number().int().nonnegative().default(900), + podActivityDeadlineSec: z.number().int().positive().default(3600), + + /** + * The adapter type that Jobs in this environment will run. + * Each Kubernetes environment is bound to one adapter; create multiple + * environments for different adapters. + * Defaults to `"claude_local"`. + */ + adapterType: z + .string() + .default("claude_local") + .refine((v) => KNOWN_ADAPTER_TYPES.has(v), { + message: "adapterType must be one of the known adapter types", + }), + + /** + * The sandbox backend to use. + * + * - `"sandbox-cr"` (default, alpha) — uses the kubernetes-sigs/agent-sandbox + * Sandbox CRD (agents.x-k8s.io/v1alpha1). Creates a long-lived pod that + * paperclip-server can exec into for multi-command adapter-install workflows. + * Requires the agent-sandbox controller to be installed in the cluster. + * + * - `"job"` — uses batch/v1 Job (stable fallback). One-shot entrypoint; does + * NOT support multi-command exec. Use this for clusters without agent-sandbox + * installed, or when you need stable (non-alpha) k8s APIs. + */ + backend: z.enum(["sandbox-cr", "job"]).default("sandbox-cr"), + }) + .refine( + (cfg) => cfg.inCluster || cfg.kubeconfig, + { + message: + "kubernetes provider requires one of `inCluster` or `kubeconfig`", + }, + ); + +export type KubernetesProviderConfig = z.infer; + +export function parseKubernetesProviderConfig(input: unknown): KubernetesProviderConfig { + return kubernetesProviderConfigSchema.parse(input); +} + +export interface KubernetesLeaseMetadata { + namespace: string; + /** Name of the workload resource (Job name for job backend, Sandbox CR name for sandbox-cr backend). */ + jobName: string; + podName: string | null; + secretName: string; + phase: "Pending" | "Running" | "Succeeded" | "Failed"; + /** Which backend provisioned this lease. */ + backend: "sandbox-cr" | "job"; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/utils.ts b/packages/plugins/sandbox-providers/kubernetes/src/utils.ts new file mode 100644 index 00000000..07e80ece --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/utils.ts @@ -0,0 +1,46 @@ +const ULID_ALPHABET = "0123456789abcdefghjkmnpqrstvwxyz"; + +export function deriveCompanySlug(input: string): string { + const slug = input + .toLowerCase() + .replace(/[^a-z0-9-]+/g, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 32) + .replace(/-+$/, ""); + return slug.length > 0 ? slug : "company"; +} + +export function deriveNamespaceName(prefix: string, slug: string): string { + return `${prefix}${slug}`; +} + +export function newRunUlidDns(now: () => number = Date.now): string { + const timestamp = now(); + let out = ""; + let t = timestamp; + for (let i = 0; i < 10; i++) { + out = ULID_ALPHABET[t & 0x1f] + out; + t = Math.floor(t / 32); + } + for (let i = 0; i < 16; i++) { + out += ULID_ALPHABET[Math.floor(Math.random() * 32)]; + } + return out; +} + +export interface LabelsInput { + runId: string; + agentId: string; + companyId: string; + adapterType: string; +} + +export function paperclipLabels(input: LabelsInput): Record { + return { + "paperclip.io/run-id": input.runId, + "paperclip.io/agent-id": input.agentId, + "paperclip.io/company-id": input.companyId, + "paperclip.io/adapter": input.adapterType, + "paperclip.io/managed-by": "paperclip-k8s-plugin", + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/worker.ts b/packages/plugins/sandbox-providers/kubernetes/src/worker.ts new file mode 100644 index 00000000..1e156024 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/worker.ts @@ -0,0 +1,5 @@ +import { runWorker } from "@paperclipai/plugin-sdk"; +import plugin from "./plugin.js"; + +export default plugin; +runWorker(plugin, import.meta.url); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/integration/_kind-harness.ts b/packages/plugins/sandbox-providers/kubernetes/test/integration/_kind-harness.ts new file mode 100644 index 00000000..a12000ca --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/integration/_kind-harness.ts @@ -0,0 +1,22 @@ +import { execSync } from "node:child_process"; +import { readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +export const KIND_CONTEXT = "kind-paperclip"; + +export function readKindKubeconfig(): string { + return readFileSync(join(homedir(), ".kube", "config"), "utf-8"); +} + +export function kubectl(args: string): string { + return execSync(`kubectl --context ${KIND_CONTEXT} ${args}`, { encoding: "utf-8" }); +} + +export function deleteNamespaceIfExists(namespace: string): void { + try { + kubectl(`delete namespace ${namespace} --wait=true --timeout=60s --ignore-not-found`); + } catch { + // ignore + } +} diff --git a/packages/plugins/sandbox-providers/kubernetes/test/integration/end-to-end-run.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/integration/end-to-end-run.test.ts new file mode 100644 index 00000000..8fb64270 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/integration/end-to-end-run.test.ts @@ -0,0 +1,205 @@ +/** + * End-to-end integration test against a local kind cluster. + * + * PREREQUISITES (operator must perform before running this test): + * 1. Create the kind cluster: + * kind create cluster --name paperclip + * 2. Pre-load the alpine image so the Job can start without network access: + * docker pull alpine:3.20 + * docker tag alpine:3.20 localhost/paperclip-agent:latest + * kind load docker-image localhost/paperclip-agent:latest --name paperclip + * 3. For the sandbox-cr backend test, the agent-sandbox controller must be installed: + * kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/install.yaml + * And a tini-bearing image pre-loaded (e.g. the same localhost/paperclip-agent:latest + * if it includes /usr/bin/tini and /bin/sh). + * 4. Set the env var and run: + * RUN_K8S_INTEGRATION_TESTS=1 pnpm test + * + * The namespace is derived from companySlug ("spike-e2e") + namespacePrefix + * ("paperclip-"), resolving to "paperclip-spike-e2e". + */ + +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import plugin from "../../src/plugin.js"; +import { createKubeConfig } from "../../src/kube-client.js"; +import { execInPod } from "../../src/pod-exec.js"; +import { sandboxCrOrchestrator } from "../../src/sandbox-cr-orchestrator.js"; +import { deleteNamespaceIfExists, kubectl, readKindKubeconfig } from "./_kind-harness.js"; + +const NAMESPACE = "paperclip-spike-e2e"; + +describe("plugin-kubernetes end-to-end", () => { + beforeAll(() => { + if (process.env.RUN_K8S_INTEGRATION_TESTS !== "1") return; + deleteNamespaceIfExists(NAMESPACE); + }); + + afterAll(() => { + if (process.env.RUN_K8S_INTEGRATION_TESTS !== "1") return; + deleteNamespaceIfExists(NAMESPACE); + }); + + // ── Job backend (stable fallback) ───────────────────────────────────────── + + it.runIf(process.env.RUN_K8S_INTEGRATION_TESTS === "1")( + "[job backend] acquireLease creates tenant + Job + supporting resources; releaseLease cascade-deletes them", + async () => { + const kubeconfig = readKindKubeconfig(); + const config = { + inCluster: false, + kubeconfig, + companySlug: "spike-e2e", + adapterType: "claude_local", + backend: "job", + imageAllowList: [] as string[], + podActivityDeadlineSec: 60, + jobTtlSecondsAfterFinished: 60, + }; + + const lease = await plugin.definition.onEnvironmentAcquireLease!({ + driverKey: "kubernetes", + config, + runId: "r-test-e2e-job", + companyId: "11111111-1111-1111-1111-111111111111", + environmentId: "env-test", + }); + + expect(lease.providerLeaseId).toMatch(/^pc-/); + + // Verify the Job exists in the tenant namespace + const jobs = kubectl(`get jobs -n ${NAMESPACE} -o name`); + expect(jobs).toContain(`job.batch/${lease.providerLeaseId}`); + + // Verify the tenant namespace has the expected supporting resources + const all = kubectl( + `get sa,role,rolebinding,resourcequota,limitrange,networkpolicy -n ${NAMESPACE} -o name`, + ); + expect(all).toContain("serviceaccount/paperclip-tenant-sa"); + expect(all).toContain("role.rbac.authorization.k8s.io/paperclip-tenant-role"); + expect(all).toContain("rolebinding.rbac.authorization.k8s.io/paperclip-tenant-rb"); + expect(all).toContain("resourcequota/paperclip-quota"); + expect(all).toContain("limitrange/paperclip-limits"); + expect(all).toContain("networkpolicy.networking.k8s.io/paperclip-deny-all"); + expect(all).toContain("networkpolicy.networking.k8s.io/paperclip-egress-allow"); + + // Verify the namespace has PSS-restricted labels + const ns = kubectl(`get namespace ${NAMESPACE} -o jsonpath='{.metadata.labels}'`); + expect(ns).toContain("pod-security.kubernetes.io/enforce"); + expect(ns).toContain("restricted"); + + // Verify the per-run Secret exists (owned by the Job for cascade deletion) + const secrets = kubectl(`get secrets -n ${NAMESPACE} -o name`); + expect(secrets).toContain(`secret/${lease.providerLeaseId}-env`); + + // Release — deletes the Job with Foreground propagation, which cascade-deletes + // the owned Secret via owner references set at acquireLease time. + await plugin.definition.onEnvironmentReleaseLease!({ + driverKey: "kubernetes", + config, + providerLeaseId: lease.providerLeaseId, + leaseMetadata: lease.metadata, + companyId: "11111111-1111-1111-1111-111111111111", + environmentId: "env-test", + }); + + // Allow a brief grace window for Foreground propagation to finish. + await new Promise((resolve) => setTimeout(resolve, 2000)); + + const jobsAfter = kubectl(`get jobs -n ${NAMESPACE} -o name 2>&1 || true`); + expect(jobsAfter).not.toContain(`job.batch/${lease.providerLeaseId}`); + }, + 180_000, + ); + + // ── Sandbox-CR backend (alpha, requires agent-sandbox controller) ────────── + + it.runIf(process.env.RUN_K8S_INTEGRATION_TESTS === "1")( + "[sandbox-cr backend] acquireLease creates Sandbox CR + supporting resources; pod becomes Ready; execInPod runs echo hello; releaseLease deletes CR", + async () => { + const kubeconfig = readKindKubeconfig(); + const config = { + inCluster: false, + kubeconfig, + companySlug: "spike-e2e", + adapterType: "claude_local", + backend: "sandbox-cr", + imageAllowList: [] as string[], + podActivityDeadlineSec: 120, + jobTtlSecondsAfterFinished: 60, + }; + + const lease = await plugin.definition.onEnvironmentAcquireLease!({ + driverKey: "kubernetes", + config, + runId: "r-test-e2e-sandbox-cr", + companyId: "22222222-2222-2222-2222-222222222222", + environmentId: "env-test-cr", + }); + + expect(lease.providerLeaseId).toMatch(/^pc-/); + + // Verify the Sandbox CR exists in the tenant namespace + const sandboxes = kubectl( + `get sandboxes.agents.x-k8s.io -n ${NAMESPACE} -o name 2>&1`, + ); + expect(sandboxes).toContain(`sandbox.agents.x-k8s.io/${lease.providerLeaseId}`); + + // Verify the per-run Secret exists (owned by the Sandbox CR) + const secrets = kubectl(`get secrets -n ${NAMESPACE} -o name`); + expect(secrets).toContain(`secret/${lease.providerLeaseId}-env`); + + // Wait for the Sandbox pod to become Ready + const kc = createKubeConfig({ inCluster: false, kubeconfig }); + const { makeKubeClients } = await import("../../src/kube-client.js"); + const clients = makeKubeClients(kc); + + await sandboxCrOrchestrator.waitForCompletion( + clients, + NAMESPACE, + lease.providerLeaseId, + { timeoutMs: 90_000, pollMs: 3000 }, + ); + + // Resolve the pod name + const podName = await sandboxCrOrchestrator.findPod( + clients, + NAMESPACE, + lease.providerLeaseId, + ); + expect(podName).toBeTruthy(); + + // Exec a simple echo command into the running pod + const execResult = await execInPod( + kc, + NAMESPACE, + podName!, + "agent", + ["echo", "hello"], + ); + + expect(execResult.exitCode).toBe(0); + expect(execResult.stdout.trim()).toBe("hello"); + + // Release — deletes the Sandbox CR with Foreground propagation. + await plugin.definition.onEnvironmentReleaseLease!({ + driverKey: "kubernetes", + config, + providerLeaseId: lease.providerLeaseId, + leaseMetadata: lease.metadata, + companyId: "22222222-2222-2222-2222-222222222222", + environmentId: "env-test-cr", + }); + + // Allow a brief grace window for Foreground propagation. + await new Promise((resolve) => setTimeout(resolve, 3000)); + + const sandboxesAfter = kubectl( + `get sandboxes.agents.x-k8s.io -n ${NAMESPACE} -o name 2>&1 || true`, + ); + expect(sandboxesAfter).not.toContain( + `sandbox.agents.x-k8s.io/${lease.providerLeaseId}`, + ); + }, + 300_000, + ); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/adapter-defaults.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/adapter-defaults.test.ts new file mode 100644 index 00000000..02beb05c --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/adapter-defaults.test.ts @@ -0,0 +1,37 @@ +import { describe, it, expect } from "vitest"; +import { getAdapterDefaults, KNOWN_ADAPTER_TYPES } from "../../src/adapter-defaults.js"; + +describe("adapter-defaults", () => { + it("returns defaults for claude_local", () => { + const d = getAdapterDefaults("claude_local"); + expect(d.runtimeImage).toBe("ghcr.io/paperclipai/agent-runtime-claude:v1"); + expect(d.envKeys).toContain("ANTHROPIC_API_KEY"); + expect(d.allowFqdns).toContain("api.anthropic.com"); + expect(d.probeCommand).toEqual(["claude", "--version"]); + }); + + it("returns defaults for codex_local", () => { + const d = getAdapterDefaults("codex_local"); + expect(d.runtimeImage).toBe("ghcr.io/paperclipai/agent-runtime-codex:v1"); + expect(d.envKeys).toContain("OPENAI_API_KEY"); + expect(d.probeCommand).toEqual(["codex", "--version"]); + }); + + it("throws on unknown adapter type", () => { + expect(() => getAdapterDefaults("nonexistent_local")).toThrow(/unknown adapter type/i); + }); + + it("KNOWN_ADAPTER_TYPES contains all 7 supported adapters", () => { + expect(KNOWN_ADAPTER_TYPES).toEqual( + new Set([ + "claude_local", + "codex_local", + "gemini_local", + "cursor_local", + "opencode_local", + "acpx_local", + "pi_local", + ]), + ); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/cilium-network-policy.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/cilium-network-policy.test.ts new file mode 100644 index 00000000..0e650363 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/cilium-network-policy.test.ts @@ -0,0 +1,60 @@ +import { describe, it, expect } from "vitest"; +import { buildCiliumNetworkPolicyManifest } from "../../src/cilium-network-policy.js"; + +describe("buildCiliumNetworkPolicyManifest", () => { + const baseInput = { + namespace: "paperclip-acme", + paperclipServerNamespace: "paperclip", + egressAllowFqdns: ["api.anthropic.com"], + egressAllowCidrs: [] as string[], + }; + + it("returns a CiliumNetworkPolicy with the correct apiVersion and kind", () => { + const cnp = buildCiliumNetworkPolicyManifest(baseInput); + expect(cnp.apiVersion).toBe("cilium.io/v2"); + expect(cnp.kind).toBe("CiliumNetworkPolicy"); + }); + + it("targets agent pods by role label", () => { + const cnp = buildCiliumNetworkPolicyManifest(baseInput); + expect(cnp.spec.endpointSelector.matchLabels["paperclip.io/role"]).toBe("agent"); + }); + + it("includes an FQDN allow rule for each adapter FQDN", () => { + const cnp = buildCiliumNetworkPolicyManifest({ + ...baseInput, + egressAllowFqdns: ["api.anthropic.com", "api.openai.com"], + }); + const fqdnRule = cnp.spec.egress.find((e: { toFQDNs?: { matchName: string }[] }) => e.toFQDNs); + expect(fqdnRule).toBeDefined(); + expect(fqdnRule.toFQDNs.map((f: { matchName: string }) => f.matchName).sort()).toEqual([ + "api.anthropic.com", + "api.openai.com", + ]); + }); + + it("permits DNS to kube-dns explicitly so FQDN resolution can happen", () => { + const cnp = buildCiliumNetworkPolicyManifest(baseInput); + const dnsRule = cnp.spec.egress.find((e: { toPorts?: { ports: { port: string }[] }[] }) => + e.toPorts?.some((tp) => tp.ports.some((p) => p.port === "53")), + ); + expect(dnsRule).toBeDefined(); + }); + + it("includes a rule for paperclip-server callback", () => { + const cnp = buildCiliumNetworkPolicyManifest(baseInput); + const cb = cnp.spec.egress.find((e: { toEndpoints?: { matchLabels: Record }[] }) => + e.toEndpoints?.some((ep) => ep.matchLabels.app === "paperclip-server"), + ); + expect(cb).toBeDefined(); + }); + + it("includes user-supplied CIDRs in toCIDRSet rule", () => { + const cnp = buildCiliumNetworkPolicyManifest({ + ...baseInput, + egressAllowCidrs: ["10.0.0.0/8"], + }); + const cidrRule = cnp.spec.egress.find((e: { toCIDRSet?: { cidr: string }[] }) => e.toCIDRSet); + expect(cidrRule.toCIDRSet[0].cidr).toBe("10.0.0.0/8"); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/image-allowlist.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/image-allowlist.test.ts new file mode 100644 index 00000000..61b6c5d7 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/image-allowlist.test.ts @@ -0,0 +1,62 @@ +import { describe, it, expect } from "vitest"; +import { globMatch, resolveImage } from "../../src/image-allowlist.js"; + +describe("globMatch", () => { + it("matches exact image", () => { + expect(globMatch("ghcr.io/paperclipai/agent-runtime-claude:v1", "ghcr.io/paperclipai/agent-runtime-claude:v1")).toBe(true); + }); + + it("matches single-character wildcard", () => { + expect(globMatch("ghcr.io/x:v?", "ghcr.io/x:v1")).toBe(true); + expect(globMatch("ghcr.io/x:v?", "ghcr.io/x:v12")).toBe(false); + }); + + it("matches multi-character wildcard", () => { + expect(globMatch("ghcr.io/paperclipai/*:v1", "ghcr.io/paperclipai/agent-runtime-claude:v1")).toBe(true); + expect(globMatch("ghcr.io/paperclipai/*:v1", "docker.io/other/img:v1")).toBe(false); + }); + + it("does not allow wildcard to span slashes by default", () => { + expect(globMatch("ghcr.io/*:v1", "ghcr.io/paperclipai/agent-runtime-claude:v1")).toBe(false); + }); +}); + +describe("resolveImage", () => { + const defaults = { runtimeImage: "ghcr.io/paperclipai/agent-runtime-claude:v1" }; + + it("uses adapter default when no override", () => { + expect(resolveImage({ imageOverride: null }, defaults, { imageAllowList: [], imageRegistry: undefined })).toBe( + "ghcr.io/paperclipai/agent-runtime-claude:v1", + ); + }); + + it("rewrites registry when imageRegistry is set", () => { + expect( + resolveImage( + { imageOverride: null }, + defaults, + { imageAllowList: [], imageRegistry: "registry.example.com/paperclip" }, + ), + ).toBe("registry.example.com/paperclip/agent-runtime-claude:v1"); + }); + + it("accepts imageOverride when in allowlist", () => { + expect( + resolveImage( + { imageOverride: "registry.example.com/mine:v2" }, + defaults, + { imageAllowList: ["registry.example.com/*:v2"], imageRegistry: undefined }, + ), + ).toBe("registry.example.com/mine:v2"); + }); + + it("rejects imageOverride not in allowlist", () => { + expect(() => + resolveImage( + { imageOverride: "evil.io/img:latest" }, + defaults, + { imageAllowList: ["registry.example.com/*"], imageRegistry: undefined }, + ), + ).toThrow(/not in allowlist/); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/job-orchestrator.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/job-orchestrator.test.ts new file mode 100644 index 00000000..eab2c009 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/job-orchestrator.test.ts @@ -0,0 +1,101 @@ +import { describe, it, expect, vi } from "vitest"; +import { createJob, deleteJob, getJobStatus, findPodForJob, JobTimeoutError, streamPodLogs, waitForJobCompletion } from "../../src/job-orchestrator.js"; + +describe("createJob", () => { + it("calls batch.createNamespacedJob with the manifest", async () => { + const create = vi.fn().mockResolvedValue({ metadata: { uid: "abc-uid" } }); + const clients = { batch: { createNamespacedJob: create } }; + const jobManifest = { apiVersion: "batch/v1", kind: "Job", metadata: { name: "r-1", namespace: "ns" }, spec: { template: {} } }; + const result = await createJob(clients as never, "ns", jobManifest); + expect(create).toHaveBeenCalledWith({ namespace: "ns", body: jobManifest }); + expect(result.uid).toBe("abc-uid"); + }); +}); + +describe("getJobStatus", () => { + it("returns phase=Succeeded when succeeded count is 1", async () => { + const get = vi.fn().mockResolvedValue({ status: { succeeded: 1, conditions: [{ type: "Complete", status: "True" }] } }); + const clients = { batch: { readNamespacedJobStatus: get } }; + const status = await getJobStatus(clients as never, "ns", "r-1"); + expect(status.phase).toBe("Succeeded"); + expect(status.complete).toBe(true); + }); + + it("returns phase=Failed when failed count is >0", async () => { + const get = vi.fn().mockResolvedValue({ status: { failed: 1, conditions: [{ type: "Failed", status: "True", reason: "DeadlineExceeded" }] } }); + const clients = { batch: { readNamespacedJobStatus: get } }; + const status = await getJobStatus(clients as never, "ns", "r-1"); + expect(status.phase).toBe("Failed"); + expect(status.reason).toBe("DeadlineExceeded"); + }); + + it("returns phase=Running when active count is >0", async () => { + const get = vi.fn().mockResolvedValue({ status: { active: 1 } }); + const clients = { batch: { readNamespacedJobStatus: get } }; + const status = await getJobStatus(clients as never, "ns", "r-1"); + expect(status.phase).toBe("Running"); + }); + + it("returns phase=Pending when no active/succeeded/failed counters set", async () => { + const get = vi.fn().mockResolvedValue({ status: {} }); + const clients = { batch: { readNamespacedJobStatus: get } }; + const status = await getJobStatus(clients as never, "ns", "r-1"); + expect(status.phase).toBe("Pending"); + }); +}); + +describe("findPodForJob", () => { + it("lists pods by job-name label and returns the first running pod", async () => { + const list = vi.fn().mockResolvedValue({ items: [{ metadata: { name: "r-1-xyz" }, status: { phase: "Running" } }] }); + const clients = { core: { listNamespacedPod: list } }; + const podName = await findPodForJob(clients as never, "ns", "r-1"); + expect(list).toHaveBeenCalledWith(expect.objectContaining({ namespace: "ns", labelSelector: "job-name=r-1" })); + expect(podName).toBe("r-1-xyz"); + }); + + it("returns null when no pod is found", async () => { + const list = vi.fn().mockResolvedValue({ items: [] }); + const clients = { core: { listNamespacedPod: list } }; + const podName = await findPodForJob(clients as never, "ns", "r-1"); + expect(podName).toBeNull(); + }); +}); + +describe("deleteJob", () => { + it("calls batch.deleteNamespacedJob with foreground propagation", async () => { + const del = vi.fn().mockResolvedValue({}); + const clients = { batch: { deleteNamespacedJob: del } }; + await deleteJob(clients as never, "ns", "r-1"); + expect(del).toHaveBeenCalledWith( + expect.objectContaining({ + namespace: "ns", + name: "r-1", + propagationPolicy: "Foreground", + }), + ); + }); +}); + +describe("streamPodLogs", () => { + it("emits pod log response bodies as stdout because Kubernetes pod logs are combined", async () => { + const readNamespacedPodLog = vi.fn().mockResolvedValue({ body: "hello\n" }); + const clients = { core: { readNamespacedPodLog } }; + const chunks: { stream: "stdout" | "stderr"; text: string }[] = []; + await streamPodLogs(clients as never, "ns", "pod-1", async (stream, text) => { + chunks.push({ stream, text }); + }); + + expect(readNamespacedPodLog).toHaveBeenCalledWith({ namespace: "ns", name: "pod-1" }); + expect(chunks).toEqual([{ stream: "stdout", text: "hello\n" }]); + }); +}); + +describe("waitForJobCompletion", () => { + it("throws JobTimeoutError when the deadline is exceeded", async () => { + const get = vi.fn().mockResolvedValue({ status: { active: 1 } }); + const clients = { batch: { readNamespacedJobStatus: get } }; + await expect( + waitForJobCompletion(clients as never, "ns", "r-1", { timeoutMs: 50, pollMs: 10 }), + ).rejects.toBeInstanceOf(JobTimeoutError); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/kube-client.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/kube-client.test.ts new file mode 100644 index 00000000..55a1731e --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/kube-client.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect, vi } from "vitest"; +import { KubeConfig } from "@kubernetes/client-node"; +import { createKubeConfig } from "../../src/kube-client.js"; + +describe("createKubeConfig", () => { + it("loads from inline kubeconfig string", () => { + const yaml = `apiVersion: v1 +kind: Config +clusters: + - name: test + cluster: + server: https://fake.example.com +contexts: + - name: test + context: + cluster: test + user: test +current-context: test +users: + - name: test + user: + token: fake-token +`; + const kc = createKubeConfig({ inCluster: false, kubeconfig: yaml }); + expect(kc.getCurrentContext()).toBe("test"); + expect(kc.getCurrentCluster()?.server).toBe("https://fake.example.com"); + }); + + it("loads from-cluster config when inCluster=true", () => { + const spy = vi.spyOn(KubeConfig.prototype, "loadFromCluster").mockImplementation(function (this: KubeConfig) { + this.loadFromString(`apiVersion: v1 +kind: Config +clusters: [{name: in-cluster, cluster: {server: 'https://kubernetes.default.svc'}}] +contexts: [{name: in-cluster, context: {cluster: in-cluster, user: in-cluster}}] +current-context: in-cluster +users: [{name: in-cluster, user: {token: tok}}]`); + }); + const kc = createKubeConfig({ inCluster: true }); + expect(spy).toHaveBeenCalledOnce(); + expect(kc.getCurrentContext()).toBe("in-cluster"); + spy.mockRestore(); + }); + + it("throws when neither inCluster nor kubeconfig string is provided", () => { + expect(() => createKubeConfig({ inCluster: false })).toThrow(/requires/i); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/network-policy.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/network-policy.test.ts new file mode 100644 index 00000000..db5d1101 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/network-policy.test.ts @@ -0,0 +1,65 @@ +import { describe, it, expect } from "vitest"; +import { buildNetworkPolicyManifests } from "../../src/network-policy.js"; + +describe("buildNetworkPolicyManifests", () => { + const baseInput = { + namespace: "paperclip-acme", + paperclipServerNamespace: "paperclip", + egressAllowFqdns: [] as string[], + egressAllowCidrs: [] as string[], + }; + + it("produces a deny-all + egress allow pair", () => { + const manifests = buildNetworkPolicyManifests(baseInput); + expect(manifests).toHaveLength(2); + expect(manifests[0].metadata.name).toBe("paperclip-deny-all"); + expect(manifests[1].metadata.name).toBe("paperclip-egress-allow"); + }); + + it("deny-all has no ingress/egress rules and applies to all pods", () => { + const [denyAll] = buildNetworkPolicyManifests(baseInput); + expect(denyAll.spec.podSelector).toEqual({}); + expect(denyAll.spec.policyTypes).toEqual(["Ingress", "Egress"]); + expect(denyAll.spec.ingress).toBeUndefined(); + expect(denyAll.spec.egress).toBeUndefined(); + }); + + it("egress allow includes kube-dns and paperclip-server callback", () => { + const [, egress] = buildNetworkPolicyManifests(baseInput); + const rules = egress.spec.egress; + const dnsRule = rules.find((r: { ports?: { protocol: string; port: number }[] }) => + r.ports?.some((p) => p.port === 53), + ); + expect(dnsRule).toBeDefined(); + const paperclipRule = rules.find((r: { to: { namespaceSelector?: { matchLabels?: Record } }[] }) => + r.to.some((t) => t.namespaceSelector?.matchLabels?.["kubernetes.io/metadata.name"] === "paperclip"), + ); + expect(paperclipRule).toBeDefined(); + }); + + it("includes user-supplied CIDRs in egress allow", () => { + const [, egress] = buildNetworkPolicyManifests({ ...baseInput, egressAllowCidrs: ["10.0.0.0/8"] }); + const cidrRule = egress.spec.egress.find((r: { to: { ipBlock?: { cidr: string } }[] }) => + r.to.some((t) => t.ipBlock?.cidr === "10.0.0.0/8"), + ); + expect(cidrRule).toBeDefined(); + }); + + it("adds a public HTTPS fallback when standard mode receives FQDN allow-list entries", () => { + const [, egress] = buildNetworkPolicyManifests({ ...baseInput, egressAllowFqdns: ["api.anthropic.com"] }); + const publicHttpsRule = egress.spec.egress.find((r: { to: { ipBlock?: { cidr: string; except?: string[] } }[]; ports?: { port: number }[] }) => + r.to.some((t) => t.ipBlock?.cidr === "0.0.0.0/0") && r.ports?.some((p) => p.port === 443), + ); + expect(publicHttpsRule).toBeDefined(); + expect(publicHttpsRule.to[0].ipBlock.except).toContain("10.0.0.0/8"); + }); + + it("uses paperclip-server pod label selector for callback ingress to paperclip ns", () => { + const [, egress] = buildNetworkPolicyManifests(baseInput); + const callbackRule = egress.spec.egress.find((r: { to: { podSelector?: { matchLabels?: Record } }[] }) => + r.to.some((t) => t.podSelector?.matchLabels?.app === "paperclip-server"), + ); + expect(callbackRule).toBeDefined(); + expect(callbackRule.ports[0].port).toBe(3100); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin.test.ts new file mode 100644 index 00000000..b005b748 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/plugin.test.ts @@ -0,0 +1,94 @@ +import { describe, it, expect } from "vitest"; +import plugin from "../../src/plugin.js"; + +describe("plugin", () => { + it("exports the kubernetes driver", () => { + expect(plugin.definition.onEnvironmentAcquireLease).toBeTypeOf("function"); + expect(plugin.definition.onEnvironmentValidateConfig).toBeTypeOf("function"); + }); + + it("validateConfig accepts inCluster=true config", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true }, + }); + expect(result.ok).toBe(true); + }); + + it("validateConfig rejects missing auth", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: {}, + }); + expect(result.ok).toBe(false); + expect(result.errors?.[0]).toMatch(/requires one of `inCluster`/); + }); + + it("validateConfig normalizes defaults", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true }, + }); + expect(result.ok).toBe(true); + expect(result.normalizedConfig).toEqual( + expect.objectContaining({ + namespacePrefix: "paperclip-", + egressMode: "standard", + jobTtlSecondsAfterFinished: 900, + podActivityDeadlineSec: 3600, + adapterType: "claude_local", + backend: "sandbox-cr", // new default + }), + ); + }); + + it("validateConfig accepts backend=sandbox-cr explicitly", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true, backend: "sandbox-cr" }, + }); + expect(result.ok).toBe(true); + expect(result.normalizedConfig?.backend).toBe("sandbox-cr"); + }); + + it("validateConfig accepts backend=job (stable fallback)", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true, backend: "job" }, + }); + expect(result.ok).toBe(true); + expect(result.normalizedConfig?.backend).toBe("job"); + }); + + it("validateConfig rejects unknown backend value", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true, backend: "kata-fc" }, + }); + expect(result.ok).toBe(false); + }); + + it("onHealth returns ok", async () => { + const result = await plugin.definition.onHealth!(); + expect(result.status).toBe("ok"); + }); + + it("validateConfig warns about FQDN limitation in standard mode", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true, adapterType: "claude_local" }, + }); + expect(result.ok).toBe(true); + expect(result.warnings).toBeDefined(); + expect(result.warnings?.some((w) => w.includes("api.anthropic.com"))).toBe(true); + }); + + it("validateConfig does NOT warn when egressMode is cilium", async () => { + const result = await plugin.definition.onEnvironmentValidateConfig!({ + driverKey: "kubernetes", + config: { inCluster: true, adapterType: "claude_local", egressMode: "cilium" }, + }); + expect(result.ok).toBe(true); + expect(result.warnings).toBeUndefined(); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/pod-spec-builder.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/pod-spec-builder.test.ts new file mode 100644 index 00000000..75d1da3c --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/pod-spec-builder.test.ts @@ -0,0 +1,95 @@ +import { describe, it, expect } from "vitest"; +import { buildJobManifest } from "../../src/pod-spec-builder.js"; + +const baseInput = { + namespace: "paperclip-acme", + jobName: "r-01h00000000000000000000000", + adapterType: "claude_local", + image: "ghcr.io/paperclipai/agent-runtime-claude:v1", + envSecretName: "r-01h00000000000000000000000-env", + serviceAccountName: "paperclip-tenant-sa", + labels: { "paperclip.io/run-id": "r1" }, + resources: { requests: { cpu: "250m", memory: "512Mi" }, limits: { cpu: "2", memory: "4Gi" } }, + runtimeClassName: undefined, + activeDeadlineSec: 3600, + ttlSecondsAfterFinished: 900, +}; + +describe("buildJobManifest", () => { + it("returns a Job manifest with the correct apiVersion and kind", () => { + const job = buildJobManifest(baseInput); + expect(job.apiVersion).toBe("batch/v1"); + expect(job.kind).toBe("Job"); + }); + + it("sets Job-level lifecycle controls: backoffLimit=0, ttlSecondsAfterFinished, activeDeadlineSeconds", () => { + const job = buildJobManifest({ ...baseInput, activeDeadlineSec: 1800, ttlSecondsAfterFinished: 600 }); + expect(job.spec.backoffLimit).toBe(0); + expect(job.spec.ttlSecondsAfterFinished).toBe(600); + expect(job.spec.activeDeadlineSeconds).toBe(1800); + }); + + it("sets the security context to non-root, drop ALL caps, read-only rootFS, seccomp RuntimeDefault", () => { + const job = buildJobManifest(baseInput); + const podSec = job.spec.template.spec.securityContext; + expect(podSec.runAsNonRoot).toBe(true); + expect(podSec.runAsUser).toBe(1000); + expect(podSec.fsGroupChangePolicy).toBe("OnRootMismatch"); + expect(podSec.seccompProfile.type).toBe("RuntimeDefault"); + + const container = job.spec.template.spec.containers[0]; + expect(container.securityContext.runAsNonRoot).toBe(true); + expect(container.securityContext.readOnlyRootFilesystem).toBe(true); + expect(container.securityContext.allowPrivilegeEscalation).toBe(false); + expect(container.securityContext.capabilities.drop).toEqual(["ALL"]); + }); + + it("wraps the entrypoint in tini for PID 1", () => { + const job = buildJobManifest(baseInput); + const container = job.spec.template.spec.containers[0]; + expect(container.command).toEqual(["/usr/bin/tini", "--", "/usr/local/bin/paperclip-agent-shim"]); + }); + + it("declares explicit writable emptyDir mounts for the standard agent paths", () => { + const job = buildJobManifest(baseInput); + const mounts = job.spec.template.spec.containers[0].volumeMounts; + const mountPaths = mounts.map((m: { mountPath: string }) => m.mountPath).sort(); + expect(mountPaths).toEqual(["/home/paperclip", "/home/paperclip/.cache", "/tmp", "/workspace"]); + + const volumes = job.spec.template.spec.volumes; + expect(volumes.every((v: { emptyDir?: unknown }) => v.emptyDir !== undefined)).toBe(true); + }); + + it("envFrom references the per-run secret", () => { + const job = buildJobManifest(baseInput); + const envFrom = job.spec.template.spec.containers[0].envFrom; + expect(envFrom[0].secretRef.name).toBe(baseInput.envSecretName); + }); + + it("applies runtimeClassName when set", () => { + const job = buildJobManifest({ ...baseInput, runtimeClassName: "kata-fc" }); + expect(job.spec.template.spec.runtimeClassName).toBe("kata-fc"); + }); + + it("does not set runtimeClassName when unset", () => { + const job = buildJobManifest(baseInput); + expect(job.spec.template.spec.runtimeClassName).toBeUndefined(); + }); + + it("sets pod restartPolicy=Never (required for Job)", () => { + const job = buildJobManifest(baseInput); + expect(job.spec.template.spec.restartPolicy).toBe("Never"); + }); + + it("disables automountServiceAccountToken to avoid exposing an unnecessary SA token", () => { + const job = buildJobManifest(baseInput); + expect(job.spec.template.spec.automountServiceAccountToken).toBe(false); + }); + + it("applies the provided labels to both Job metadata and pod template", () => { + const job = buildJobManifest(baseInput); + expect(job.metadata.labels["paperclip.io/run-id"]).toBe("r1"); + expect(job.spec.template.metadata.labels["paperclip.io/run-id"]).toBe("r1"); + expect(job.spec.template.metadata.labels["paperclip.io/role"]).toBe("agent"); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-builder.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-builder.test.ts new file mode 100644 index 00000000..97beb8e9 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-builder.test.ts @@ -0,0 +1,137 @@ +import { describe, it, expect } from "vitest"; +import { buildSandboxCrManifest } from "../../src/sandbox-cr-builder.js"; + +const baseInput = { + namespace: "paperclip-acme", + sandboxName: "pc-01h00000000000000000000000", + adapterType: "claude_local", + image: "ghcr.io/paperclipai/agent-runtime-claude:v1", + envSecretName: "pc-01h00000000000000000000000-env", + serviceAccountName: "paperclip-tenant-sa", + labels: { "paperclip.io/run-id": "r1" }, + resources: { + requests: { cpu: "250m", memory: "512Mi" }, + limits: { cpu: "2", memory: "4Gi" }, + }, + runtimeClassName: undefined, +}; + +describe("buildSandboxCrManifest", () => { + it("returns a Sandbox CR with the correct apiVersion and kind", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.apiVersion).toBe("agents.x-k8s.io/v1alpha1"); + expect(cr.kind).toBe("Sandbox"); + }); + + it("sets metadata name and namespace correctly", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.metadata.name).toBe(baseInput.sandboxName); + expect(cr.metadata.namespace).toBe(baseInput.namespace); + }); + + it("does NOT set ownerReferences (out-of-cluster server, explicit release path)", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.metadata.ownerReferences).toBeUndefined(); + }); + + it("sets restartPolicy=Always on the pod template (required for long-lived Sandbox pod)", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.spec.podTemplate.spec.restartPolicy).toBe("Always"); + }); + + it("uses sleep-infinity entrypoint via Tini for multi-command exec", () => { + const cr = buildSandboxCrManifest(baseInput); + const container = cr.spec.podTemplate.spec.containers[0]; + expect(container.command).toEqual([ + "/usr/bin/tini", + "--", + "/bin/sh", + "-c", + "sleep infinity", + ]); + }); + + it("applies the same security baseline as Job backend (non-root, drop ALL, RO rootFS, seccomp)", () => { + const cr = buildSandboxCrManifest(baseInput); + const podSec = cr.spec.podTemplate.spec.securityContext; + expect(podSec.runAsNonRoot).toBe(true); + expect(podSec.runAsUser).toBe(1000); + expect(podSec.fsGroupChangePolicy).toBe("OnRootMismatch"); + expect(podSec.seccompProfile.type).toBe("RuntimeDefault"); + + const container = cr.spec.podTemplate.spec.containers[0]; + expect(container.securityContext.runAsNonRoot).toBe(true); + expect(container.securityContext.readOnlyRootFilesystem).toBe(true); + expect(container.securityContext.allowPrivilegeEscalation).toBe(false); + expect(container.securityContext.capabilities.drop).toEqual(["ALL"]); + }); + + it("disables automountServiceAccountToken", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.spec.podTemplate.spec.automountServiceAccountToken).toBe(false); + }); + + it("declares emptyDir volume mounts for standard agent paths", () => { + const cr = buildSandboxCrManifest(baseInput); + const mounts = cr.spec.podTemplate.spec.containers[0].volumeMounts; + const mountPaths = mounts + .map((m: { mountPath: string }) => m.mountPath) + .sort(); + expect(mountPaths).toEqual([ + "/home/paperclip", + "/home/paperclip/.cache", + "/tmp", + "/workspace", + ]); + + const volumes = cr.spec.podTemplate.spec.volumes; + expect( + volumes.every((v: { emptyDir?: unknown }) => v.emptyDir !== undefined), + ).toBe(true); + }); + + it("envFrom references the per-run secret", () => { + const cr = buildSandboxCrManifest(baseInput); + const envFrom = cr.spec.podTemplate.spec.containers[0].envFrom; + expect(envFrom[0].secretRef.name).toBe(baseInput.envSecretName); + }); + + it("applies runtimeClassName when set", () => { + const cr = buildSandboxCrManifest({ + ...baseInput, + runtimeClassName: "kata-fc", + }); + expect(cr.spec.podTemplate.spec.runtimeClassName).toBe("kata-fc"); + }); + + it("does not set runtimeClassName when unset", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.spec.podTemplate.spec.runtimeClassName).toBeUndefined(); + }); + + it("applies provided labels to CR metadata and pod template labels (with role=agent added)", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.metadata.labels["paperclip.io/run-id"]).toBe("r1"); + expect( + cr.spec.podTemplate.metadata.labels["paperclip.io/run-id"], + ).toBe("r1"); + expect(cr.spec.podTemplate.metadata.labels["paperclip.io/role"]).toBe( + "agent", + ); + }); + + it("applies imagePullSecrets when provided", () => { + const cr = buildSandboxCrManifest({ + ...baseInput, + imagePullSecrets: ["my-pull-secret"], + }); + expect(cr.spec.podTemplate.spec.imagePullSecrets).toEqual([ + { name: "my-pull-secret" }, + ]); + }); + + it("does not set imagePullSecrets when not provided", () => { + const cr = buildSandboxCrManifest(baseInput); + expect(cr.spec.podTemplate.spec.imagePullSecrets).toBeUndefined(); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-orchestrator.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-orchestrator.test.ts new file mode 100644 index 00000000..8037968b --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-orchestrator.test.ts @@ -0,0 +1,216 @@ +import { describe, it, expect, vi } from "vitest"; +import { + createSandboxCr, + deleteSandboxCr, + getSandboxCrStatus, + findPodForSandbox, + SandboxCrTimeoutError, + waitForSandboxReady, +} from "../../src/sandbox-cr-orchestrator.js"; + +const SANDBOX_GROUP = "agents.x-k8s.io"; +const SANDBOX_VERSION = "v1alpha1"; +const SANDBOX_PLURAL = "sandboxes"; + +// Helpers to build mock CR objects with given phase +function makeCr(phase: string, podName?: string): Record { + return { + metadata: { uid: "sandbox-uid-123" }, + status: { + phase, + ...(podName ? { podName } : {}), + }, + }; +} + +describe("createSandboxCr", () => { + it("calls custom.createNamespacedCustomObject with the correct params", async () => { + const create = vi.fn().mockResolvedValue({ metadata: { uid: "test-uid" } }); + const clients = { custom: { createNamespacedCustomObject: create } }; + const manifest = { + apiVersion: "agents.x-k8s.io/v1alpha1", + kind: "Sandbox", + metadata: { name: "pc-abc", namespace: "paperclip-acme" }, + }; + const result = await createSandboxCr(clients as never, "paperclip-acme", manifest); + expect(create).toHaveBeenCalledWith({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace: "paperclip-acme", + plural: SANDBOX_PLURAL, + body: manifest, + }); + expect(result.uid).toBe("test-uid"); + }); + + it("throws if the API response has no UID", async () => { + const create = vi.fn().mockResolvedValue({ metadata: {} }); + const clients = { custom: { createNamespacedCustomObject: create } }; + await expect( + createSandboxCr(clients as never, "ns", {}), + ).rejects.toThrow("Sandbox CR created without a UID"); + }); +}); + +describe("getSandboxCrStatus", () => { + it("maps phase=Ready to SandboxStatus.phase=Running with active=1", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Ready")); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc"); + expect(status.phase).toBe("Running"); + expect(status.active).toBe(1); + expect(status.complete).toBe(false); + }); + + it("maps phase=Pending to SandboxStatus.phase=Pending", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Pending")); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc"); + expect(status.phase).toBe("Pending"); + expect(status.active).toBe(0); + }); + + it("maps phase=Failed to SandboxStatus.phase=Failed with failed=1", async () => { + const get = vi.fn().mockResolvedValue({ + metadata: { uid: "uid-1" }, + status: { + phase: "Failed", + conditions: [ + { type: "Failed", reason: "ImagePullFailed", message: "no image" }, + ], + }, + }); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc"); + expect(status.phase).toBe("Failed"); + expect(status.failed).toBe(1); + expect(status.reason).toBe("ImagePullFailed"); + }); + + it("maps phase=Terminating to SandboxStatus.phase=Running with reason=Terminating", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Terminating")); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc"); + expect(status.phase).toBe("Running"); + expect(status.reason).toBe("Terminating"); + }); +}); + +describe("findPodForSandbox", () => { + it("returns status.podName from the Sandbox CR when set", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Ready", "pc-abc-pod-xyz")); + const clients = { + custom: { getNamespacedCustomObject: get }, + core: { listNamespacedPod: vi.fn() }, + }; + const podName = await findPodForSandbox(clients as never, "ns", "pc-abc"); + expect(podName).toBe("pc-abc-pod-xyz"); + // Should NOT have called listNamespacedPod (primary path succeeded) + expect(clients.core.listNamespacedPod).not.toHaveBeenCalled(); + }); + + it("falls back to pod listing when status.podName is absent", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Pending")); // no podName + const list = vi.fn().mockResolvedValue({ + items: [ + { + metadata: { name: "pc-abc-001", labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" } }, + status: { phase: "Running" }, + }, + ], + }); + const clients = { + custom: { getNamespacedCustomObject: get }, + core: { listNamespacedPod: list }, + }; + const podName = await findPodForSandbox(clients as never, "ns", "pc-abc"); + // name starts with "pc-abc" → matched by prefix heuristic + expect(podName).toBe("pc-abc-001"); + }); + + it("returns null when no pod is found in fallback", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Pending")); + const list = vi.fn().mockResolvedValue({ items: [] }); + const clients = { + custom: { getNamespacedCustomObject: get }, + core: { listNamespacedPod: list }, + }; + const podName = await findPodForSandbox(clients as never, "ns", "pc-abc"); + expect(podName).toBeNull(); + }); +}); + +describe("deleteSandboxCr", () => { + it("calls custom.deleteNamespacedCustomObject with Foreground propagation", async () => { + const del = vi.fn().mockResolvedValue({}); + const clients = { custom: { deleteNamespacedCustomObject: del } }; + await deleteSandboxCr(clients as never, "ns", "pc-abc"); + expect(del).toHaveBeenCalledWith( + expect.objectContaining({ + group: SANDBOX_GROUP, + version: SANDBOX_VERSION, + namespace: "ns", + plural: SANDBOX_PLURAL, + name: "pc-abc", + propagationPolicy: "Foreground", + }), + ); + }); +}); + +describe("waitForSandboxReady", () => { + it("resolves immediately when Sandbox is already Ready", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Ready")); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await waitForSandboxReady( + clients as never, + "ns", + "pc-abc", + { timeoutMs: 5000, pollMs: 10 }, + ); + expect(status.phase).toBe("Running"); // Ready maps to Running + expect(get).toHaveBeenCalledTimes(1); + }); + + it("polls until Ready", async () => { + const get = vi + .fn() + .mockResolvedValueOnce(makeCr("Pending")) + .mockResolvedValueOnce(makeCr("Pending")) + .mockResolvedValueOnce(makeCr("Ready")); + const clients = { custom: { getNamespacedCustomObject: get } }; + const status = await waitForSandboxReady( + clients as never, + "ns", + "pc-abc", + { timeoutMs: 5000, pollMs: 10 }, + ); + expect(status.phase).toBe("Running"); + expect(get).toHaveBeenCalledTimes(3); + }); + + it("throws SandboxCrTimeoutError when deadline is exceeded", async () => { + const get = vi.fn().mockResolvedValue(makeCr("Pending")); + const clients = { custom: { getNamespacedCustomObject: get } }; + await expect( + waitForSandboxReady(clients as never, "ns", "pc-abc", { + timeoutMs: 50, + pollMs: 10, + }), + ).rejects.toBeInstanceOf(SandboxCrTimeoutError); + }); + + it("throws an error describing the failure when Sandbox fails", async () => { + const get = vi.fn().mockResolvedValue({ + metadata: { uid: "u1" }, + status: { phase: "Failed", conditions: [{ type: "Failed", reason: "OOMKilled" }] }, + }); + const clients = { custom: { getNamespacedCustomObject: get } }; + await expect( + waitForSandboxReady(clients as never, "ns", "pc-abc", { + timeoutMs: 5000, + pollMs: 10, + }), + ).rejects.toThrow(/failed.*OOMKilled/i); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/secret-manager.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/secret-manager.test.ts new file mode 100644 index 00000000..b608c647 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/secret-manager.test.ts @@ -0,0 +1,68 @@ +import { describe, it, expect, vi } from "vitest"; +import { createPerRunSecret } from "../../src/secret-manager.js"; + +describe("createPerRunSecret", () => { + const baseInput = { + namespace: "paperclip-acme", + secretName: "r-abcd-env", + runId: "r-abcd", + ownerKind: "Job", + ownerApiVersion: "batch/v1", + ownerName: "r-abcd", + ownerUid: "11111111-1111-1111-1111-111111111111", + bootstrapToken: "tok-xyz", + adapterEnv: { ANTHROPIC_API_KEY: "sk-test" }, + }; + + it("creates a Secret with the correct name and namespace", async () => { + const created: { body: Record }[] = []; + const clients = { + core: { createNamespacedSecret: vi.fn(async (args: { body: Record }) => { created.push(args); }) }, + }; + await createPerRunSecret(clients as never, baseInput); + expect(clients.core.createNamespacedSecret).toHaveBeenCalledOnce(); + const body = created[0].body as { metadata: { name: string; namespace: string } }; + expect(body.metadata.name).toBe("r-abcd-env"); + expect(body.metadata.namespace).toBe("paperclip-acme"); + }); + + it("includes BOOTSTRAP_TOKEN and adapter env keys in stringData", async () => { + const created: { body: Record }[] = []; + const clients = { + core: { createNamespacedSecret: vi.fn(async (args: { body: Record }) => { created.push(args); }) }, + }; + await createPerRunSecret(clients as never, baseInput); + const body = created[0].body as { stringData: Record }; + expect(body.stringData.BOOTSTRAP_TOKEN).toBe("tok-xyz"); + expect(body.stringData.ANTHROPIC_API_KEY).toBe("sk-test"); + }); + + it("sets ownerReferences to the owner resource for cascade delete", async () => { + const created: { body: Record }[] = []; + const clients = { + core: { createNamespacedSecret: vi.fn(async (args: { body: Record }) => { created.push(args); }) }, + }; + await createPerRunSecret(clients as never, baseInput); + const body = created[0].body as { metadata: { ownerReferences: { uid: string; controller: boolean }[] } }; + expect(body.metadata.ownerReferences).toHaveLength(1); + expect(body.metadata.ownerReferences[0].uid).toBe("11111111-1111-1111-1111-111111111111"); + expect(body.metadata.ownerReferences[0].controller).toBe(true); + }); + + it("throws if adapterEnv contains BOOTSTRAP_TOKEN", async () => { + const clients = { core: { createNamespacedSecret: vi.fn() } }; + await expect( + createPerRunSecret(clients as never, { + ...baseInput, + adapterEnv: { BOOTSTRAP_TOKEN: "evil" }, + }), + ).rejects.toThrow(/BOOTSTRAP_TOKEN/); + }); + + it("throws if ownerUid is empty", async () => { + const clients = { core: { createNamespacedSecret: vi.fn() } }; + await expect( + createPerRunSecret(clients as never, { ...baseInput, ownerUid: "" }), + ).rejects.toThrow(/ownerUid/); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/tenant-orchestrator.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/tenant-orchestrator.test.ts new file mode 100644 index 00000000..d61b278e --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/tenant-orchestrator.test.ts @@ -0,0 +1,153 @@ +import { describe, it, expect, vi } from "vitest"; +import { ensureTenant } from "../../src/tenant-orchestrator.js"; + +function makeMockClients() { + const calls: { kind: string; name: string; namespace?: string; body?: unknown }[] = []; + function track(kind: string) { + return vi.fn(async (...args: unknown[]) => { + const arg = (args[0] ?? {}) as { name?: string; namespace?: string; body?: unknown }; + calls.push({ kind, name: arg.name ?? "", namespace: arg.namespace, body: arg.body }); + return { body: arg.body }; + }); + } + return { + calls, + core: { + createNamespace: track("Namespace"), + readNamespacedServiceAccount: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedServiceAccount: track("ServiceAccount"), + replaceNamespacedServiceAccount: track("ServiceAccountReplace"), + readNamespacedResourceQuota: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedResourceQuota: track("ResourceQuota"), + replaceNamespacedResourceQuota: track("ResourceQuotaReplace"), + readNamespacedLimitRange: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedLimitRange: track("LimitRange"), + replaceNamespacedLimitRange: track("LimitRangeReplace"), + readNamespace: vi.fn().mockRejectedValue({ code: 404 }), + }, + rbac: { + readNamespacedRole: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedRole: track("Role"), + replaceNamespacedRole: track("RoleReplace"), + readNamespacedRoleBinding: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedRoleBinding: track("RoleBinding"), + replaceNamespacedRoleBinding: track("RoleBindingReplace"), + }, + networking: { + readNamespacedNetworkPolicy: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedNetworkPolicy: track("NetworkPolicy"), + replaceNamespacedNetworkPolicy: track("NetworkPolicyReplace"), + deleteNamespacedNetworkPolicy: vi.fn().mockRejectedValue({ code: 404 }), + }, + custom: { + getNamespacedCustomObject: vi.fn().mockRejectedValue({ code: 404 }), + createNamespacedCustomObject: track("CiliumNetworkPolicy"), + replaceNamespacedCustomObject: track("CiliumNetworkPolicyReplace"), + deleteNamespacedCustomObject: vi.fn().mockRejectedValue({ code: 404 }), + }, + }; +} + +describe("ensureTenant", () => { + const baseInput = { + namespace: "paperclip-acme", + companyId: "11111111-1111-1111-1111-111111111111", + paperclipServerNamespace: "paperclip", + serviceAccountAnnotations: {}, + egressMode: "standard" as const, + egressAllowFqdns: ["api.anthropic.com"], + egressAllowCidrs: [] as string[], + resourceQuota: { pods: "20", requestsCpu: "5", requestsMemory: "20Gi", limitsCpu: "20", limitsMemory: "80Gi" }, + }; + + it("creates all required resources in the correct order on a fresh tenant", async () => { + const clients = makeMockClients(); + await ensureTenant(clients as never, baseInput); + const order = clients.calls.map((c) => c.kind); + expect(order).toEqual([ + "Namespace", + "ServiceAccount", + "Role", + "RoleBinding", + "ResourceQuota", + "LimitRange", + "NetworkPolicy", + "NetworkPolicy", + ]); + }); + + it("creates a CiliumNetworkPolicy instead of standard egress when egressMode=cilium", async () => { + const clients = makeMockClients(); + await ensureTenant(clients as never, { ...baseInput, egressMode: "cilium" }); + const cnpCall = clients.calls.find((c) => c.kind === "CiliumNetworkPolicy"); + expect(cnpCall).toBeDefined(); + const npCalls = clients.calls.filter((c) => c.kind === "NetworkPolicy"); + expect(npCalls).toHaveLength(1); + expect((npCalls[0].body as { metadata: { name: string } }).metadata.name).toBe("paperclip-deny-all"); + }); + + it("applies serviceAccountAnnotations to the ServiceAccount", async () => { + const clients = makeMockClients(); + await ensureTenant(clients as never, { + ...baseInput, + serviceAccountAnnotations: { "eks.amazonaws.com/role-arn": "arn:aws:iam::123:role/paperclip" }, + }); + const saCall = clients.calls.find((c) => c.kind === "ServiceAccount"); + const sa = saCall!.body as { metadata: { annotations: Record } }; + expect(sa.metadata.annotations["eks.amazonaws.com/role-arn"]).toBe("arn:aws:iam::123:role/paperclip"); + }); + + it("does not recreate a namespace that already exists", async () => { + const clients = makeMockClients(); + clients.core.readNamespace.mockResolvedValue({ body: { metadata: { name: baseInput.namespace } } }); + await ensureTenant(clients as never, baseInput); + expect(clients.core.createNamespace).not.toHaveBeenCalled(); + }); + + it("reconciles existing managed resources with the latest desired manifests", async () => { + const clients = makeMockClients(); + const existing = { metadata: { resourceVersion: "rv-1" } }; + clients.core.readNamespace.mockResolvedValue({ metadata: { name: baseInput.namespace } }); + clients.core.readNamespacedServiceAccount.mockResolvedValue(existing); + clients.rbac.readNamespacedRole.mockResolvedValue(existing); + clients.rbac.readNamespacedRoleBinding.mockResolvedValue(existing); + clients.core.readNamespacedResourceQuota.mockResolvedValue(existing); + clients.core.readNamespacedLimitRange.mockResolvedValue(existing); + clients.networking.readNamespacedNetworkPolicy.mockResolvedValue(existing); + + await ensureTenant(clients as never, { + ...baseInput, + serviceAccountAnnotations: { "eks.amazonaws.com/role-arn": "arn:aws:iam::123:role/paperclip" }, + resourceQuota: { ...baseInput.resourceQuota, pods: "25" }, + }); + + expect(clients.core.replaceNamespacedServiceAccount).toHaveBeenCalledWith( + expect.objectContaining({ + body: expect.objectContaining({ + metadata: expect.objectContaining({ + annotations: { "eks.amazonaws.com/role-arn": "arn:aws:iam::123:role/paperclip" }, + resourceVersion: "rv-1", + }), + }), + }), + ); + expect(clients.core.replaceNamespacedResourceQuota).toHaveBeenCalledWith( + expect.objectContaining({ + body: expect.objectContaining({ + metadata: expect.objectContaining({ resourceVersion: "rv-1" }), + spec: expect.objectContaining({ hard: expect.objectContaining({ pods: "25" }) }), + }), + }), + ); + expect(clients.networking.replaceNamespacedNetworkPolicy).toHaveBeenCalled(); + }); + + it("removes stale standard egress NetworkPolicy when cilium mode is selected", async () => { + const clients = makeMockClients(); + await ensureTenant(clients as never, { ...baseInput, egressMode: "cilium" }); + expect(clients.networking.deleteNamespacedNetworkPolicy).toHaveBeenCalledWith({ + namespace: baseInput.namespace, + name: "paperclip-egress-allow", + }); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/types.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/types.test.ts new file mode 100644 index 00000000..c6f8ceb5 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/types.test.ts @@ -0,0 +1,39 @@ +import { describe, it, expect } from "vitest"; +import { kubernetesProviderConfigSchema, parseKubernetesProviderConfig } from "../../src/types.js"; + +describe("kubernetesProviderConfigSchema", () => { + it("accepts inCluster=true with no kubeconfig", () => { + const parsed = parseKubernetesProviderConfig({ inCluster: true }); + expect(parsed.inCluster).toBe(true); + expect(parsed.namespacePrefix).toBe("paperclip-"); + expect(parsed.imageAllowList).toEqual([]); + expect(parsed.egressMode).toBe("standard"); + expect(parsed.jobTtlSecondsAfterFinished).toBe(900); + }); + + it("accepts inline kubeconfig", () => { + const parsed = parseKubernetesProviderConfig({ + inCluster: false, + kubeconfig: "apiVersion: v1\nkind: Config\n", + }); + expect(parsed.kubeconfig).toContain("apiVersion"); + }); + + it("rejects when neither inCluster nor any kubeconfig source is set", () => { + expect(() => parseKubernetesProviderConfig({ inCluster: false })).toThrow( + /requires one of `inCluster` or `kubeconfig`/, + ); + }); + + it("rejects invalid companySlug", () => { + expect(() => + parseKubernetesProviderConfig({ inCluster: true, companySlug: "INVALID UPPER" }), + ).toThrow(); + }); + + it("rejects egressAllowCidrs entries that are not valid CIDR", () => { + expect(() => + parseKubernetesProviderConfig({ inCluster: true, egressAllowCidrs: ["not-a-cidr"] }), + ).toThrow(/CIDR/i); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/test/unit/utils.test.ts b/packages/plugins/sandbox-providers/kubernetes/test/unit/utils.test.ts new file mode 100644 index 00000000..8ead8684 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/test/unit/utils.test.ts @@ -0,0 +1,42 @@ +import { describe, it, expect } from "vitest"; +import { deriveCompanySlug, deriveNamespaceName, newRunUlidDns, paperclipLabels } from "../../src/utils.js"; + +describe("deriveCompanySlug", () => { + it("lowercases and replaces non-alphanumerics", () => { + expect(deriveCompanySlug("Acme Co!")).toBe("acme-co"); + }); + + it("truncates to 32 chars and strips trailing dashes", () => { + expect(deriveCompanySlug("A".repeat(50))).toBe("a".repeat(32)); + expect(deriveCompanySlug("ab---")).toBe("ab"); + }); + + it("falls back to 'company' on empty/zero-letter input", () => { + expect(deriveCompanySlug("!!!")).toBe("company"); + expect(deriveCompanySlug("")).toBe("company"); + }); +}); + +describe("deriveNamespaceName", () => { + it("concatenates prefix and slug", () => { + expect(deriveNamespaceName("paperclip-", "acme-co")).toBe("paperclip-acme-co"); + }); +}); + +describe("newRunUlidDns", () => { + it("produces a DNS-safe 26-char lowercase id", () => { + const id = newRunUlidDns(); + expect(id).toMatch(/^[a-z0-9]{26}$/); + }); +}); + +describe("paperclipLabels", () => { + it("returns canonical label map", () => { + const labels = paperclipLabels({ runId: "r1", agentId: "a1", companyId: "c1", adapterType: "claude_local" }); + expect(labels["paperclip.io/run-id"]).toBe("r1"); + expect(labels["paperclip.io/agent-id"]).toBe("a1"); + expect(labels["paperclip.io/company-id"]).toBe("c1"); + expect(labels["paperclip.io/adapter"]).toBe("claude_local"); + expect(labels["paperclip.io/managed-by"]).toBe("paperclip-k8s-plugin"); + }); +}); diff --git a/packages/plugins/sandbox-providers/kubernetes/tsconfig.json b/packages/plugins/sandbox-providers/kubernetes/tsconfig.json new file mode 100644 index 00000000..000e3293 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/tsconfig.json @@ -0,0 +1,11 @@ +{ + "extends": "../../../../tsconfig.json", + "compilerOptions": { + "outDir": "dist", + "rootDir": "src", + "lib": ["ES2023"], + "types": ["node"] + }, + "include": ["src"], + "exclude": ["src/**/*.test.ts"] +} diff --git a/packages/plugins/sandbox-providers/kubernetes/vitest.config.ts b/packages/plugins/sandbox-providers/kubernetes/vitest.config.ts new file mode 100644 index 00000000..dc447fb7 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/vitest.config.ts @@ -0,0 +1,12 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + include: [ + "test/unit/**/*.test.ts", + ...(process.env.RUN_K8S_INTEGRATION_TESTS === "1" ? ["test/integration/**/*.test.ts"] : []), + ], + testTimeout: process.env.RUN_K8S_INTEGRATION_TESTS === "1" ? 120_000 : 5_000, + environment: "node", + }, +}); diff --git a/scripts/release-package-manifest.json b/scripts/release-package-manifest.json index 954c258e..761065e6 100644 --- a/scripts/release-package-manifest.json +++ b/scripts/release-package-manifest.json @@ -94,6 +94,11 @@ "name": "@paperclipai/plugin-daytona", "publishFromCi": true }, + { + "dir": "packages/plugins/sandbox-providers/kubernetes", + "name": "@paperclipai/plugin-kubernetes", + "publishFromCi": false + }, { "dir": "packages/plugins/sandbox-providers/exe-dev", "name": "@paperclipai/plugin-exe-dev",