diff --git a/OPERATIONS.md b/OPERATIONS.md new file mode 100644 index 0000000..44d7175 --- /dev/null +++ b/OPERATIONS.md @@ -0,0 +1,201 @@ +# Privileged Escalation — Pod Operations Runbook + +This document covers the pod-side setup required to run Privileged Escalation agents on the Paperclip pod. All agents run as child processes of the Paperclip server inside the `paperclip` namespace. + +## Prerequisites + +- Paperclip pod running in `paperclip` namespace (`kubectl -n paperclip`) +- Shared Claude credentials at `/paperclip/.claude/.credentials.json` +- GitHub App PEM keys mounted at `/paperclip/secrets/github-pems/` (via K8s Secret) +- Agent records created in the Paperclip DB with correct `adapterConfig` + +## Directory Layout (on pod) + +``` +/paperclip/ +├── .claude/.credentials.json # Shared Anthropic credentials (all agents symlink to this) +├── secrets/github-pems/ # K8s Secret mount — one PEM per GitHub App +│ ├── countess.pem +│ ├── nancy.pem +│ ├── addison.pem +│ ├── gandalf.pem +│ ├── regina.pem +│ ├── hugh.pem +│ └── samuel.pem +└── privilegedescalation/ + └── agents/ # This repo, cloned here + ├── ceo/ + │ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json + ├── cto/ + │ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json + ├── cmo/ + │ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json + ├── engineering/ + │ ├── gandalf/.claude/.credentials.json -> /paperclip/.claude/.credentials.json + │ ├── regina/.claude/.credentials.json -> /paperclip/.claude/.credentials.json + │ └── hugh/.claude/.credentials.json -> /paperclip/.claude/.credentials.json + └── marketing/ + └── samuel/.claude/.credentials.json -> /paperclip/.claude/.credentials.json +``` + +## Initial Setup + +### 1. Clone the repo + +```bash +kubectl exec -n paperclip deploy/paperclip -- bash -c ' + mkdir -p /paperclip/privilegedescalation + cd /paperclip/privilegedescalation + git clone https://github.com/privilegedescalation/agents.git +' +``` + +> **Note:** The `privilegedescalation` org is private. Clone requires a valid token: +> ```bash +> git clone https://x-access-token:@github.com/privilegedescalation/agents.git +> ``` +> Use a personal access token or a GitHub App installation token. The repo remote must include the token for future pulls (see Routine Maintenance). + +### 2. Create `.claude` directories and symlink credentials + +Each agent's HOME is set to its directory in this repo (e.g., `/paperclip/privilegedescalation/agents/ceo`). Claude Code expects credentials at `$HOME/.claude/.credentials.json`. All agents share the same Anthropic credentials, so we symlink. + +```bash +kubectl exec -n paperclip deploy/paperclip -- bash -c ' + AGENTS_DIR=/paperclip/privilegedescalation/agents + CRED_SOURCE=/paperclip/.claude/.credentials.json + + for agent_dir in \ + "$AGENTS_DIR/ceo" \ + "$AGENTS_DIR/cto" \ + "$AGENTS_DIR/cmo" \ + "$AGENTS_DIR/engineering/gandalf" \ + "$AGENTS_DIR/engineering/regina" \ + "$AGENTS_DIR/engineering/hugh" \ + "$AGENTS_DIR/marketing/samuel"; do + + mkdir -p "$agent_dir/.claude" + ln -sf "$CRED_SOURCE" "$agent_dir/.claude/.credentials.json" + done +' +``` + +### 3. Verify GitHub PEM keys + +PEM keys are mounted from a K8s Secret at `/paperclip/secrets/github-pems/`. Each agent's `adapterConfig.env` references its PEM via `GITHUB_PEM_PATH_`. Verify they exist: + +```bash +kubectl exec -n paperclip deploy/paperclip -- ls -la /paperclip/secrets/github-pems/ +``` + +To add a new PEM, update the K8s Secret (managed via sealed-secrets or SOPS) and the mount will auto-refresh. + +## Adding a New Agent + +1. **Create profile files** in this repo: `AGENTS.md`, `SOUL.md`, `HEARTBEAT.md`, `TOOLS.md`, `CONFIG.md` +2. **Create the DB record** via Paperclip API or direct SQL — include `adapterConfig` with: + - `cwd`: agent directory path (e.g., `/paperclip/privilegedescalation/agents/engineering/newagent`) + - `env.HOME`: same as `cwd` + - `env.GITHUB_APP_ID_`: the GitHub App ID + - `env.GITHUB_PEM_PATH_`: path to PEM (e.g., `/paperclip/secrets/github-pems/newagent.pem`) + - `instructionsFilePath`: path to `AGENTS.md` + - `model`: model ID (e.g., `claude-opus-4-6`) +3. **Create the GitHub App** via GitHub UI manifest flow, install on the `privilegedescalation` org +4. **Add the PEM** to the K8s Secret at `/paperclip/secrets/github-pems/` +5. **On the pod**, create the `.claude` symlink: + ```bash + kubectl exec -n paperclip deploy/paperclip -- bash -c ' + mkdir -p /paperclip/privilegedescalation/agents/engineering/newagent/.claude + ln -sf /paperclip/.claude/.credentials.json \ + /paperclip/privilegedescalation/agents/engineering/newagent/.claude/.credentials.json + ' + ``` +6. **Pull the repo** on the pod: + ```bash + kubectl exec -n paperclip deploy/paperclip -- bash -c ' + cd /paperclip/privilegedescalation/agents && git pull + ' + ``` +7. **Create DB records** for `company_memberships` and `principal_permission_grants` +8. **Update COMPANY.md** roster table +9. **Commit, push, pull** on pod + +## Routine Maintenance + +### Pulling repo updates + +The `privilegedescalation` org is private. The git remote must include a valid token. If `git pull` fails with auth errors, use a personal access token: + +```bash +GH_TOKEN=$(gh auth token) # or any valid PAT with repo access +kubectl exec -n paperclip deploy/paperclip -- bash -c " + cd /paperclip/privilegedescalation/agents + git -c url.\"https://${GH_TOKEN}@github.com/\".insteadOf=\"https://github.com/\" pull +" +``` + +Or regenerate using a GitHub App installation token: +```bash +kubectl exec -n paperclip deploy/paperclip -- bash -c ' + export GITHUB_APP_ID_COUNTESS=1234567 + export GITHUB_PEM_PATH_COUNTESS=/paperclip/secrets/github-pems/countess.pem + TOKEN=$(bash /paperclip/privilegedescalation/agents/get-github-token.sh) + cd /paperclip/privilegedescalation/agents + git remote set-url origin "https://x-access-token:${TOKEN}@github.com/privilegedescalation/agents.git" + git pull +' +``` + +### Verifying credential symlinks + +```bash +kubectl exec -n paperclip deploy/paperclip -- bash -c ' + find /paperclip/privilegedescalation/agents -maxdepth 4 -name ".credentials.json" -type l -exec ls -la {} \; +' +``` + +### Checking agent HOME isolation + +Each agent must have its own HOME to prevent cross-contamination of caches, config, and tool state. Verify: + +```bash +kubectl exec -n paperclip deploy/paperclip -- bash -c ' + psql "$DATABASE_URL" -c " + SELECT name, adapter_config->'\''env'\''->'\''HOME'\''->'\''value'\'' as home + FROM agents + WHERE company_id = '\''38ad87cc-54cd-41c6-93f5-1bc68be94349'\'' + ORDER BY name; + " +' +``` + +## Special: Regina (opencode_local) + +Regina uses the `opencode_local` adapter, not `claude_local`. Her prompt is stored as `promptTemplate` in the DB, not loaded from a file. See COMPANY.md "Known Issues" section for: + +- **Prompt restoration** after UI saves (which wipe `promptTemplate`) +- **Env/model restoration** after UI saves (which wipe `env` and `model`) + +## Troubleshooting + +### Agent says "Claude credentials not found" +The `.claude/.credentials.json` symlink is missing or broken. Re-create it: +```bash +mkdir -p /paperclip/privilegedescalation/agents//.claude +ln -sf /paperclip/.claude/.credentials.json /paperclip/privilegedescalation/agents//.claude/.credentials.json +``` + +### Agent says "PEM file not found" +The K8s Secret mount may not have refreshed, or the PEM name doesn't match `GITHUB_PEM_PATH_` in the agent's adapter config. Check: +```bash +ls -la /paperclip/secrets/github-pems/ +``` + +### Git pull auth failure +The privesc org is private — tokens expire. Use a PAT or regenerate an installation token (see Routine Maintenance above). + +### Agent can't reach API (`HTTP 000`) +Transient issue at heartbeat startup — the server may be briefly busy spawning the agent process. Agents self-recover by retrying. If persistent, verify `PAPERCLIP_API_URL` resolves correctly: +```bash +curl -s http://localhost:3100/api/health +```