docs: add pod operations runbook (symlinks, credentials, scaffolding)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
+201
@@ -0,0 +1,201 @@
|
||||
# Privileged Escalation — Pod Operations Runbook
|
||||
|
||||
This document covers the pod-side setup required to run Privileged Escalation agents on the Paperclip pod. All agents run as child processes of the Paperclip server inside the `paperclip` namespace.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Paperclip pod running in `paperclip` namespace (`kubectl -n paperclip`)
|
||||
- Shared Claude credentials at `/paperclip/.claude/.credentials.json`
|
||||
- GitHub App PEM keys mounted at `/paperclip/secrets/github-pems/` (via K8s Secret)
|
||||
- Agent records created in the Paperclip DB with correct `adapterConfig`
|
||||
|
||||
## Directory Layout (on pod)
|
||||
|
||||
```
|
||||
/paperclip/
|
||||
├── .claude/.credentials.json # Shared Anthropic credentials (all agents symlink to this)
|
||||
├── secrets/github-pems/ # K8s Secret mount — one PEM per GitHub App
|
||||
│ ├── countess.pem
|
||||
│ ├── nancy.pem
|
||||
│ ├── addison.pem
|
||||
│ ├── gandalf.pem
|
||||
│ ├── regina.pem
|
||||
│ ├── hugh.pem
|
||||
│ └── samuel.pem
|
||||
└── privilegedescalation/
|
||||
└── agents/ # This repo, cloned here
|
||||
├── ceo/
|
||||
│ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
├── cto/
|
||||
│ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
├── cmo/
|
||||
│ └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
├── engineering/
|
||||
│ ├── gandalf/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
│ ├── regina/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
│ └── hugh/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
└── marketing/
|
||||
└── samuel/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
|
||||
```
|
||||
|
||||
## Initial Setup
|
||||
|
||||
### 1. Clone the repo
|
||||
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
mkdir -p /paperclip/privilegedescalation
|
||||
cd /paperclip/privilegedescalation
|
||||
git clone https://github.com/privilegedescalation/agents.git
|
||||
'
|
||||
```
|
||||
|
||||
> **Note:** The `privilegedescalation` org is private. Clone requires a valid token:
|
||||
> ```bash
|
||||
> git clone https://x-access-token:<TOKEN>@github.com/privilegedescalation/agents.git
|
||||
> ```
|
||||
> Use a personal access token or a GitHub App installation token. The repo remote must include the token for future pulls (see Routine Maintenance).
|
||||
|
||||
### 2. Create `.claude` directories and symlink credentials
|
||||
|
||||
Each agent's HOME is set to its directory in this repo (e.g., `/paperclip/privilegedescalation/agents/ceo`). Claude Code expects credentials at `$HOME/.claude/.credentials.json`. All agents share the same Anthropic credentials, so we symlink.
|
||||
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
AGENTS_DIR=/paperclip/privilegedescalation/agents
|
||||
CRED_SOURCE=/paperclip/.claude/.credentials.json
|
||||
|
||||
for agent_dir in \
|
||||
"$AGENTS_DIR/ceo" \
|
||||
"$AGENTS_DIR/cto" \
|
||||
"$AGENTS_DIR/cmo" \
|
||||
"$AGENTS_DIR/engineering/gandalf" \
|
||||
"$AGENTS_DIR/engineering/regina" \
|
||||
"$AGENTS_DIR/engineering/hugh" \
|
||||
"$AGENTS_DIR/marketing/samuel"; do
|
||||
|
||||
mkdir -p "$agent_dir/.claude"
|
||||
ln -sf "$CRED_SOURCE" "$agent_dir/.claude/.credentials.json"
|
||||
done
|
||||
'
|
||||
```
|
||||
|
||||
### 3. Verify GitHub PEM keys
|
||||
|
||||
PEM keys are mounted from a K8s Secret at `/paperclip/secrets/github-pems/`. Each agent's `adapterConfig.env` references its PEM via `GITHUB_PEM_PATH_<NAME>`. Verify they exist:
|
||||
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- ls -la /paperclip/secrets/github-pems/
|
||||
```
|
||||
|
||||
To add a new PEM, update the K8s Secret (managed via sealed-secrets or SOPS) and the mount will auto-refresh.
|
||||
|
||||
## Adding a New Agent
|
||||
|
||||
1. **Create profile files** in this repo: `AGENTS.md`, `SOUL.md`, `HEARTBEAT.md`, `TOOLS.md`, `CONFIG.md`
|
||||
2. **Create the DB record** via Paperclip API or direct SQL — include `adapterConfig` with:
|
||||
- `cwd`: agent directory path (e.g., `/paperclip/privilegedescalation/agents/engineering/newagent`)
|
||||
- `env.HOME`: same as `cwd`
|
||||
- `env.GITHUB_APP_ID_<NAME>`: the GitHub App ID
|
||||
- `env.GITHUB_PEM_PATH_<NAME>`: path to PEM (e.g., `/paperclip/secrets/github-pems/newagent.pem`)
|
||||
- `instructionsFilePath`: path to `AGENTS.md`
|
||||
- `model`: model ID (e.g., `claude-opus-4-6`)
|
||||
3. **Create the GitHub App** via GitHub UI manifest flow, install on the `privilegedescalation` org
|
||||
4. **Add the PEM** to the K8s Secret at `/paperclip/secrets/github-pems/`
|
||||
5. **On the pod**, create the `.claude` symlink:
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
mkdir -p /paperclip/privilegedescalation/agents/engineering/newagent/.claude
|
||||
ln -sf /paperclip/.claude/.credentials.json \
|
||||
/paperclip/privilegedescalation/agents/engineering/newagent/.claude/.credentials.json
|
||||
'
|
||||
```
|
||||
6. **Pull the repo** on the pod:
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
cd /paperclip/privilegedescalation/agents && git pull
|
||||
'
|
||||
```
|
||||
7. **Create DB records** for `company_memberships` and `principal_permission_grants`
|
||||
8. **Update COMPANY.md** roster table
|
||||
9. **Commit, push, pull** on pod
|
||||
|
||||
## Routine Maintenance
|
||||
|
||||
### Pulling repo updates
|
||||
|
||||
The `privilegedescalation` org is private. The git remote must include a valid token. If `git pull` fails with auth errors, use a personal access token:
|
||||
|
||||
```bash
|
||||
GH_TOKEN=$(gh auth token) # or any valid PAT with repo access
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c "
|
||||
cd /paperclip/privilegedescalation/agents
|
||||
git -c url.\"https://${GH_TOKEN}@github.com/\".insteadOf=\"https://github.com/\" pull
|
||||
"
|
||||
```
|
||||
|
||||
Or regenerate using a GitHub App installation token:
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
export GITHUB_APP_ID_COUNTESS=1234567
|
||||
export GITHUB_PEM_PATH_COUNTESS=/paperclip/secrets/github-pems/countess.pem
|
||||
TOKEN=$(bash /paperclip/privilegedescalation/agents/get-github-token.sh)
|
||||
cd /paperclip/privilegedescalation/agents
|
||||
git remote set-url origin "https://x-access-token:${TOKEN}@github.com/privilegedescalation/agents.git"
|
||||
git pull
|
||||
'
|
||||
```
|
||||
|
||||
### Verifying credential symlinks
|
||||
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
find /paperclip/privilegedescalation/agents -maxdepth 4 -name ".credentials.json" -type l -exec ls -la {} \;
|
||||
'
|
||||
```
|
||||
|
||||
### Checking agent HOME isolation
|
||||
|
||||
Each agent must have its own HOME to prevent cross-contamination of caches, config, and tool state. Verify:
|
||||
|
||||
```bash
|
||||
kubectl exec -n paperclip deploy/paperclip -- bash -c '
|
||||
psql "$DATABASE_URL" -c "
|
||||
SELECT name, adapter_config->'\''env'\''->'\''HOME'\''->'\''value'\'' as home
|
||||
FROM agents
|
||||
WHERE company_id = '\''38ad87cc-54cd-41c6-93f5-1bc68be94349'\''
|
||||
ORDER BY name;
|
||||
"
|
||||
'
|
||||
```
|
||||
|
||||
## Special: Regina (opencode_local)
|
||||
|
||||
Regina uses the `opencode_local` adapter, not `claude_local`. Her prompt is stored as `promptTemplate` in the DB, not loaded from a file. See COMPANY.md "Known Issues" section for:
|
||||
|
||||
- **Prompt restoration** after UI saves (which wipe `promptTemplate`)
|
||||
- **Env/model restoration** after UI saves (which wipe `env` and `model`)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Agent says "Claude credentials not found"
|
||||
The `.claude/.credentials.json` symlink is missing or broken. Re-create it:
|
||||
```bash
|
||||
mkdir -p /paperclip/privilegedescalation/agents/<path>/.claude
|
||||
ln -sf /paperclip/.claude/.credentials.json /paperclip/privilegedescalation/agents/<path>/.claude/.credentials.json
|
||||
```
|
||||
|
||||
### Agent says "PEM file not found"
|
||||
The K8s Secret mount may not have refreshed, or the PEM name doesn't match `GITHUB_PEM_PATH_<NAME>` in the agent's adapter config. Check:
|
||||
```bash
|
||||
ls -la /paperclip/secrets/github-pems/
|
||||
```
|
||||
|
||||
### Git pull auth failure
|
||||
The privesc org is private — tokens expire. Use a PAT or regenerate an installation token (see Routine Maintenance above).
|
||||
|
||||
### Agent can't reach API (`HTTP 000`)
|
||||
Transient issue at heartbeat startup — the server may be briefly busy spawning the agent process. Agents self-recover by retrying. If persistent, verify `PAPERCLIP_API_URL` resolves correctly:
|
||||
```bash
|
||||
curl -s http://localhost:3100/api/health
|
||||
```
|
||||
Reference in New Issue
Block a user