org/OPERATIONS.md

# Privileged Escalation — Pod Operations Runbook

This document covers the pod-side setup required to run Privileged Escalation agents on the Paperclip pod. All agents run as child processes of the Paperclip server inside the `paperclip` namespace.

## Prerequisites

- Paperclip pod running in `paperclip` namespace (`kubectl -n paperclip`)
- Shared Claude credentials at `/paperclip/.claude/.credentials.json`
- GitHub App PEM keys mounted at `/paperclip/secrets/github-pems/` (via K8s Secret)
- Agent records created in the Paperclip DB with correct `adapterConfig`

## Directory Layout (on pod)

```
/paperclip/
├── .claude/.credentials.json          # Shared Anthropic credentials (all agents symlink to this)
├── secrets/github-pems/               # K8s Secret mount — one PEM per GitHub App
│   ├── countess.pem
│   ├── nancy.pem
│   ├── addison.pem
│   ├── gandalf.pem
│   ├── regina.pem
│   ├── hugh.pem
│   └── samuel.pem
└── privilegedescalation/
    └── agents/                        # This repo, cloned here
        ├── ceo/
        │   └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
        ├── cto/
        │   └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
        ├── cmo/
        │   └── .claude/.credentials.json -> /paperclip/.claude/.credentials.json
        ├── engineering/
        │   ├── gandalf/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
        │   ├── regina/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
        │   └── hugh/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
        └── marketing/
            └── samuel/.claude/.credentials.json -> /paperclip/.claude/.credentials.json
```

## Initial Setup

### 1. Clone the repo

```bash
kubectl exec -n paperclip deploy/paperclip -- bash -c '
  mkdir -p /paperclip/privilegedescalation
  cd /paperclip/privilegedescalation
  git clone https://github.com/privilegedescalation/agents.git
'
```

> **Note:** The `privilegedescalation` org is private. Clone requires a valid token:
> ```bash
> git clone https://x-access-token:<TOKEN>@github.com/privilegedescalation/agents.git
> ```
> Use a personal access token or a GitHub App installation token. The repo remote must include the token for future pulls (see Routine Maintenance).

### 2. Create `.claude` directories and symlink credentials

Each agent's HOME is set to its directory in this repo (e.g., `/paperclip/privilegedescalation/agents/ceo`). Claude Code expects credentials at `$HOME/.claude/.credentials.json`. All agents share the same Anthropic credentials, so we symlink.

```bash
kubectl exec -n paperclip deploy/paperclip -- bash -c '
  AGENTS_DIR=/paperclip/privilegedescalation/agents
  CRED_SOURCE=/paperclip/.claude/.credentials.json

  for agent_dir in \
    "$AGENTS_DIR/ceo" \
    "$AGENTS_DIR/cto" \
    "$AGENTS_DIR/cmo" \
    "$AGENTS_DIR/engineering/gandalf" \
    "$AGENTS_DIR/engineering/regina" \
    "$AGENTS_DIR/engineering/hugh" \
    "$AGENTS_DIR/marketing/samuel"; do

    mkdir -p "$agent_dir/.claude"
    ln -sf "$CRED_SOURCE" "$agent_dir/.claude/.credentials.json"
  done
'
```

### 3. Verify GitHub PEM keys

PEM keys are mounted from a K8s Secret at `/paperclip/secrets/github-pems/`. Each agent's `adapterConfig.env` references its PEM via `GITHUB_PEM_PATH_<NAME>`. Verify they exist:

```bash
kubectl exec -n paperclip deploy/paperclip -- ls -la /paperclip/secrets/github-pems/
```

To add a new PEM, update the K8s Secret (managed via sealed-secrets or SOPS) and the mount will auto-refresh.

## Adding a New Agent

1. **Create profile files** in this repo: `AGENTS.md`, `SOUL.md`, `HEARTBEAT.md`, `TOOLS.md`, `CONFIG.md`
2. **Create the DB record** via Paperclip API or direct SQL — include `adapterConfig` with:
   - `cwd`: agent directory path (e.g., `/paperclip/privilegedescalation/agents/engineering/newagent`)
   - `env.HOME`: same as `cwd`
   - `env.GITHUB_APP_ID_<NAME>`: the GitHub App ID
   - `env.GITHUB_PEM_PATH_<NAME>`: path to PEM (e.g., `/paperclip/secrets/github-pems/newagent.pem`)
   - `instructionsFilePath`: path to `AGENTS.md`
   - `model`: model ID (e.g., `claude-opus-4-6`)
3. **Create the GitHub App** via GitHub UI manifest flow, install on the `privilegedescalation` org
4. **Add the PEM** to the K8s Secret at `/paperclip/secrets/github-pems/`
5. **On the pod**, create the `.claude` symlink:
   ```bash
   kubectl exec -n paperclip deploy/paperclip -- bash -c '
     mkdir -p /paperclip/privilegedescalation/agents/engineering/newagent/.claude
     ln -sf /paperclip/.claude/.credentials.json \
       /paperclip/privilegedescalation/agents/engineering/newagent/.claude/.credentials.json
   '
   ```
6. **Pull the repo** on the pod:
   ```bash
   kubectl exec -n paperclip deploy/paperclip -- bash -c '
     cd /paperclip/privilegedescalation/agents && git pull
   '
   ```
7. **Create DB records** for `company_memberships` and `principal_permission_grants`
8. **Update COMPANY.md** roster table
9. **Commit, push, pull** on pod

## Routine Maintenance

### Pulling repo updates

The `privilegedescalation` org is private. The git remote must include a valid token. If `git pull` fails with auth errors, use a personal access token:

```bash
GH_TOKEN=$(gh auth token)  # or any valid PAT with repo access
kubectl exec -n paperclip deploy/paperclip -- bash -c "
  cd /paperclip/privilegedescalation/agents
  git -c url.\"https://${GH_TOKEN}@github.com/\".insteadOf=\"https://github.com/\" pull
"
```

Or regenerate using a GitHub App installation token:
```bash
kubectl exec -n paperclip deploy/paperclip -- bash -c '
  export GITHUB_APP_ID_COUNTESS=1234567
  export GITHUB_PEM_PATH_COUNTESS=/paperclip/secrets/github-pems/countess.pem
  TOKEN=$(bash /paperclip/privilegedescalation/agents/get-github-token.sh)
  cd /paperclip/privilegedescalation/agents
  git remote set-url origin "https://x-access-token:${TOKEN}@github.com/privilegedescalation/agents.git"
  git pull
'
```

### Verifying credential symlinks

```bash
kubectl exec -n paperclip deploy/paperclip -- bash -c '
  find /paperclip/privilegedescalation/agents -maxdepth 4 -name ".credentials.json" -type l -exec ls -la {} \;
'
```

### Checking agent HOME isolation

Each agent must have its own HOME to prevent cross-contamination of caches, config, and tool state. Verify:

```bash
kubectl exec -n paperclip deploy/paperclip -- bash -c '
  psql "$DATABASE_URL" -c "
    SELECT name, adapter_config->'\''env'\''->'\''HOME'\''->'\''value'\'' as home
    FROM agents
    WHERE company_id = '\''38ad87cc-54cd-41c6-93f5-1bc68be94349'\''
    ORDER BY name;
  "
'
```

## Special: Regina (opencode_local)

Regina uses the `opencode_local` adapter, not `claude_local`. Her prompt is stored as `promptTemplate` in the DB, not loaded from a file. See COMPANY.md "Known Issues" section for:

- **Prompt restoration** after UI saves (which wipe `promptTemplate`)
- **Env/model restoration** after UI saves (which wipe `env` and `model`)

## Troubleshooting

### Agent says "Claude credentials not found"
The `.claude/.credentials.json` symlink is missing or broken. Re-create it:
```bash
mkdir -p /paperclip/privilegedescalation/agents/<path>/.claude
ln -sf /paperclip/.claude/.credentials.json /paperclip/privilegedescalation/agents/<path>/.claude/.credentials.json
```

### Agent says "PEM file not found"
The K8s Secret mount may not have refreshed, or the PEM name doesn't match `GITHUB_PEM_PATH_<NAME>` in the agent's adapter config. Check:
```bash
ls -la /paperclip/secrets/github-pems/
```

### Git pull auth failure
The privesc org is private — tokens expire. Use a PAT or regenerate an installation token (see Routine Maintenance above).

### Agent can't reach API (`HTTP 000`)
Transient issue at heartbeat startup — the server may be briefly busy spawning the agent process. Agents self-recover by retrying. If persistent, verify `PAPERCLIP_API_URL` resolves correctly:
```bash
curl -s http://localhost:3100/api/health
```