forked from farhoodlabs/paperclip
[codex] Harden heartbeat scheduling and runtime controls (#4223)
## Thinking Path > - Paperclip orchestrates AI agents through issue checkout, heartbeat runs, routines, and auditable control-plane state > - The runtime path has to recover from lost local processes, transient adapter failures, blocked dependencies, and routine coalescing without stranding work > - The existing branch carried several reliability fixes across heartbeat scheduling, issue runtime controls, routine dispatch, and operator-facing run state > - These changes belong together because they share backend contracts, migrations, and runtime status semantics > - This pull request groups the control-plane/runtime slice so it can merge independently from board UI polish and adapter sandbox work > - The benefit is safer heartbeat recovery, clearer runtime controls, and more predictable recurring execution behavior ## What Changed - Adds bounded heartbeat retry scheduling, scheduled retry state, and Codex transient failure recovery handling. - Tightens heartbeat process recovery, blocker wake behavior, issue comment wake handling, routine dispatch coalescing, and activity/dashboard bounds. - Adds runtime-control MCP tools and Paperclip skill docs for issue workspace runtime management. - Adds migrations `0061_lively_thor_girl.sql` and `0062_routine_run_dispatch_fingerprint.sql`. - Surfaces retry state in run ledger/agent UI and keeps related shared types synchronized. ## Verification - `pnpm exec vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/routines-service.test.ts` - `pnpm exec vitest run src/tools.test.ts` from `packages/mcp-server` ## Risks - Medium risk: this touches heartbeat recovery and routine dispatch, which are central execution paths. - Migration order matters if split branches land out of order: merge this PR before branches that assume the new runtime/routine fields. - Runtime retry behavior should be watched in CI and in local operator smoke tests because it changes how transient failures are resumed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5-based coding agent runtime, shell/git tool use enabled. Exact hosted model build and context window are not exposed in this Paperclip heartbeat environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
@@ -747,6 +747,11 @@ Terminal states: `done`, `cancelled`
|
||||
| GET | `/api/issues/:issueId/approvals` | List approvals linked to issue |
|
||||
| POST | `/api/issues/:issueId/approvals` | Link approval to issue |
|
||||
| DELETE | `/api/issues/:issueId/approvals/:approvalId` | Unlink approval from issue |
|
||||
| GET | `/api/issues/:issueId/heartbeat-context` | Compact issue context including `currentExecutionWorkspace` when one is linked |
|
||||
| GET | `/api/execution-workspaces/:workspaceId` | Execution workspace detail including runtime services and service URLs |
|
||||
| POST | `/api/execution-workspaces/:workspaceId/runtime-services/start` | Start configured workspace services |
|
||||
| POST | `/api/execution-workspaces/:workspaceId/runtime-services/restart` | Restart configured workspace services |
|
||||
| POST | `/api/execution-workspaces/:workspaceId/runtime-services/stop` | Stop workspace runtime services |
|
||||
|
||||
### Companies, Projects, Goals
|
||||
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Issue Workspace Runtime Controls
|
||||
|
||||
Use this reference when an issue has an isolated execution workspace and you need to inspect or run that workspace's services, especially for QA/browser verification.
|
||||
|
||||
## Discover the Workspace
|
||||
|
||||
Start from the issue, not from memory:
|
||||
|
||||
```sh
|
||||
curl -sS -H "Authorization: Bearer $PAPERCLIP_API_KEY" \
|
||||
"$PAPERCLIP_API_URL/api/issues/$PAPERCLIP_TASK_ID/heartbeat-context"
|
||||
```
|
||||
|
||||
Read `currentExecutionWorkspace`:
|
||||
|
||||
- `id` — execution workspace id for control endpoints
|
||||
- `cwd` / `branchName` — local checkout context
|
||||
- `status` / `closedAt` — whether the workspace is usable
|
||||
- `runtimeServices[]` — current services, including `serviceName`, `status`, `healthStatus`, `url`, `port`, and `runtimeServiceId`
|
||||
|
||||
If `currentExecutionWorkspace` is `null`, the issue does not currently have a realized execution workspace. For child/follow-up work, create the child with `parentId` or use `inheritExecutionWorkspaceFromIssueId` so Paperclip preserves workspace continuity.
|
||||
|
||||
## Control Services
|
||||
|
||||
Prefer Paperclip-managed runtime service controls over manual `pnpm dev &` or ad-hoc background processes. These endpoints keep service state, URLs, logs, and ownership visible to other agents and the board.
|
||||
|
||||
```sh
|
||||
# Start all configured services; waits for configured readiness checks.
|
||||
curl -sS -X POST \
|
||||
-H "Authorization: Bearer $PAPERCLIP_API_KEY" \
|
||||
-H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
|
||||
-H "Content-Type: application/json" \
|
||||
"$PAPERCLIP_API_URL/api/execution-workspaces/<workspace-id>/runtime-services/start" \
|
||||
-d '{}'
|
||||
|
||||
# Restart all configured services.
|
||||
curl -sS -X POST \
|
||||
-H "Authorization: Bearer $PAPERCLIP_API_KEY" \
|
||||
-H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
|
||||
-H "Content-Type: application/json" \
|
||||
"$PAPERCLIP_API_URL/api/execution-workspaces/<workspace-id>/runtime-services/restart" \
|
||||
-d '{}'
|
||||
|
||||
# Stop all running services.
|
||||
curl -sS -X POST \
|
||||
-H "Authorization: Bearer $PAPERCLIP_API_KEY" \
|
||||
-H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
|
||||
-H "Content-Type: application/json" \
|
||||
"$PAPERCLIP_API_URL/api/execution-workspaces/<workspace-id>/runtime-services/stop" \
|
||||
-d '{}'
|
||||
```
|
||||
|
||||
To target a configured service, pass one of:
|
||||
|
||||
```json
|
||||
{ "workspaceCommandId": "web" }
|
||||
{ "runtimeServiceId": "<runtime-service-id>" }
|
||||
{ "serviceIndex": 0 }
|
||||
```
|
||||
|
||||
The response includes an updated `workspace.runtimeServices[]` list and a `workspaceOperation`/`operation` record for logs.
|
||||
|
||||
## Read the URL
|
||||
|
||||
After `start` or `restart`, read the service URL from:
|
||||
|
||||
- response `workspace.runtimeServices[].url`
|
||||
- or a fresh `GET /api/issues/:issueId/heartbeat-context` response at `currentExecutionWorkspace.runtimeServices[].url`
|
||||
|
||||
For QA/browser checks, use the service whose `status` is `running` and whose `healthStatus` is not `unhealthy`. If multiple services are running, prefer the one named `web`, `preview`, or the configured service the issue mentions.
|
||||
|
||||
## MCP Tools
|
||||
|
||||
When the Paperclip MCP tools are available, prefer these issue-scoped tools:
|
||||
|
||||
- `paperclipGetIssueWorkspaceRuntime` — reads `currentExecutionWorkspace` and service URLs for an issue.
|
||||
- `paperclipControlIssueWorkspaceServices` — starts, stops, or restarts the current issue workspace services.
|
||||
- `paperclipWaitForIssueWorkspaceService` — waits until a selected service is running and returns its URL when exposed.
|
||||
|
||||
These tools resolve the issue's workspace id for you, so QA agents do not need to know the lower-level execution workspace endpoint first.
|
||||
@@ -0,0 +1,141 @@
|
||||
# Paperclip Workflow Playbooks
|
||||
|
||||
Reference material for niche workflows that are pointed to from `SKILL.md`. Load only when the task matches.
|
||||
|
||||
---
|
||||
|
||||
## Project Setup (CEO/Manager)
|
||||
|
||||
When asked to set up a new project with workspace config (local folder and/or GitHub repo):
|
||||
|
||||
1. `POST /api/companies/{companyId}/projects` with project fields.
|
||||
2. Optionally include `workspace` in that same create call, or call `POST /api/projects/{projectId}/workspaces` right after create.
|
||||
|
||||
Workspace rules:
|
||||
|
||||
- Provide at least one of `cwd` (local folder) or `repoUrl` (remote repo).
|
||||
- For repo-only setup, omit `cwd` and provide `repoUrl`.
|
||||
- Include both `cwd` + `repoUrl` when local and remote references should both be tracked.
|
||||
|
||||
---
|
||||
|
||||
## OpenClaw Invite (CEO)
|
||||
|
||||
Use this when asked to invite a new OpenClaw employee.
|
||||
|
||||
1. Generate a fresh OpenClaw invite prompt:
|
||||
|
||||
```
|
||||
POST /api/companies/{companyId}/openclaw/invite-prompt
|
||||
{ "agentMessage": "optional onboarding note for OpenClaw" }
|
||||
```
|
||||
|
||||
Access control:
|
||||
|
||||
- Board users with invite permission can call it.
|
||||
- Agent callers: only the company CEO agent can call it.
|
||||
|
||||
2. Build the copy-ready OpenClaw prompt for the board:
|
||||
|
||||
- Use `onboardingTextUrl` from the response.
|
||||
- Ask the board to paste that prompt into OpenClaw.
|
||||
- If the issue includes an OpenClaw URL (for example `ws://127.0.0.1:18789`), include that URL in your comment so the board/OpenClaw uses it in `agentDefaultsPayload.url`.
|
||||
|
||||
3. Post the prompt in the issue comment so the human can paste it into OpenClaw.
|
||||
|
||||
4. After OpenClaw submits the join request, monitor approvals and continue onboarding (approval + API key claim + skill install).
|
||||
|
||||
---
|
||||
|
||||
## Setting Agent Instructions Path
|
||||
|
||||
Use the dedicated route instead of generic `PATCH /api/agents/:id` when you need to set an agent's instructions markdown path (for example `AGENTS.md`).
|
||||
|
||||
```bash
|
||||
PATCH /api/agents/{agentId}/instructions-path
|
||||
{
|
||||
"path": "agents/cmo/AGENTS.md"
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Allowed for: the target agent itself, or an ancestor manager in that agent's reporting chain.
|
||||
- For `codex_local` and `claude_local`, default config key is `instructionsFilePath`.
|
||||
- Relative paths are resolved against the target agent's `adapterConfig.cwd`; absolute paths are accepted as-is.
|
||||
- To clear the path, send `{ "path": null }`.
|
||||
- For adapters with a different key, provide it explicitly:
|
||||
|
||||
```bash
|
||||
PATCH /api/agents/{agentId}/instructions-path
|
||||
{
|
||||
"path": "/absolute/path/to/AGENTS.md",
|
||||
"adapterConfigKey": "yourAdapterSpecificPathField"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Company Import / Export
|
||||
|
||||
Use the company-scoped routes when a CEO agent needs to inspect or move package content.
|
||||
|
||||
- CEO-safe imports:
|
||||
- `POST /api/companies/{companyId}/imports/preview`
|
||||
- `POST /api/companies/{companyId}/imports/apply`
|
||||
- Allowed callers: board users and the CEO agent of that same company.
|
||||
- Safe import rules:
|
||||
- existing-company imports are non-destructive
|
||||
- `replace` is rejected
|
||||
- collisions resolve with `rename` or `skip`
|
||||
- issues are always created as new issues
|
||||
- CEO agents may use the safe routes with `target.mode = "new_company"` to create a new company directly. Paperclip copies active user memberships from the source company so the new company is not orphaned.
|
||||
|
||||
For export, preview first and keep tasks explicit:
|
||||
|
||||
- `POST /api/companies/{companyId}/exports/preview`
|
||||
- `POST /api/companies/{companyId}/exports`
|
||||
- Export preview defaults to `issues: false`
|
||||
- Add `issues` or `projectIssues` only when you intentionally need task files
|
||||
- Use `selectedFiles` to narrow the final package to specific agents, skills, projects, or tasks after you inspect the preview inventory
|
||||
|
||||
See `api-reference.md` for full schema examples.
|
||||
|
||||
---
|
||||
|
||||
## Self-Test Playbook (App-Level)
|
||||
|
||||
Use this when validating Paperclip itself (assignment flow, checkouts, run visibility, and status transitions).
|
||||
|
||||
1. Create a throwaway issue assigned to a known local agent (`claudecoder` or `codexcoder`):
|
||||
|
||||
```bash
|
||||
npx paperclipai issue create \
|
||||
--company-id "$PAPERCLIP_COMPANY_ID" \
|
||||
--title "Self-test: assignment/watch flow" \
|
||||
--description "Temporary validation issue" \
|
||||
--status todo \
|
||||
--assignee-agent-id "$PAPERCLIP_AGENT_ID"
|
||||
```
|
||||
|
||||
2. Trigger and watch a heartbeat for that assignee:
|
||||
|
||||
```bash
|
||||
npx paperclipai heartbeat run --agent-id "$PAPERCLIP_AGENT_ID"
|
||||
```
|
||||
|
||||
3. Verify the issue transitions (`todo -> in_progress -> done` or `blocked`) and that comments are posted:
|
||||
|
||||
```bash
|
||||
npx paperclipai issue get <issue-id-or-identifier>
|
||||
```
|
||||
|
||||
4. Reassignment test (optional): move the same issue between `claudecoder` and `codexcoder` and confirm wake/run behavior:
|
||||
|
||||
```bash
|
||||
npx paperclipai issue update <issue-id> --assignee-agent-id <other-agent-id> --status todo
|
||||
```
|
||||
|
||||
5. Cleanup: mark temporary issues done/cancelled with a clear note.
|
||||
|
||||
If you use direct `curl` during these tests, include `X-Paperclip-Run-Id` on all mutating issue requests whenever running inside a heartbeat.
|
||||
Reference in New Issue
Block a user