1 Commits

Author SHA1 Message Date
Flea Flicker 607c57edab docs(execution-semantics): §9.1/§10/§11 stalled-at-dispatch + adapter-staleness gaps (GRO-2055)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-02 02:44:27 +00:00
3 changed files with 17 additions and 173 deletions
+16 -3
View File
@@ -376,9 +376,10 @@ Example:
Recovery rule:
- if the latest issue-linked run failed/timed out/cancelled and no live execution path remains, Paperclip queues one automatic assignment recovery wake
- if the issue has **no prior issue-linked run at all** — it is assigned and `todo`, no run was ever dispatched, no wake remains queued or running, and no recovery action is open — and its age exceeds the dispatch timeout (default 5 min), Paperclip treats it as **stalled-at-dispatch** and queues one automatic assignment recovery wake. Stalled-at-dispatch recovery does **not** require a prior failed, timed-out, or cancelled run; a never-dispatched assignment is a recoverable stall, not intentional rest.
- if that recovery wake also finishes and the issue is still stranded, Paperclip moves the issue to `blocked` and opens or updates an explicit recovery action when a bounded owner/action is known; the visible comment is evidence, not the recovery path by itself
This is a dispatch recovery, not a continuation recovery.
This is a dispatch recovery, not a continuation recovery. It covers both the post-crash stranded-run case and the zero-prior-run case where dispatch never produced a run.
### 9.2 Stranded assigned `in_progress`
@@ -410,11 +411,11 @@ On startup and on the periodic recovery loop, Paperclip now does five things in
1. reap orphaned `running` runs
2. resume persisted `queued` runs
3. reconcile stranded assigned work
3. reconcile stranded assigned work, including assigned `todo`/`in_progress` issues that have **never produced a linked run**; stalled-at-dispatch detection does not require a prior linked run
4. scan silent active runs, revalidate their source issues, and either fold source-resolved watchdogs or create/update explicit watchdog recovery actions
5. reconcile productivity reviews
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. It also covers the never-dispatched case: an assigned `todo` whose dispatch never started a run, has no queued wake, and has exceeded the dispatch timeout is reconciled as stalled-at-dispatch even though no prior run exists. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
## 11. Silent Active-Run Watchdog
@@ -441,6 +442,18 @@ Operators should prefer `snooze` for known time-bounded quiet periods. `continue
The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot.
### Adapter heartbeat staleness (pre-run)
The silent active-run watchdog above covers a run that is `running` but has stopped producing output. It does **not** cover an agent adapter that is wedged *before* any run is linked to an assigned issue. A wedged adapter can report `status: running` while its `lastHeartbeatAt` stops advancing, so dispatch triggers (assignment, @-mention, blocker-resolved wakes) fire without ever starting a run. `status: running` is therefore not, by itself, evidence of liveness — `lastHeartbeatAt` advancement is.
For every agent adapter assigned to a non-terminal issue, if `lastHeartbeatAt` has not advanced beyond a configured staleness threshold (default 15 min), Paperclip MUST, independent of whether any run is linked to the issue:
- open an explicit recovery action on the stalled issue that names the wedged adapter, the heartbeat-staleness evidence (last `lastHeartbeatAt`, staleness duration), the recovery owner, and the next action
- alert/escalate to the assignee's manager
- surface the stall visibly in activity and UI so operators can distinguish a wedged adapter from healthy idle work
This extends the watchdog contract from run-output silence to adapter-level silence that predates any linked run. Bounds mirror the active-run watchdog: at most one open adapter-staleness recovery action per adapter per staleness window, and the action folds through the normal explicit-recovery lifecycle once `lastHeartbeatAt` resumes advancing (the adapter self-recovered) or the issue otherwise reaches a valid live/waiting/terminal path.
### Source-aware watchdog folding
Active-run watchdog work is source-aware. Before the watchdog creates, refreshes, escalates, or blocks on reviewer work, it must re-read the linked source issue and decide whether the watchdog signal is still about productive source work or only about stale run/process bookkeeping.
@@ -1,167 +0,0 @@
import express from "express";
import request from "supertest";
import { beforeEach, describe, expect, it, vi } from "vitest";
const mockList = vi.hoisted(() => vi.fn());
const mockIssueService = vi.hoisted(() => ({
list: mockList,
getById: vi.fn(),
getByIdentifier: vi.fn(async () => null),
getComment: vi.fn(),
getCommentCursor: vi.fn(async () => ({
totalComments: 0,
latestCommentId: null,
latestCommentAt: null,
})),
getRelationSummaries: vi.fn(),
update: vi.fn(),
getAncestors: vi.fn(async () => []),
listWakeableBlockedDependents: vi.fn(async () => []),
getWakeableParentAfterChildCompletion: vi.fn(async () => null),
findMentionedAgents: vi.fn(async () => []),
}));
vi.mock("../services/index.js", async () => {
const actual = await vi.importActual<typeof import("../services/index.js")>(
"../services/index.js",
);
return {
...actual,
companyService: () => ({
getById: vi.fn(async () => ({ id: "company-1", attachmentMaxBytes: 10 * 1024 * 1024 })),
}),
accessService: () => ({
canUser: vi.fn(),
hasPermission: vi.fn(),
}),
agentService: () => ({
getById: vi.fn(),
}),
documentAnnotationService: () => ({ remapOpenThreadsForDocument: async () => [] }),
documentService: () => ({
getIssueDocumentPayload: vi.fn(async () => ({})),
}),
executionWorkspaceService: () => ({
getById: vi.fn(),
}),
feedbackService: () => ({}),
goalService: () => ({
getById: vi.fn(),
getDefaultCompanyGoal: vi.fn(),
}),
heartbeatService: () => ({
wakeup: vi.fn(async () => undefined),
reportRunActivity: vi.fn(async () => undefined),
}),
instanceSettingsService: () => ({
get: vi.fn(),
listCompanyIds: vi.fn(),
}),
issueApprovalService: () => ({}),
issueReferenceService: () => ({
deleteDocumentSource: async () => undefined,
diffIssueReferenceSummary: () => ({
addedReferencedIssues: [],
removedReferencedIssues: [],
currentReferencedIssues: [],
}),
emptySummary: () => ({ outbound: [], inbound: [] }),
listIssueReferenceSummary: async () => ({ outbound: [], inbound: [] }),
syncComment: async () => undefined,
syncDocument: async () => undefined,
syncIssue: async () => undefined,
}),
issueRecoveryActionService: () => ({
getActiveForIssue: vi.fn(async () => null),
listActiveForIssues: vi.fn(async () => new Map()),
}),
issueThreadInteractionService: () => ({
listForIssue: vi.fn(async () => []),
expireRequestConfirmationsSupersededByComment: vi.fn(async () => []),
expireStaleRequestConfirmationsForIssueDocument: vi.fn(async () => []),
}),
issueService: () => mockIssueService,
projectService: () => ({
getById: vi.fn(),
listByIds: vi.fn(async () => []),
}),
routineService: () => ({
syncRunStatusForIssue: vi.fn(async () => undefined),
}),
workProductService: () => ({
listForIssue: vi.fn(async () => []),
}),
};
});
async function createApp() {
const [{ issueRoutes }, { errorHandler }] = await Promise.all([
vi.importActual<typeof import("../routes/issues.js")>("../routes/issues.js"),
vi.importActual<typeof import("../middleware/index.js")>("../middleware/index.js"),
]);
const app = express();
app.use(express.json());
app.use((req, _res, next) => {
(req as any).actor = {
type: "board",
userId: "local-board",
companyIds: ["company-1"],
memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }],
source: "local_implicit",
isInstanceAdmin: false,
};
next();
});
app.use("/api", issueRoutes({} as any, {} as any));
app.use(errorHandler);
return app;
}
describe("GET /companies/:companyId/issues includeBlockedBy default", () => {
beforeEach(() => {
vi.resetModules();
vi.doUnmock("../routes/issues.js");
vi.doUnmock("../routes/authz.js");
vi.doUnmock("../middleware/index.js");
vi.clearAllMocks();
mockList.mockResolvedValue([]);
});
it("defaults includeBlockedBy to true so list responses are consistent with GET /api/issues/:id", async () => {
const res = await request(await createApp()).get("/api/companies/company-1/issues");
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockList).toHaveBeenCalledTimes(1);
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
expect(callArgs).toMatchObject({ includeBlockedBy: true });
});
it("defaults includeBlockedBy to true when the status filter is blocked (GRO-2096 regression guard)", async () => {
const res = await request(await createApp())
.get("/api/companies/company-1/issues")
.query({ status: "blocked" });
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockList).toHaveBeenCalledTimes(1);
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
expect(callArgs).toMatchObject({ status: "blocked", includeBlockedBy: true });
});
it("opts out of includeBlockedBy when the caller passes ?includeBlockedBy=false", async () => {
const res = await request(await createApp())
.get("/api/companies/company-1/issues")
.query({ includeBlockedBy: "false" });
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockList).toHaveBeenCalledTimes(1);
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
expect(callArgs).toMatchObject({ includeBlockedBy: false });
});
it("opts out of includeBlockedBy when the caller passes ?includeBlockedBy=0", async () => {
const res = await request(await createApp())
.get("/api/companies/company-1/issues")
.query({ includeBlockedBy: "0" });
expect(res.status, JSON.stringify(res.body)).toBe(200);
expect(mockList).toHaveBeenCalledTimes(1);
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
expect(callArgs).toMatchObject({ includeBlockedBy: false });
});
});
+1 -3
View File
@@ -1944,9 +1944,7 @@ export function issueRoutes(
req.query.excludeRoutineExecutions === "true" || req.query.excludeRoutineExecutions === "1",
includePluginOperations:
req.query.includePluginOperations === "true" || req.query.includePluginOperations === "1",
// Default to including blockedBy so list responses are consistent with GET /api/issues/:id.
// Opt out with ?includeBlockedBy=false (or 0) for perf-sensitive callers that don't need the graph.
includeBlockedBy: req.query.includeBlockedBy !== "false" && req.query.includeBlockedBy !== "0",
includeBlockedBy: req.query.includeBlockedBy === "true" || req.query.includeBlockedBy === "1",
includeBlockedInboxAttention:
req.query.includeBlockedInboxAttention === "true" || req.query.includeBlockedInboxAttention === "1",
q: req.query.q as string | undefined,