forked from farhoodlabs/paperclip
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 27296ccbe7 |
@@ -376,10 +376,9 @@ Example:
|
||||
Recovery rule:
|
||||
|
||||
- if the latest issue-linked run failed/timed out/cancelled and no live execution path remains, Paperclip queues one automatic assignment recovery wake
|
||||
- if the issue has **no prior issue-linked run at all** — it is assigned and `todo`, no run was ever dispatched, no wake remains queued or running, and no recovery action is open — and its age exceeds the dispatch timeout (default 5 min), Paperclip treats it as **stalled-at-dispatch** and queues one automatic assignment recovery wake. Stalled-at-dispatch recovery does **not** require a prior failed, timed-out, or cancelled run; a never-dispatched assignment is a recoverable stall, not intentional rest.
|
||||
- if that recovery wake also finishes and the issue is still stranded, Paperclip moves the issue to `blocked` and opens or updates an explicit recovery action when a bounded owner/action is known; the visible comment is evidence, not the recovery path by itself
|
||||
|
||||
This is a dispatch recovery, not a continuation recovery. It covers both the post-crash stranded-run case and the zero-prior-run case where dispatch never produced a run.
|
||||
This is a dispatch recovery, not a continuation recovery.
|
||||
|
||||
### 9.2 Stranded assigned `in_progress`
|
||||
|
||||
@@ -411,11 +410,11 @@ On startup and on the periodic recovery loop, Paperclip now does five things in
|
||||
|
||||
1. reap orphaned `running` runs
|
||||
2. resume persisted `queued` runs
|
||||
3. reconcile stranded assigned work, including assigned `todo`/`in_progress` issues that have **never produced a linked run**; stalled-at-dispatch detection does not require a prior linked run
|
||||
3. reconcile stranded assigned work
|
||||
4. scan silent active runs, revalidate their source issues, and either fold source-resolved watchdogs or create/update explicit watchdog recovery actions
|
||||
5. reconcile productivity reviews
|
||||
|
||||
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. It also covers the never-dispatched case: an assigned `todo` whose dispatch never started a run, has no queued wake, and has exceeded the dispatch timeout is reconciled as stalled-at-dispatch even though no prior run exists. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
|
||||
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
|
||||
|
||||
## 11. Silent Active-Run Watchdog
|
||||
|
||||
@@ -442,18 +441,6 @@ Operators should prefer `snooze` for known time-bounded quiet periods. `continue
|
||||
|
||||
The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot.
|
||||
|
||||
### Adapter heartbeat staleness (pre-run)
|
||||
|
||||
The silent active-run watchdog above covers a run that is `running` but has stopped producing output. It does **not** cover an agent adapter that is wedged *before* any run is linked to an assigned issue. A wedged adapter can report `status: running` while its `lastHeartbeatAt` stops advancing, so dispatch triggers (assignment, @-mention, blocker-resolved wakes) fire without ever starting a run. `status: running` is therefore not, by itself, evidence of liveness — `lastHeartbeatAt` advancement is.
|
||||
|
||||
For every agent adapter assigned to a non-terminal issue, if `lastHeartbeatAt` has not advanced beyond a configured staleness threshold (default 15 min), Paperclip MUST, independent of whether any run is linked to the issue:
|
||||
|
||||
- open an explicit recovery action on the stalled issue that names the wedged adapter, the heartbeat-staleness evidence (last `lastHeartbeatAt`, staleness duration), the recovery owner, and the next action
|
||||
- alert/escalate to the assignee's manager
|
||||
- surface the stall visibly in activity and UI so operators can distinguish a wedged adapter from healthy idle work
|
||||
|
||||
This extends the watchdog contract from run-output silence to adapter-level silence that predates any linked run. Bounds mirror the active-run watchdog: at most one open adapter-staleness recovery action per adapter per staleness window, and the action folds through the normal explicit-recovery lifecycle once `lastHeartbeatAt` resumes advancing (the adapter self-recovered) or the issue otherwise reaches a valid live/waiting/terminal path.
|
||||
|
||||
### Source-aware watchdog folding
|
||||
|
||||
Active-run watchdog work is source-aware. Before the watchdog creates, refreshes, escalates, or blocks on reviewer work, it must re-read the linked source issue and decide whether the watchdog signal is still about productive source work or only about stale run/process bookkeeping.
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mockList = vi.hoisted(() => vi.fn());
|
||||
const mockIssueService = vi.hoisted(() => ({
|
||||
list: mockList,
|
||||
getById: vi.fn(),
|
||||
getByIdentifier: vi.fn(async () => null),
|
||||
getComment: vi.fn(),
|
||||
getCommentCursor: vi.fn(async () => ({
|
||||
totalComments: 0,
|
||||
latestCommentId: null,
|
||||
latestCommentAt: null,
|
||||
})),
|
||||
getRelationSummaries: vi.fn(),
|
||||
update: vi.fn(),
|
||||
getAncestors: vi.fn(async () => []),
|
||||
listWakeableBlockedDependents: vi.fn(async () => []),
|
||||
getWakeableParentAfterChildCompletion: vi.fn(async () => null),
|
||||
findMentionedAgents: vi.fn(async () => []),
|
||||
}));
|
||||
|
||||
vi.mock("../services/index.js", async () => {
|
||||
const actual = await vi.importActual<typeof import("../services/index.js")>(
|
||||
"../services/index.js",
|
||||
);
|
||||
return {
|
||||
...actual,
|
||||
companyService: () => ({
|
||||
getById: vi.fn(async () => ({ id: "company-1", attachmentMaxBytes: 10 * 1024 * 1024 })),
|
||||
}),
|
||||
accessService: () => ({
|
||||
canUser: vi.fn(),
|
||||
hasPermission: vi.fn(),
|
||||
}),
|
||||
agentService: () => ({
|
||||
getById: vi.fn(),
|
||||
}),
|
||||
documentAnnotationService: () => ({ remapOpenThreadsForDocument: async () => [] }),
|
||||
documentService: () => ({
|
||||
getIssueDocumentPayload: vi.fn(async () => ({})),
|
||||
}),
|
||||
executionWorkspaceService: () => ({
|
||||
getById: vi.fn(),
|
||||
}),
|
||||
feedbackService: () => ({}),
|
||||
goalService: () => ({
|
||||
getById: vi.fn(),
|
||||
getDefaultCompanyGoal: vi.fn(),
|
||||
}),
|
||||
heartbeatService: () => ({
|
||||
wakeup: vi.fn(async () => undefined),
|
||||
reportRunActivity: vi.fn(async () => undefined),
|
||||
}),
|
||||
instanceSettingsService: () => ({
|
||||
get: vi.fn(),
|
||||
listCompanyIds: vi.fn(),
|
||||
}),
|
||||
issueApprovalService: () => ({}),
|
||||
issueReferenceService: () => ({
|
||||
deleteDocumentSource: async () => undefined,
|
||||
diffIssueReferenceSummary: () => ({
|
||||
addedReferencedIssues: [],
|
||||
removedReferencedIssues: [],
|
||||
currentReferencedIssues: [],
|
||||
}),
|
||||
emptySummary: () => ({ outbound: [], inbound: [] }),
|
||||
listIssueReferenceSummary: async () => ({ outbound: [], inbound: [] }),
|
||||
syncComment: async () => undefined,
|
||||
syncDocument: async () => undefined,
|
||||
syncIssue: async () => undefined,
|
||||
}),
|
||||
issueRecoveryActionService: () => ({
|
||||
getActiveForIssue: vi.fn(async () => null),
|
||||
listActiveForIssues: vi.fn(async () => new Map()),
|
||||
}),
|
||||
issueThreadInteractionService: () => ({
|
||||
listForIssue: vi.fn(async () => []),
|
||||
expireRequestConfirmationsSupersededByComment: vi.fn(async () => []),
|
||||
expireStaleRequestConfirmationsForIssueDocument: vi.fn(async () => []),
|
||||
}),
|
||||
issueService: () => mockIssueService,
|
||||
projectService: () => ({
|
||||
getById: vi.fn(),
|
||||
listByIds: vi.fn(async () => []),
|
||||
}),
|
||||
routineService: () => ({
|
||||
syncRunStatusForIssue: vi.fn(async () => undefined),
|
||||
}),
|
||||
workProductService: () => ({
|
||||
listForIssue: vi.fn(async () => []),
|
||||
}),
|
||||
};
|
||||
});
|
||||
|
||||
async function createApp() {
|
||||
const [{ issueRoutes }, { errorHandler }] = await Promise.all([
|
||||
vi.importActual<typeof import("../routes/issues.js")>("../routes/issues.js"),
|
||||
vi.importActual<typeof import("../middleware/index.js")>("../middleware/index.js"),
|
||||
]);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = {
|
||||
type: "board",
|
||||
userId: "local-board",
|
||||
companyIds: ["company-1"],
|
||||
memberships: [{ companyId: "company-1", membershipRole: "owner", status: "active" }],
|
||||
source: "local_implicit",
|
||||
isInstanceAdmin: false,
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", issueRoutes({} as any, {} as any));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
describe("GET /companies/:companyId/issues includeBlockedBy default", () => {
|
||||
beforeEach(() => {
|
||||
vi.resetModules();
|
||||
vi.doUnmock("../routes/issues.js");
|
||||
vi.doUnmock("../routes/authz.js");
|
||||
vi.doUnmock("../middleware/index.js");
|
||||
vi.clearAllMocks();
|
||||
mockList.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
it("defaults includeBlockedBy to true so list responses are consistent with GET /api/issues/:id", async () => {
|
||||
const res = await request(await createApp()).get("/api/companies/company-1/issues");
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockList).toHaveBeenCalledTimes(1);
|
||||
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
|
||||
expect(callArgs).toMatchObject({ includeBlockedBy: true });
|
||||
});
|
||||
|
||||
it("defaults includeBlockedBy to true when the status filter is blocked (GRO-2096 regression guard)", async () => {
|
||||
const res = await request(await createApp())
|
||||
.get("/api/companies/company-1/issues")
|
||||
.query({ status: "blocked" });
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockList).toHaveBeenCalledTimes(1);
|
||||
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
|
||||
expect(callArgs).toMatchObject({ status: "blocked", includeBlockedBy: true });
|
||||
});
|
||||
|
||||
it("opts out of includeBlockedBy when the caller passes ?includeBlockedBy=false", async () => {
|
||||
const res = await request(await createApp())
|
||||
.get("/api/companies/company-1/issues")
|
||||
.query({ includeBlockedBy: "false" });
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockList).toHaveBeenCalledTimes(1);
|
||||
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
|
||||
expect(callArgs).toMatchObject({ includeBlockedBy: false });
|
||||
});
|
||||
|
||||
it("opts out of includeBlockedBy when the caller passes ?includeBlockedBy=0", async () => {
|
||||
const res = await request(await createApp())
|
||||
.get("/api/companies/company-1/issues")
|
||||
.query({ includeBlockedBy: "0" });
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockList).toHaveBeenCalledTimes(1);
|
||||
const callArgs = mockList.mock.calls[0]?.[1] ?? {};
|
||||
expect(callArgs).toMatchObject({ includeBlockedBy: false });
|
||||
});
|
||||
});
|
||||
@@ -1944,7 +1944,9 @@ export function issueRoutes(
|
||||
req.query.excludeRoutineExecutions === "true" || req.query.excludeRoutineExecutions === "1",
|
||||
includePluginOperations:
|
||||
req.query.includePluginOperations === "true" || req.query.includePluginOperations === "1",
|
||||
includeBlockedBy: req.query.includeBlockedBy === "true" || req.query.includeBlockedBy === "1",
|
||||
// Default to including blockedBy so list responses are consistent with GET /api/issues/:id.
|
||||
// Opt out with ?includeBlockedBy=false (or 0) for perf-sensitive callers that don't need the graph.
|
||||
includeBlockedBy: req.query.includeBlockedBy !== "false" && req.query.includeBlockedBy !== "0",
|
||||
includeBlockedInboxAttention:
|
||||
req.query.includeBlockedInboxAttention === "true" || req.query.includeBlockedInboxAttention === "1",
|
||||
q: req.query.q as string | undefined,
|
||||
|
||||
Reference in New Issue
Block a user