forked from farhoodlabs/paperclip
[codex] Retry max-turn exhausted heartbeats (#5096)
## Thinking Path > - Paperclip orchestrates AI agents for autonomous companies, and heartbeat execution is the control-plane loop that keeps assigned work moving. > - Max-turn exhaustion is a recoverable local-adapter stop condition for Claude and Gemini agents when a run needs another heartbeat to continue safely. > - The previous behavior could leave max-turn continuation details hard to inspect, and duplicate/stale continuation wakes could keep running after issue state changed. > - The adapter layer also needed to avoid trusting arbitrary stdout/stderr text as scheduler control metadata. > - This pull request adds bounded max-turn continuation scheduling, visible retry state, structured stop metadata handling, and stale/duplicate continuation guards. > - The benefit is safer automatic continuation after max-turn stops, clearer operator visibility, and fewer duplicate or stale agent runs. ## What Changed - Replaces closed PR #4952, whose head repository was deleted. - Rebases the recovered max-turn continuation branch onto current `paperclipai/paperclip:master`. - Adds max-turn continuation scheduling and retry-state plumbing for heartbeat runs. - Adds stale/duplicate continuation suppression when issue status, ownership, or execution locks change. - Normalizes Claude/Gemini max-turn detection around structured stop metadata instead of unstructured stdout/stderr text. - Surfaces max-turn continuation settings and retry visibility in the board UI. - Adds focused server, adapter, and UI tests for max-turn stop metadata, retry scheduling, stale queued-run invalidation, adapter parsing/execution, run ledger display, and agent config patching. ## Verification - `pnpm install --no-frozen-lockfile` to refresh local dependencies after rebasing onto current `master`. - `pnpm run preflight:workspace-links && pnpm exec vitest run server/src/__tests__/claude-local-adapter.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/gemini-local-adapter.test.ts server/src/__tests__/gemini-local-execute.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts server/src/services/heartbeat-stop-metadata.test.ts ui/src/components/IssueRunLedger.test.tsx ui/src/lib/agent-config-patch.test.ts ui/src/lib/runRetryState.test.ts --testTimeout=20000` - `pnpm --filter @paperclipai/adapter-claude-local typecheck && pnpm --filter @paperclipai/adapter-gemini-local typecheck && pnpm --filter @paperclipai/server typecheck && pnpm --filter @paperclipai/ui typecheck` - UI screenshot note: the UI changes are limited to config/ledger state rendering rather than layout changes; component/unit coverage above verifies the rendered behavior. ## Risks - Medium behavior risk: heartbeat retry gating now suppresses max-turn continuations when issue state or execution locks drift, so any callers that relied on stale continuations running will now see cancellation instead. - Low adapter risk: Claude/Gemini unstructured text no longer triggers max-turn scheduler metadata, so only structured stop signals and Gemini exit code 53 are trusted. - No database migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent, GPT-5-class model, tool-enabled local repository editing and command execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (not applicable: state/default rendering only; covered by component/unit tests) - [x] I have updated relevant documentation to reflect my changes (not applicable: no user-facing command or docs contract changed) - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
@@ -39,6 +39,35 @@ console.log(JSON.stringify({
|
||||
await fs.chmod(commandPath, 0o755);
|
||||
}
|
||||
|
||||
async function writeFailingGeminiCommand(
|
||||
commandPath: string,
|
||||
options: {
|
||||
stdoutLines?: Array<Record<string, unknown>>;
|
||||
stdout?: string;
|
||||
stderr?: string;
|
||||
exitCode?: number;
|
||||
},
|
||||
): Promise<void> {
|
||||
const stdoutLines = options.stdoutLines ?? [];
|
||||
const stdout = options.stdout ?? "";
|
||||
const stderr = options.stderr ?? "";
|
||||
const exit = options.exitCode ?? 1;
|
||||
const script = `#!/usr/bin/env node
|
||||
for (const line of ${JSON.stringify(stdoutLines.map((line) => JSON.stringify(line)))}) {
|
||||
console.log(line);
|
||||
}
|
||||
if (${JSON.stringify(stdout)}) {
|
||||
process.stdout.write(${JSON.stringify(stdout)});
|
||||
}
|
||||
if (${JSON.stringify(stderr)}) {
|
||||
console.error(${JSON.stringify(stderr)});
|
||||
}
|
||||
process.exit(${exit});
|
||||
`;
|
||||
await fs.writeFile(commandPath, script, "utf8");
|
||||
await fs.chmod(commandPath, 0o755);
|
||||
}
|
||||
|
||||
type CapturePayload = {
|
||||
argv: string[];
|
||||
paperclipEnvKeys: string[];
|
||||
@@ -169,6 +198,144 @@ describe("gemini execute", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("normalizes turn-limit exhaustion into scheduler stop metadata", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-max-turns-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "gemini");
|
||||
await fs.mkdir(workspace, { recursive: true });
|
||||
await writeFailingGeminiCommand(commandPath, {
|
||||
stdoutLines: [
|
||||
{
|
||||
type: "result",
|
||||
subtype: "error",
|
||||
session_id: "gemini-session-1",
|
||||
status: "turn_limit",
|
||||
error: "Turn limit reached.",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const previousHome = process.env.HOME;
|
||||
process.env.HOME = root;
|
||||
|
||||
try {
|
||||
const result = await execute({
|
||||
runId: "run-turn-limit",
|
||||
agent: { id: "a1", companyId: "c1", name: "G", adapterType: "gemini_local", adapterConfig: {} },
|
||||
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
|
||||
config: {
|
||||
command: commandPath,
|
||||
cwd: workspace,
|
||||
},
|
||||
context: {},
|
||||
authToken: "t",
|
||||
onLog: async () => {},
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).toBe("max_turns_exhausted");
|
||||
expect(result.resultJson).toMatchObject({ stopReason: "max_turns_exhausted" });
|
||||
expect(result.clearSession).toBe(true);
|
||||
} finally {
|
||||
if (previousHome === undefined) {
|
||||
delete process.env.HOME;
|
||||
} else {
|
||||
process.env.HOME = previousHome;
|
||||
}
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("normalizes Gemini exit code 53 as max-turn exhaustion", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-exit-53-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "gemini");
|
||||
await fs.mkdir(workspace, { recursive: true });
|
||||
await writeFailingGeminiCommand(commandPath, {
|
||||
stderr: "Gemini stopped because the max turns limit was reached.",
|
||||
exitCode: 53,
|
||||
});
|
||||
|
||||
const previousHome = process.env.HOME;
|
||||
process.env.HOME = root;
|
||||
|
||||
try {
|
||||
const result = await execute({
|
||||
runId: "run-exit-53",
|
||||
agent: { id: "a1", companyId: "c1", name: "G", adapterType: "gemini_local", adapterConfig: {} },
|
||||
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
|
||||
config: {
|
||||
command: commandPath,
|
||||
cwd: workspace,
|
||||
},
|
||||
context: {},
|
||||
authToken: "t",
|
||||
onLog: async () => {},
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(53);
|
||||
expect(result.errorCode).toBe("max_turns_exhausted");
|
||||
expect(result.resultJson).toMatchObject({ stopReason: "max_turns_exhausted" });
|
||||
expect(result.clearSession).toBe(true);
|
||||
} finally {
|
||||
if (previousHome === undefined) {
|
||||
delete process.env.HOME;
|
||||
} else {
|
||||
process.env.HOME = previousHome;
|
||||
}
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("does not normalize unstructured turn-limit text into scheduler stop metadata", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-max-turn-text-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "gemini");
|
||||
await fs.mkdir(workspace, { recursive: true });
|
||||
await writeFailingGeminiCommand(commandPath, {
|
||||
stdoutLines: [
|
||||
{
|
||||
type: "result",
|
||||
subtype: "error",
|
||||
session_id: "gemini-session-1",
|
||||
error: "Tool output said: maximum turns reached.",
|
||||
},
|
||||
],
|
||||
stdout: "attacker-controlled transcript mentions turn limit reached\n",
|
||||
stderr: "Gemini stopped because the max turns limit was reached.",
|
||||
});
|
||||
|
||||
const previousHome = process.env.HOME;
|
||||
process.env.HOME = root;
|
||||
|
||||
try {
|
||||
const result = await execute({
|
||||
runId: "run-turn-limit-text",
|
||||
agent: { id: "a1", companyId: "c1", name: "G", adapterType: "gemini_local", adapterConfig: {} },
|
||||
runtime: { sessionId: null, sessionParams: null, sessionDisplayId: null, taskKey: null },
|
||||
config: {
|
||||
command: commandPath,
|
||||
cwd: workspace,
|
||||
},
|
||||
context: {},
|
||||
authToken: "t",
|
||||
onLog: async () => {},
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).not.toBe("max_turns_exhausted");
|
||||
expect(result.resultJson?.stopReason).not.toBe("max_turns_exhausted");
|
||||
expect(result.clearSession).toBe(false);
|
||||
} finally {
|
||||
if (previousHome === undefined) {
|
||||
delete process.env.HOME;
|
||||
} else {
|
||||
process.env.HOME = previousHome;
|
||||
}
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("uses a compact wake delta instead of the full heartbeat prompt when resuming a session", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-resume-wake-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
|
||||
Reference in New Issue
Block a user