forked from farhoodlabs/paperclip
[codex] Retry max-turn exhausted heartbeats (#5096)
## Thinking Path > - Paperclip orchestrates AI agents for autonomous companies, and heartbeat execution is the control-plane loop that keeps assigned work moving. > - Max-turn exhaustion is a recoverable local-adapter stop condition for Claude and Gemini agents when a run needs another heartbeat to continue safely. > - The previous behavior could leave max-turn continuation details hard to inspect, and duplicate/stale continuation wakes could keep running after issue state changed. > - The adapter layer also needed to avoid trusting arbitrary stdout/stderr text as scheduler control metadata. > - This pull request adds bounded max-turn continuation scheduling, visible retry state, structured stop metadata handling, and stale/duplicate continuation guards. > - The benefit is safer automatic continuation after max-turn stops, clearer operator visibility, and fewer duplicate or stale agent runs. ## What Changed - Replaces closed PR #4952, whose head repository was deleted. - Rebases the recovered max-turn continuation branch onto current `paperclipai/paperclip:master`. - Adds max-turn continuation scheduling and retry-state plumbing for heartbeat runs. - Adds stale/duplicate continuation suppression when issue status, ownership, or execution locks change. - Normalizes Claude/Gemini max-turn detection around structured stop metadata instead of unstructured stdout/stderr text. - Surfaces max-turn continuation settings and retry visibility in the board UI. - Adds focused server, adapter, and UI tests for max-turn stop metadata, retry scheduling, stale queued-run invalidation, adapter parsing/execution, run ledger display, and agent config patching. ## Verification - `pnpm install --no-frozen-lockfile` to refresh local dependencies after rebasing onto current `master`. - `pnpm run preflight:workspace-links && pnpm exec vitest run server/src/__tests__/claude-local-adapter.test.ts server/src/__tests__/claude-local-execute.test.ts server/src/__tests__/gemini-local-adapter.test.ts server/src/__tests__/gemini-local-execute.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts server/src/services/heartbeat-stop-metadata.test.ts ui/src/components/IssueRunLedger.test.tsx ui/src/lib/agent-config-patch.test.ts ui/src/lib/runRetryState.test.ts --testTimeout=20000` - `pnpm --filter @paperclipai/adapter-claude-local typecheck && pnpm --filter @paperclipai/adapter-gemini-local typecheck && pnpm --filter @paperclipai/server typecheck && pnpm --filter @paperclipai/ui typecheck` - UI screenshot note: the UI changes are limited to config/ledger state rendering rather than layout changes; component/unit coverage above verifies the rendered behavior. ## Risks - Medium behavior risk: heartbeat retry gating now suppresses max-turn continuations when issue state or execution locks drift, so any callers that relied on stale continuations running will now see cancellation instead. - Low adapter risk: Claude/Gemini unstructured text no longer triggers max-turn scheduler metadata, so only structured stop signals and Gemini exit code 53 are trusted. - No database migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent, GPT-5-class model, tool-enabled local repository editing and command execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (not applicable: state/default rendering only; covered by component/unit tests) - [x] I have updated relevant documentation to reflect my changes (not applicable: no user-facing command or docs contract changed) - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
@@ -1,15 +1,17 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
agentRuntimeState,
|
||||
agentWakeupRequests,
|
||||
budgetPolicies,
|
||||
companies,
|
||||
createDb,
|
||||
environmentLeases,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueRelations,
|
||||
issues,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
@@ -18,6 +20,8 @@ import {
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS,
|
||||
MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
heartbeatService,
|
||||
} from "../services/heartbeat.ts";
|
||||
|
||||
@@ -44,10 +48,12 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
afterEach(async () => {
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issues);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(budgetPolicies);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
@@ -124,6 +130,92 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
});
|
||||
}
|
||||
|
||||
async function seedMaxTurnFixture(input?: {
|
||||
companyId?: string;
|
||||
agentId?: string;
|
||||
issueId?: string;
|
||||
runId?: string;
|
||||
now?: Date;
|
||||
scheduledRetryAttempt?: number;
|
||||
runtimeConfig?: Record<string, unknown>;
|
||||
issueStatus?: string;
|
||||
}) {
|
||||
const companyId = input?.companyId ?? randomUUID();
|
||||
const agentId = input?.agentId ?? randomUUID();
|
||||
const issueId = input?.issueId ?? randomUUID();
|
||||
const runId = input?.runId ?? randomUUID();
|
||||
const now = input?.now ?? new Date("2026-04-20T12:00:00.000Z");
|
||||
const issuePrefix = `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`;
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "ClaudeCoder",
|
||||
role: "engineer",
|
||||
status: "active",
|
||||
adapterType: "claude_local",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: input?.runtimeConfig ?? {
|
||||
heartbeat: {
|
||||
wakeOnDemand: true,
|
||||
maxConcurrentRuns: 1,
|
||||
maxTurnContinuation: {
|
||||
enabled: true,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
},
|
||||
},
|
||||
},
|
||||
permissions: {},
|
||||
});
|
||||
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId,
|
||||
invocationSource: "assignment",
|
||||
triggerDetail: "system",
|
||||
status: "failed",
|
||||
error: "Maximum turns reached",
|
||||
errorCode: "adapter_failed",
|
||||
finishedAt: now,
|
||||
scheduledRetryAttempt: input?.scheduledRetryAttempt ?? 0,
|
||||
scheduledRetryReason: input?.scheduledRetryAttempt ? MAX_TURN_CONTINUATION_RETRY_REASON : null,
|
||||
resultJson: {
|
||||
stopReason: "max_turns_exhausted",
|
||||
},
|
||||
contextSnapshot: {
|
||||
issueId,
|
||||
wakeReason: "issue_assigned",
|
||||
},
|
||||
updatedAt: now,
|
||||
createdAt: now,
|
||||
});
|
||||
|
||||
await db.insert(issues).values({
|
||||
id: issueId,
|
||||
companyId,
|
||||
title: "Continue after max turns",
|
||||
status: input?.issueStatus ?? "in_progress",
|
||||
priority: "medium",
|
||||
assigneeAgentId: agentId,
|
||||
executionRunId: runId,
|
||||
executionAgentNameKey: "claudecoder",
|
||||
executionLockedAt: now,
|
||||
issueNumber: 1,
|
||||
identifier: `${issuePrefix}-1`,
|
||||
});
|
||||
|
||||
return { companyId, agentId, issueId, runId, now };
|
||||
}
|
||||
|
||||
it("schedules a retry with durable metadata and only promotes it when due", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
@@ -218,6 +310,416 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
expect(promotedRun?.status).toBe("queued");
|
||||
});
|
||||
|
||||
it("schedules max-turn continuations with distinct retry metadata", async () => {
|
||||
const { runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") return;
|
||||
expect(scheduled.attempt).toBe(1);
|
||||
expect(scheduled.dueAt.toISOString()).toBe(new Date(now.getTime() + 1_000).toISOString());
|
||||
|
||||
const retryRun = await db
|
||||
.select({
|
||||
retryOfRunId: heartbeatRuns.retryOfRunId,
|
||||
status: heartbeatRuns.status,
|
||||
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
|
||||
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, scheduled.run.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
expect(retryRun).toMatchObject({
|
||||
retryOfRunId: runId,
|
||||
status: "scheduled_retry",
|
||||
scheduledRetryAttempt: 1,
|
||||
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
});
|
||||
expect((retryRun?.contextSnapshot as Record<string, unknown> | null)?.wakeReason).toBe(
|
||||
MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
);
|
||||
expect((retryRun?.contextSnapshot as Record<string, unknown> | null)?.codexTransientFallbackMode ?? null).toBeNull();
|
||||
|
||||
const wakeupRequest = await db
|
||||
.select({ reason: agentWakeupRequests.reason, payload: agentWakeupRequests.payload })
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.id, retryRun?.wakeupRequestId ?? ""))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(wakeupRequest?.reason).toBe(MAX_TURN_CONTINUATION_WAKE_REASON);
|
||||
expect(wakeupRequest?.payload).toMatchObject({
|
||||
retryOfRunId: runId,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
scheduledRetryAttempt: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it("coalesces duplicate max-turn continuation schedules for the same source run and attempt", async () => {
|
||||
const { issueId, runId, now } = await seedMaxTurnFixture();
|
||||
const retryOptions = {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
};
|
||||
|
||||
const [first, second] = await Promise.all([
|
||||
heartbeat.scheduleBoundedRetry(runId, retryOptions),
|
||||
heartbeat.scheduleBoundedRetry(runId, retryOptions),
|
||||
]);
|
||||
|
||||
expect(first.outcome).toBe("scheduled");
|
||||
expect(second.outcome).toBe("scheduled");
|
||||
if (first.outcome !== "scheduled" || second.outcome !== "scheduled") return;
|
||||
|
||||
expect(new Set([first.run.id, second.run.id]).size).toBe(1);
|
||||
|
||||
const retryRuns = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.retryOfRunId, runId),
|
||||
eq(heartbeatRuns.scheduledRetryReason, MAX_TURN_CONTINUATION_RETRY_REASON),
|
||||
eq(heartbeatRuns.scheduledRetryAttempt, 1),
|
||||
),
|
||||
);
|
||||
expect(retryRuns).toHaveLength(1);
|
||||
|
||||
const wakeups = await db
|
||||
.select({
|
||||
id: agentWakeupRequests.id,
|
||||
coalescedCount: agentWakeupRequests.coalescedCount,
|
||||
idempotencyKey: agentWakeupRequests.idempotencyKey,
|
||||
})
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.reason, MAX_TURN_CONTINUATION_WAKE_REASON));
|
||||
expect(wakeups).toHaveLength(1);
|
||||
expect(wakeups[0]).toMatchObject({
|
||||
id: retryRuns[0]?.wakeupRequestId,
|
||||
coalescedCount: 1,
|
||||
});
|
||||
expect(wakeups[0]?.idempotencyKey).toContain(`:${issueId}:${runId}:1`);
|
||||
|
||||
const issue = await db
|
||||
.select({ executionRunId: issues.executionRunId })
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(issue?.executionRunId).toBe(retryRuns[0]?.id);
|
||||
});
|
||||
|
||||
it("does not promote a duplicate max-turn continuation that does not own the issue lock", async () => {
|
||||
const { companyId, agentId, issueId, runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") return;
|
||||
|
||||
const duplicateWakeupId = randomUUID();
|
||||
const duplicateRunId = randomUUID();
|
||||
await db.insert(agentWakeupRequests).values({
|
||||
id: duplicateWakeupId,
|
||||
companyId,
|
||||
agentId,
|
||||
source: "automation",
|
||||
triggerDetail: "system",
|
||||
reason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
payload: {
|
||||
issueId,
|
||||
retryOfRunId: runId,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
scheduledRetryAttempt: 1,
|
||||
},
|
||||
status: "queued",
|
||||
requestedByActorType: "system",
|
||||
});
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: duplicateRunId,
|
||||
companyId,
|
||||
agentId,
|
||||
invocationSource: "automation",
|
||||
triggerDetail: "system",
|
||||
status: "scheduled_retry",
|
||||
wakeupRequestId: duplicateWakeupId,
|
||||
retryOfRunId: runId,
|
||||
scheduledRetryAt: scheduled.dueAt,
|
||||
scheduledRetryAttempt: 1,
|
||||
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
contextSnapshot: {
|
||||
issueId,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
},
|
||||
});
|
||||
await db
|
||||
.update(agentWakeupRequests)
|
||||
.set({ runId: duplicateRunId })
|
||||
.where(eq(agentWakeupRequests.id, duplicateWakeupId));
|
||||
|
||||
const promotion = await heartbeat.promoteDueScheduledRetries(scheduled.dueAt);
|
||||
expect(promotion).toEqual({ promoted: 1, runIds: [scheduled.run.id] });
|
||||
|
||||
const duplicate = await db
|
||||
.select({
|
||||
status: heartbeatRuns.status,
|
||||
errorCode: heartbeatRuns.errorCode,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, duplicateRunId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(duplicate).toEqual({
|
||||
status: "cancelled",
|
||||
errorCode: "issue_execution_lock_changed",
|
||||
});
|
||||
|
||||
const duplicateWakeup = await db
|
||||
.select({ status: agentWakeupRequests.status })
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.id, duplicateWakeupId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(duplicateWakeup?.status).toBe("cancelled");
|
||||
});
|
||||
|
||||
it.each(["blocked", "todo", "backlog"] as const)(
|
||||
"does not schedule a max-turn continuation when the issue is already %s",
|
||||
async (issueStatus) => {
|
||||
const { issueId, runId, now } = await seedMaxTurnFixture({ issueStatus });
|
||||
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
|
||||
expect(scheduled).toMatchObject({
|
||||
outcome: "not_scheduled",
|
||||
errorCode: "issue_not_in_progress",
|
||||
issueId,
|
||||
});
|
||||
|
||||
const retryRuns = await db
|
||||
.select({ count: sql<number>`count(*)::int` })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, runId))
|
||||
.then((rows) => rows[0]?.count ?? 0);
|
||||
expect(retryRuns).toBe(0);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["blocked", "todo", "backlog"] as const)(
|
||||
"cancels a due max-turn continuation when the issue moves to %s before retry promotion",
|
||||
async (issueStatus) => {
|
||||
const { issueId, runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") return;
|
||||
|
||||
await db.update(issues).set({
|
||||
status: issueStatus,
|
||||
updatedAt: new Date(now.getTime() + 500),
|
||||
}).where(eq(issues.id, issueId));
|
||||
|
||||
const promotion = await heartbeat.promoteDueScheduledRetries(scheduled.dueAt);
|
||||
expect(promotion).toEqual({ promoted: 0, runIds: [] });
|
||||
|
||||
const retryRun = await db
|
||||
.select({
|
||||
status: heartbeatRuns.status,
|
||||
errorCode: heartbeatRuns.errorCode,
|
||||
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, scheduled.run.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(retryRun).toMatchObject({
|
||||
status: "cancelled",
|
||||
errorCode: "issue_not_in_progress",
|
||||
});
|
||||
|
||||
const wakeupRequest = await db
|
||||
.select({ status: agentWakeupRequests.status })
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.id, retryRun?.wakeupRequestId ?? ""))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(wakeupRequest?.status).toBe("cancelled");
|
||||
|
||||
const issue = await db
|
||||
.select({
|
||||
executionRunId: issues.executionRunId,
|
||||
executionAgentNameKey: issues.executionAgentNameKey,
|
||||
executionLockedAt: issues.executionLockedAt,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(issue).toEqual({
|
||||
executionRunId: null,
|
||||
executionAgentNameKey: null,
|
||||
executionLockedAt: null,
|
||||
});
|
||||
|
||||
const event = await db
|
||||
.select({
|
||||
message: heartbeatRunEvents.message,
|
||||
payload: heartbeatRunEvents.payload,
|
||||
})
|
||||
.from(heartbeatRunEvents)
|
||||
.where(eq(heartbeatRunEvents.runId, scheduled.run.id))
|
||||
.orderBy(sql`${heartbeatRunEvents.seq} desc`)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(event?.message).toContain("no longer in_progress");
|
||||
expect(event?.payload).toMatchObject({
|
||||
currentStatus: issueStatus,
|
||||
requiredStatus: "in_progress",
|
||||
scheduledRetryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("does not queue max-turn continuations after the configured cap", async () => {
|
||||
const { runId, now } = await seedMaxTurnFixture({ scheduledRetryAttempt: 2 });
|
||||
|
||||
const exhausted = await heartbeat.scheduleBoundedRetry(runId, {
|
||||
now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
|
||||
expect(exhausted).toEqual({
|
||||
outcome: "retry_exhausted",
|
||||
attempt: 3,
|
||||
maxAttempts: 2,
|
||||
});
|
||||
|
||||
const runCount = await db
|
||||
.select({ count: sql<number>`count(*)::int` })
|
||||
.from(heartbeatRuns)
|
||||
.then((rows) => rows[0]?.count ?? 0);
|
||||
expect(runCount).toBe(1);
|
||||
|
||||
const exhaustionEvent = await db
|
||||
.select({ message: heartbeatRunEvents.message, payload: heartbeatRunEvents.payload })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(eq(heartbeatRunEvents.runId, runId))
|
||||
.orderBy(sql`${heartbeatRunEvents.id} desc`)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(exhaustionEvent?.message).toContain("Bounded retry exhausted");
|
||||
expect(exhaustionEvent?.payload).toMatchObject({
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
maxAttempts: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it("suppresses max-turn continuation scheduling when budget or dependencies block the issue", async () => {
|
||||
const budgetBlocked = await seedMaxTurnFixture({ now: new Date("2026-04-20T16:00:00.000Z") });
|
||||
await db.insert(budgetPolicies).values({
|
||||
companyId: budgetBlocked.companyId,
|
||||
scopeType: "agent",
|
||||
scopeId: budgetBlocked.agentId,
|
||||
windowKind: "monthly",
|
||||
metric: "billed_cents",
|
||||
amount: 0,
|
||||
hardStopEnabled: true,
|
||||
isActive: true,
|
||||
});
|
||||
await db
|
||||
.update(agents)
|
||||
.set({ status: "paused", pauseReason: "budget" })
|
||||
.where(eq(agents.id, budgetBlocked.agentId));
|
||||
|
||||
const budgetResult = await heartbeat.scheduleBoundedRetry(budgetBlocked.runId, {
|
||||
now: budgetBlocked.now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
expect(budgetResult).toMatchObject({
|
||||
outcome: "not_scheduled",
|
||||
errorCode: "budget_blocked",
|
||||
issueId: budgetBlocked.issueId,
|
||||
});
|
||||
|
||||
await db.delete(budgetPolicies);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issues);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
|
||||
const dependencyBlocked = await seedMaxTurnFixture({ now: new Date("2026-04-20T17:00:00.000Z") });
|
||||
const blockerId = randomUUID();
|
||||
await db.insert(issues).values({
|
||||
id: blockerId,
|
||||
companyId: dependencyBlocked.companyId,
|
||||
title: "Blocker",
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
issueNumber: 2,
|
||||
identifier: `T${dependencyBlocked.companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}-2`,
|
||||
});
|
||||
await db.insert(issueRelations).values({
|
||||
companyId: dependencyBlocked.companyId,
|
||||
issueId: blockerId,
|
||||
relatedIssueId: dependencyBlocked.issueId,
|
||||
type: "blocks",
|
||||
});
|
||||
|
||||
const dependencyResult = await heartbeat.scheduleBoundedRetry(dependencyBlocked.runId, {
|
||||
now: dependencyBlocked.now,
|
||||
retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON,
|
||||
maxAttempts: 2,
|
||||
delayMs: 1_000,
|
||||
});
|
||||
expect(dependencyResult).toMatchObject({
|
||||
outcome: "not_scheduled",
|
||||
errorCode: "issue_dependencies_blocked",
|
||||
issueId: dependencyBlocked.issueId,
|
||||
});
|
||||
|
||||
const retryRuns = await db
|
||||
.select({ count: sql<number>`count(*)::int` })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, dependencyBlocked.runId))
|
||||
.then((rows) => rows[0]?.count ?? 0);
|
||||
expect(retryRuns).toBe(0);
|
||||
});
|
||||
|
||||
it("does not defer a new assignee behind the previous assignee's scheduled retry", async () => {
|
||||
const companyId = randomUUID();
|
||||
const oldAgentId = randomUUID();
|
||||
|
||||
Reference in New Issue
Block a user