fix: reattach to orphaned K8s Jobs across Paperclip restarts (FAR-124)
When the Paperclip pod restarts mid-run, the in-process setInterval keepalive dies, `updatedAt` goes stale, and the server's orphan reaper fails the run with the (misleading) "child pid 1 is no longer running" message. Paperclip then dispatches a continuation run, whose execute() finds the previous run's K8s Job still happily running and deletes it as an "orphan" — throwing away work and producing the transcript/run cascade reported on FAR-124. Changes: - job-manifest: add `paperclip.io/task-id` and `paperclip.io/session-id` labels (sanitized via new `sanitizeLabelValue` helper) so a later execute() can identify an orphan as the continuation of the same logical unit of work. - execute: in the concurrency guard, when `reattachOrphanedJobs` is on (default) and an orphan matches agent + task + session + is not terminal, pick it as the reattach target; delete only the other orphans. Branch the build/create/waitForPod block so the reattach path skips manifest building, Secret creation, Job creation, and scheduling wait — it jumps straight to streaming logs and waiting for the existing pod's completion. - config-schema: expose `reattachOrphanedJobs` toggle (default true). - Tests: `sanitizeLabelValue`, `isReattachableOrphan`, new label presence/absence, config default. No server-side changes; the misleading reaper message and lack of a non-local retry path will be addressed in a follow-up upstream PR. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
@@ -202,6 +202,17 @@ function sanitizeForK8sName(value: string, maxLen = 16): string {
|
||||
return value.toLowerCase().replace(/[^a-z0-9-]/g, "").slice(0, maxLen);
|
||||
}
|
||||
|
||||
/**
|
||||
* Sanitize a string for use as a Kubernetes label value (RFC 1123 subset:
|
||||
* `[a-zA-Z0-9]([-_.a-zA-Z0-9]*[a-zA-Z0-9])?`, max 63 chars). Returns `null`
|
||||
* when no usable characters remain — the caller should omit the label.
|
||||
*/
|
||||
export function sanitizeLabelValue(value: string, maxLen = 63): string | null {
|
||||
const cleaned = value.replace(/[^a-zA-Z0-9._-]/g, "").slice(0, maxLen);
|
||||
const trimmed = cleaned.replace(/^[^a-zA-Z0-9]+/, "").replace(/[^a-zA-Z0-9]+$/, "");
|
||||
return trimmed.length > 0 ? trimmed : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a short deterministic hash suffix from the raw inputs to avoid
|
||||
* collisions when sanitized slugs happen to be identical.
|
||||
@@ -428,6 +439,15 @@ export function buildJobManifest(input: JobBuildInput): JobBuildResult {
|
||||
"paperclip.io/company-id": agent.companyId,
|
||||
"paperclip.io/adapter-type": "claude_k8s",
|
||||
};
|
||||
// Reattach-target labels: let a future execute() identify this Job as the
|
||||
// continuation of the same logical unit of work (same task + same resume
|
||||
// session) so it can attach to the running pod across a Paperclip restart
|
||||
// instead of deleting it and starting over (FAR-124).
|
||||
const taskIdRaw = asString(context.taskId, "") || asString(context.issueId, "");
|
||||
const taskLabel = taskIdRaw ? sanitizeLabelValue(taskIdRaw) : null;
|
||||
if (taskLabel) labels["paperclip.io/task-id"] = taskLabel;
|
||||
const sessionLabel = runtimeSessionId ? sanitizeLabelValue(runtimeSessionId) : null;
|
||||
if (sessionLabel) labels["paperclip.io/session-id"] = sessionLabel;
|
||||
for (const [key, value] of Object.entries(extraLabels)) {
|
||||
labels[key] = value;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user