- Merge both one-shot log fallbacks into a single conditional block using a
cheap string-scan guard (`stdout.includes('"type":"result"')`) to avoid
calling parseClaudeStreamJson twice and prevent double readPodLogs calls
when the first fallback already ran.
- Extract error-message logic into `buildPartialRunError(exitCode, model, stdout)`
(exported for tests) so the `!parsed` branch is a one-liner and the logic
is independently testable.
- Export `isK8s404` for tests.
- Add execute.test.ts with 15 unit tests covering:
- isK8s404: v0.x response.statusCode, v1.0+ response.status, direct
statusCode, message-based detection, non-404 codes
- buildPartialRunError: exitCode=0 path, empty stdout, init-only output
(model surfaced), first non-system content line, null exitCode (-1),
multiple consecutive system events
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add a second log fallback: if the follow stream captured partial output (init
event present but no result event), attempt a one-shot readPodLogs before the
pod is cleaned up. Fast-exiting containers (bad model, missing API key, etc.)
can cause the follow stream to return only the init line before the connection
drops; the one-shot read is more reliable for already-terminated containers.
- Improve the `!parsed` error message: skip system/init events when searching
for the first content line, so the error reads "Claude started but did not
produce a result (model: MiniMax-M2.7) — check API credentials..." instead of
"Claude exited with code -1: {"type":"system","subtype":"init",...}".
Co-Authored-By: Paperclip <noreply@paperclip.ing>
- Add `isK8s404()` helper compatible with @kubernetes/client-node v0.x and v1.0+
(checks response.statusCode, response.status, err.statusCode, and message text)
- `waitForJobCompletion` now catches 404 and returns `{ jobGone: true }` instead
of throwing — prevents uncaught exceptions when the K8s Job is TTL-deleted or
externally removed while the adapter is polling for a terminal condition
- Keepalive job-liveness check now uses `isK8s404` (was checking `response.statusCode`
which is absent in the v1.0+ fetch-based client, silently breaking 404 detection)
- `jobGone` case in completion handler logs a diagnostic and falls through to stdout
parsing rather than returning an opaque 404 error to the user
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The adapter was calling onSpawn({ pid: -1 }) as a sentinel value for
K8s Jobs (which run out-of-process), then the server's orphan reaper
was checking isProcessAlive(-1) which always returns false, causing
legitimate runs to be reaped as 'process_lost'.
Using process.pid (the Paperclip server's own PID) is always alive
while the adapter runs in-process, preventing false reaping.
Fixes FAR-116.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5. Cap log stream reconnect attempts at 50 — prevents infinite
reconnect loops during sustained API partitions.
6. Fire keepalive refresh earlier — tick 1 + every 12 ticks (~3min)
instead of every 16 ticks (~4min), providing better safety margin
under the 5-minute reaper window.
7. Catch rejections from onLog inside keepalive — add .catch(() => {})
to prevent unhandledRejection on SSE backpressure.
8. Prevent sanitized-name collisions — extend slugs to 16 chars each,
add a 6-char SHA-256 hash suffix, shorten prefix to `ac-` to stay
well within the 63-char DNS label limit.
10. Fix config-hint parity for nodeSelector and labels — parse both
`key=value` multiline text and JSON objects, matching what the
textarea hint promises.
11. Large-prompt fallback via Secret — prompts >256 KiB are staged as a
K8s Secret and mounted as a volume instead of passed via env var,
protecting against the ~1 MiB PodSpec limit.
13. Track last-seen log timestamp on reconnect — anchor sinceSeconds at
the last received log line instead of stream start, fixing FAR-105
duplicative logs. Belt-and-braces: dedupe assistantTexts at the
parser boundary in parse.ts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
1. Inherit envFrom and env.valueFrom from self pod — secrets wired via
valueFrom.secretKeyRef or envFrom.secretRef are now forwarded to Job
pods, fixing credentials silently dropped for K8s-idiomatic secret
patterns (e.g. ANTHROPIC_API_KEY via Secret).
2. Distinguish 404 vs transient errors in keepalive — only mark the
keepalive as terminal on 404 (Job deleted). Transient 5xx/connection
errors are logged and retried on the next tick, preventing premature
reaper kills during API instability.
3. Fail closed on concurrency-guard read failure — a failing
listNamespacedJob now returns k8s_concurrency_guard_unreachable
instead of silently proceeding, protecting against zombie Jobs on
shared PVCs.
4. Bound the waitForJobCompletion re-check — pass a 60s timeout instead
of polling forever, preventing indefinite hangs when the K8s API is
degraded.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The previous fix (df856e6) made the keepalive timer call onSpawn every
~4 minutes to refresh the run's updatedAt in the DB, so the stale-run
reaper wouldn't kill live runs in multi-instance deployments. That was
correct for live jobs, but it was unconditional — if execute() stalled
after the pod terminated (slow K8s API call, hung log stream drain, or
a Job whose Complete condition lags pod termination), the keepalive
kept the run marked "alive" indefinitely even though the pod was gone.
That manifests as the opposite of the original bug: the UI shows jobs
as running when they have actually finished.
Two changes:
1. Verify the Job is still alive before the keepalive refreshes
updatedAt. If the Job has reached a terminal Complete/Failed
condition (or has been deleted / the API read fails), stop
refreshing. If execute() truly ends up stuck past that point, the
reaper will catch the run within the normal 5-minute staleness
window instead of never.
2. Clear the keepalive interval immediately once Promise.allSettled
resolves, rather than only in the finally block. Post-completion
work (exit-code fetch, log fallback read, job cleanup) must not be
able to emit another onSpawn refresh that keeps the run "alive".
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Two root causes behind the "plugin losing sync" issue:
1. After a server restart, the in-memory activeRunExecutions set is lost.
The K8s Job keeps running but the reaper marks the server-side run as
failed after 5 min (stale updatedAt). Next heartbeat fires a new run,
the adapter's concurrency guard blocks it because the old Job is still
alive, and this loops indefinitely.
Fix: the concurrency guard now compares each running Job's
paperclip.io/run-id label against the current runId. Jobs from a
previous (dead) run are cleaned up automatically so the new run
can proceed.
2. onLog (keepalive) does NOT update the run's updatedAt in the DB —
it only writes to the log store and publishes SSE events. In
multi-instance deployments, a reaper on instance B can mark a run
being executed on instance A as stale after 5 min of no DB updates.
Fix: the keepalive timer now calls onSpawn every ~4 min (16 ticks)
to refresh updatedAt, staying within the 5-min reaper threshold.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The k8s adapter never called ctx.onSpawn(), so the Paperclip server
had no processStartedAt timestamp for the run. The stale-run reaper
(reapOrphanedRuns) would then mark live k8s runs as failed/orphaned,
causing the UI to show no active runs and triggering duplicate run
attempts that hit the concurrency guard.
Uses pid=-1 as a sentinel since there is no local process — the
server's isProcessAlive check safely returns false for pid <= 0.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
When waitForJobCompletion threw and the job was still not terminal, we
were returning an error but still deleting the job in the finally block.
This left the UI holding an error while the job (still alive) would be
cleaned up by Kubernetes, causing the next heartbeat to find nothing and
think it was safe to retry — spawning a concurrent pod.
Now we set skipCleanup=true when returning the mismatch error, so the
job is retained and the heartbeat can still find and wait on it.
Also removes a duplicate empty-stdout fallback block.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When waitForJobCompletion threw a transient error (API disconnect, etc.),
the code fell through with jobTimedOut=true and returned a result even
though the job was still running. This caused the UI to think the run
was complete while the job kept running, resulting in concurrency errors.
Now when completion throws, we re-check the job's actual state. If still
not terminal, we return a k8s_job_state_mismatch error so the UI knows
the run is not done.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The adapter opened a single follow-stream to the K8s API for pod logs.
If that TCP connection silently dropped (API server hiccup, network
timeout, load-balancer idle cut), streamPodLogs returned early and no
more real Claude output reached the UI — only keepalive pings. The
pod kept running and producing logs (visible via kubectl), but the
adapter never reconnected.
Splits streamPodLogs into streamPodLogsOnce (single follow attempt) and
a reconnecting wrapper that retries with sinceSeconds until a shared
stop signal fires when waitForJobCompletion resolves. On reconnect,
requests logs from the original stream start time (+5s overlap) so no
output is lost; the UI deduplicates chunks.
Bumps version to 0.1.12.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The adapter had no mechanism to signal liveness while a K8s Job was
running. When Claude entered long thinking phases with no log output,
the Paperclip UI could lose sync and consider the run stuck even though
the pod was still actively working.
Adds a 15-second interval keepalive that sends status messages via
onLog during execution. The keepalive tracks time since last real log
output and reports it, keeping the connection alive. The timer is
cleaned up in the finally block to prevent leaks on any exit path.
Bumps version to 0.1.11.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Test User
Chris Farhood
Pawla Abdul