3 Commits

Author SHA1 Message Date
Flea Flicker 607c57edab docs(execution-semantics): §9.1/§10/§11 stalled-at-dispatch + adapter-staleness gaps (GRO-2055)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-06-02 02:44:27 +00:00
Chris Farhood 9f3f71a199 fork: add CLAUDE.md describing fork model and don'ts
Build: Dev / build (push) Failing after 9s
Build: Production / build (push) Successful in 3m17s
Build: Dev / update-infra (push) Has been skipped
Captures branch model (master/dev/local), the 3-file fork delta, upstream
sync procedure, and the post-reset rule against re-introducing fork code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-31 08:35:03 -04:00
Chris Farhood 1a1c57461f fork: production Dockerfile additions + Gitea registry build workflows
Build: Dev / update-infra (push) Successful in 1s
Build: Production / build (push) Successful in 4m6s
Build: Dev / build (push) Successful in 4m15s
Only fork divergence from upstream/master. Adds to the production stage:
  - kubectl, kubeseal (Kubernetes ops in deployed pods)
  - uv, uvx (Python tooling)
  - forgejo-cli (fj, fj-ex, fgj)
  - gitea tea CLI
  - mmx-cli
  - nano, vim

Workflows push to git.farh.net/farhoodlabs/paperclip{,-dev}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-31 08:11:45 -04:00
5 changed files with 206 additions and 6 deletions
+77
View File
@@ -0,0 +1,77 @@
name: "Build: Dev"
on:
push:
branches: [dev]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
image-tag: ${{ steps.tag.outputs.sha }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set image tag
id: tag
run: echo "sha=$(echo ${{ github.sha }} | cut -c1-7)" >> $GITHUB_OUTPUT
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Gitea Registry
uses: docker/login-action@v3
with:
registry: git.farh.net
username: admin
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: git.farh.net/farhoodlabs/paperclip-dev
tags: |
type=sha,prefix=
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ startsWith(gitea.ref, 'refs/tags/v') }}
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
no-cache: true
update-infra:
needs: build
runs-on: ubuntu-latest
steps:
- name: Update dev image tag in infra repo
run: |
SHA="${{ needs.build.outputs.image-tag }}"
FILE="overlays/dev/kustomization.yaml"
response=$(curl -sS \
-H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
"https://git.farh.net/api/v1/repos/farhoodlabs/paperclip-infra/contents/$FILE")
file_sha=$(echo "$response" | jq -r '.sha')
content=$(echo "$response" | jq -r '.content' | base64 -d)
new_content=$(echo "$content" | sed "s/newTag: \".*\"/newTag: \"$SHA\"/")
encoded=$(printf '%s' "$new_content" | base64 -w 0)
curl -sS -X PUT \
-H "Authorization: token ${{ secrets.REGISTRY_TOKEN }}" \
"https://git.farh.net/api/v1/repos/farhoodlabs/paperclip-infra/contents/$FILE" \
-d "{\"message\":\"chore(cd): update paperclip-dev to $SHA\",\"content\":\"$encoded\",\"sha\":\"$file_sha\"}"
+48
View File
@@ -0,0 +1,48 @@
name: "Build: Production"
on:
push:
branches: [local]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Gitea Registry
uses: docker/login-action@v3
with:
registry: git.farh.net
username: admin
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: git.farh.net/farhoodlabs/paperclip
tags: |
type=sha,prefix=
type=semver,pattern={{version}}
type=raw,value=latest,enable=${{ startsWith(gitea.ref, 'refs/tags/v') }}
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
no-cache: true
+43
View File
@@ -0,0 +1,43 @@
# Paperclip fork — farhoodlabs
This is a thin fork of [paperclipai/paperclip](https://github.com/paperclipai/paperclip).
Fork repo: https://git.farh.net/farhoodlabs/paperclip
## Branch model
| Branch | Purpose |
|---|---|
| `master` | Pure mirror of `upstream/master`. No fork files. Sync via `git push origin upstream/master:master --force-with-lease`. |
| `dev` | `master` + one fork commit (Dockerfile prod stage + 2 build workflows). Builds `git.farh.net/farhoodlabs/paperclip-dev:*` on push. |
| `local` | **Deployed branch.** Same content as `dev`. Builds `git.farh.net/farhoodlabs/paperclip:*` on push. |
The fork tree differs from upstream by exactly **3 files**:
```
Dockerfile (production stage adds kubectl, kubeseal, uv, forgejo CLIs, tea, mmx-cli, nano, vim)
.github/workflows/build-prod.yml (pushes to git.farh.net/farhoodlabs/paperclip)
.github/workflows/build-dev.yml (pushes to git.farh.net/farhoodlabs/paperclip-dev)
```
The base/deps/build stages of the Dockerfile match upstream verbatim so upstream changes apply cleanly.
## Sync upstream
```bash
git fetch upstream
git push origin upstream/master:master --force-with-lease
git checkout dev && git merge master && git push origin dev
git checkout local && git merge dev && git push origin local
```
Conflicts should only ever appear on `Dockerfile` itself (if upstream changes the production stage). Resolution rule: keep upstream's deps/base/build stages exactly; preserve the fork's `RUN` block in the production stage.
## Deployment
Production runs in Kubernetes (`paperclip` namespace, single replica). Image: `git.farh.net/farhoodlabs/paperclip:<tag>`. Flux does not watch moving tags — rolling a fix means either pushing a semver-tagged release or `kubectl rollout restart deploy/paperclip -n paperclip`.
## Don't
- **Don't add fork code changes.** This fork is intentionally minimal after the 2026-05-31 reset (event-loop starvation bug from accumulated drift). If a feature is missing relative to a prior fork iteration (Gitea-hosted skills, PAT support for private skill repos, secret export/import, k8s sandbox-provider plugin, agentId threading), surface the regression — don't pull it back from `git log` without explicit go-ahead.
- **Don't commit to `local` without going through `dev` first** (and through `master` for upstream syncs). The promotion order is enforced.
- **Don't recreate `.farhoodlabs/` overlay or `assemble-local.yml`.** That model was retired.
+22 -3
View File
@@ -57,10 +57,29 @@ ARG USER_UID=1000
ARG USER_GID=1000
WORKDIR /app
COPY --chown=node:node --from=build /app /app
RUN npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai \
&& apt-get update \
&& apt-get install -y --no-install-recommends openssh-client jq \
# Fork additions: kubectl, kubeseal, uv, forgejo CLIs, gitea tea CLI, editor tools, mmx-cli
# Upstream installs: claude-code, codex, opencode-ai, openssh-client, jq
RUN apt-get update \
&& apt-get install -y --no-install-recommends openssh-client jq nano vim \
&& rm -rf /var/lib/apt/lists/* \
&& curl -fsSL https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl \
&& curl -fsSL https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.36.6/kubeseal-0.36.6-linux-amd64.tar.gz | tar -xzf - -C /tmp \
&& mv /tmp/kubeseal /usr/local/bin/kubeseal \
&& rm -rf /tmp/kubeseal /tmp/LICENSE /tmp/README.md \
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
&& mv /root/.local/bin/uv /usr/local/bin/uv \
&& mv /root/.local/bin/uvx /usr/local/bin/uvx \
&& curl -fsSL https://codeberg.org/forgejo-contrib/forgejo-cli/releases/download/v0.4.1/forgejo-cli-linux.tar.gz | tar -xzf - -C /usr/local/bin \
&& chmod +x /usr/local/bin/fj \
&& curl -fsSL https://github.com/JKamsker/forgejo-cli-ex/releases/download/v0.1.7/fj-ex-linux-x86_64.tar.gz | tar -xzf - -C /usr/local/bin \
&& chmod +x /usr/local/bin/fj-ex \
&& curl -fsSL https://codeberg.org/romaintb/fgj/releases/download/v0.3.0/fgj_linux_amd64 -o /usr/local/bin/fgj \
&& chmod +x /usr/local/bin/fgj \
&& curl -fsSL https://dl.gitea.com/tea/0.14.0/tea-0.14.0-linux-amd64 -o /usr/local/bin/tea \
&& chmod +x /usr/local/bin/tea \
&& npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai \
&& npm install --global --omit=dev mmx-cli \
&& mkdir -p /paperclip \
&& chown node:node /paperclip
+16 -3
View File
@@ -376,9 +376,10 @@ Example:
Recovery rule:
- if the latest issue-linked run failed/timed out/cancelled and no live execution path remains, Paperclip queues one automatic assignment recovery wake
- if the issue has **no prior issue-linked run at all** — it is assigned and `todo`, no run was ever dispatched, no wake remains queued or running, and no recovery action is open — and its age exceeds the dispatch timeout (default 5 min), Paperclip treats it as **stalled-at-dispatch** and queues one automatic assignment recovery wake. Stalled-at-dispatch recovery does **not** require a prior failed, timed-out, or cancelled run; a never-dispatched assignment is a recoverable stall, not intentional rest.
- if that recovery wake also finishes and the issue is still stranded, Paperclip moves the issue to `blocked` and opens or updates an explicit recovery action when a bounded owner/action is known; the visible comment is evidence, not the recovery path by itself
This is a dispatch recovery, not a continuation recovery.
This is a dispatch recovery, not a continuation recovery. It covers both the post-crash stranded-run case and the zero-prior-run case where dispatch never produced a run.
### 9.2 Stranded assigned `in_progress`
@@ -410,11 +411,11 @@ On startup and on the periodic recovery loop, Paperclip now does five things in
1. reap orphaned `running` runs
2. resume persisted `queued` runs
3. reconcile stranded assigned work
3. reconcile stranded assigned work, including assigned `todo`/`in_progress` issues that have **never produced a linked run**; stalled-at-dispatch detection does not require a prior linked run
4. scan silent active runs, revalidate their source issues, and either fold source-resolved watchdogs or create/update explicit watchdog recovery actions
5. reconcile productivity reviews
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
The stranded-work pass closes the gap where issue state survives a crash but the wake/run path does not. It also covers the never-dispatched case: an assigned `todo` whose dispatch never started a run, has no queued wake, and has exceeded the dispatch timeout is reconciled as stalled-at-dispatch even though no prior run exists. The silent-run scan covers the separate case where a live process exists but has stopped producing observable output. The productivity-review pass is later and separate; it reviews unusual progression patterns on assigned source issues, not stale run handles after a source issue already has a valid disposition.
## 11. Silent Active-Run Watchdog
@@ -441,6 +442,18 @@ Operators should prefer `snooze` for known time-bounded quiet periods. `continue
The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot.
### Adapter heartbeat staleness (pre-run)
The silent active-run watchdog above covers a run that is `running` but has stopped producing output. It does **not** cover an agent adapter that is wedged *before* any run is linked to an assigned issue. A wedged adapter can report `status: running` while its `lastHeartbeatAt` stops advancing, so dispatch triggers (assignment, @-mention, blocker-resolved wakes) fire without ever starting a run. `status: running` is therefore not, by itself, evidence of liveness — `lastHeartbeatAt` advancement is.
For every agent adapter assigned to a non-terminal issue, if `lastHeartbeatAt` has not advanced beyond a configured staleness threshold (default 15 min), Paperclip MUST, independent of whether any run is linked to the issue:
- open an explicit recovery action on the stalled issue that names the wedged adapter, the heartbeat-staleness evidence (last `lastHeartbeatAt`, staleness duration), the recovery owner, and the next action
- alert/escalate to the assignee's manager
- surface the stall visibly in activity and UI so operators can distinguish a wedged adapter from healthy idle work
This extends the watchdog contract from run-output silence to adapter-level silence that predates any linked run. Bounds mirror the active-run watchdog: at most one open adapter-staleness recovery action per adapter per staleness window, and the action folds through the normal explicit-recovery lifecycle once `lastHeartbeatAt` resumes advancing (the adapter self-recovered) or the issue otherwise reaches a valid live/waiting/terminal path.
### Source-aware watchdog folding
Active-run watchdog work is source-aware. Before the watchdog creates, refreshes, escalates, or blocks on reviewer work, it must re-read the linked source issue and decide whether the watchdog signal is still about productive source work or only about stale run/process bookkeeping.