31 Commits

Author SHA1 Message Date
Devin Foley 81d18f2d77 ci: speed up PR verify workflow (#6137)
## Thinking Path

> - Paperclip orchestrates AI agents through a control-plane repo that
relies on GitHub Actions as part of its release and verification safety
net.
> - The PR workflow in `.github/workflows/pr.yml` is the core CI path
protecting pull requests before merge.
> - Baseline measurement work in [PAPA-335](/PAPA/issues/PAPA-335)
showed the old single `verify` job was the critical-path bottleneck,
with general tests and build serialized together.
> - Follow-up implementation in [PAPA-338](/PAPA/issues/PAPA-338) and
[PAPA-339](/PAPA/issues/PAPA-339) split that work into parallel lanes
and removed redundant clean-runner prebuild work.
> - [PAPA-340](/PAPA/issues/PAPA-340) now needs real post-change PR
workflow evidence, not local inference, to compare against the May 15,
2026 baseline and decide whether phase-2 work is still justified.
> - This pull request publishes the already-implemented CI speedup
branch so GitHub can run the actual `PR` workflow against it.
> - The benefit is that CI timing decisions are based on measured runs
from the exact workflow shape we intend to ship.

## What Changed

- Split the PR workflow so `policy` fans out into separate `Typecheck +
Release Registry`, grouped `General tests`, and `Build` jobs.
- Kept the serialized server matrix, canary dry run, and e2e jobs intact
while removing the old monolithic `verify` bottleneck.
- Reworked grouped general-test execution in
`scripts/run-vitest-stable.mjs` so the workflow can run balanced
non-serialized lanes.
- Replaced redundant clean-runner prebuild gates with the idempotent
`ensure-build-deps` path used by the relevant CI entrypoints.

## Verification

- `ruby -e "require 'yaml'; YAML.load_file('.github/workflows/pr.yml');
puts 'yaml-ok'"`
- `node scripts/run-vitest-stable.mjs --mode general --dry-run`
- `node scripts/run-vitest-stable.mjs --mode general --group
general-server --dry-run`
- `node scripts/run-vitest-stable.mjs --mode general --group
general-workspaces-a --dry-run`
- `node scripts/run-vitest-stable.mjs --mode general --group
general-workspaces-b --dry-run`
- `pnpm test:run:general -- --group general-workspaces-b`
- `pnpm test:run:general -- --group general-workspaces-a`
- `pnpm test:run:general -- --group general-server`
- `pnpm run typecheck:build-gaps`
- `pnpm --filter @paperclipai/plugin-hello-world-example typecheck`

## Risks

- Required-check and branch-protection settings may still reference the
old single `verify` job name.
- Parallel CI lanes can expose hidden ordering assumptions or
clean-runner bootstrap gaps that local grouped dry-runs did not surface.
- Because the branch is behind current `master`, merge conflicts or
unrelated upstream drift could affect the measured runtime until the
branch is rebased.

> Checked `ROADMAP.md`; this work is CI throughput maintenance for the
existing PR verification path, not duplicate feature work.

## Model Used

- OpenAI Codex via Paperclip `codex_local`, GPT-5-class coding agent
with repository read/write, shell execution, and GitHub CLI/tool use.
The runtime does not expose a more specific backend model ID in-session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-16 11:28:25 -07:00
Devin Foley a72731f118 fix: harden release registry verification against npm lag (#4816)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Its release automation publishes canary packages to npm and then
validates the published registry state before considering the release
healthy
> - The failing canary run `25139465018` showed that npm can expose a
newly published version through version-specific endpoints before the
root package document has fully converged
> - That made a successful canary publish look like a failed release
because the verifier trusted stale root metadata too early
> - This pull request hardens the registry verification path by
preferring version-specific manifest checks, retrying
convergence-sensitive failures, and distinguishing permanent failures
from propagation lag
> - While validating that change in CI, a separate teardown race in
`heartbeat-stale-queue-invalidation.test.ts` surfaced and was hardened
so the PR could pass reliably
> - The benefit is that transient npm propagation lag no longer fails a
successful canary publish, while genuine registry-state and
dependency-integrity failures still stop the release flow promptly

## What Changed

- Hardened `scripts/verify-release-registry-state.mjs` so it prefers
version-specific manifest resolution over stale root metadata, adds
bounded registry-fetch timeouts, and classifies failures as retriable vs
non-retriable.
- Updated `scripts/release-lib.sh` and `scripts/release.sh` so
post-publish registry verification retries only convergence-sensitive
failures and reports immediate permanent failures clearly.
- Expanded `scripts/verify-release-registry-state.test.mjs` with
regression coverage for stale root metadata, fetch timeout behavior,
peer dependency range handling, non-retriable canary-latest cases, and
related verifier edge cases.
- Hardened
`server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`
teardown to tolerate the late-comment foreign-key race that CI exposed
while validating this branch.

## Verification

- `pnpm run test:release-registry`
- `node --check scripts/verify-release-registry-state.mjs`
- `bash -n scripts/release.sh && bash -n scripts/release-lib.sh`
- PR checks passed on head `5c422600fc12acac61f6b7c267a4dc915df622b1`:
`policy`, `verify`, `e2e`, `security/snyk`, and `Greptile Review`

## Risks

- Low risk. The main behavioral changes are limited to release
automation and verifier retry semantics, plus a test-only teardown
hardening for a CI race.

> I checked [`ROADMAP.md`](ROADMAP.md). This is a narrow release bugfix
and does not overlap planned core feature work.

## Model Used

- OpenAI Codex via Paperclip `codex_local` with tool use and local code
execution enabled. This agent session runs on a GPT-5-class coding
model; the exact backend model ID/context window is not exposed by the
local adapter runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I have addressed all Greptile and reviewer comments before
requesting merge
2026-05-09 22:18:12 -07:00
Devin Foley f784d8d90e Retry canary registry verification (#5579)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies, and the
release pipeline is part of keeping that control plane shippable.
> - The relevant subsystem here is the release automation in
`scripts/release.sh`, which publishes canary builds and then verifies
npm registry state.
> - The failing CI run showed a successful publish followed by an
immediate registry-state verification failure while npm dist-tag
metadata was still propagating.
> - That made the canary job flaky even when the publish itself had
succeeded, which is the wrong failure mode for release automation.
> - This pull request adds bounded retries around the post-publish
registry-state verification step instead of failing on the first stale
read.
> - The benefit is that canary releases tolerate transient npm
propagation lag while still failing clearly if registry metadata never
converges.

## What Changed

- Wrapped the post-publish `verify-release-registry-state.mjs` call in a
bounded retry loop inside `scripts/release.sh`.
- Reused the existing publish verification retry defaults and added
optional overrides via `NPM_REGISTRY_STATE_VERIFY_ATTEMPTS` and
`NPM_REGISTRY_STATE_VERIFY_DELAY_SECONDS` for dist-tag-specific tuning.

## Verification

- `bash -n scripts/release.sh`
- CI will also exercise the release path via the existing `Canary Dry
Run` workflow job in `.github/workflows/pr.yml`.

## Risks

- Low risk. The main behavioral change is that a genuinely broken
registry-state verification can now wait through the configured retry
window before failing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex local agent, GPT-5-based Codex runtime in Paperclip with
tool use and shell execution. The exact backend model ID/context window
is not surfaced in this local heartbeat environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-05-09 11:40:02 -07:00
Devin Foley a0f5cbffd7 Harden release flow with registry verification and dist-tag checks (#4800)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Paperclip is distributed as npm packages, including plugins like
`plugin-e2b`
> - The release process publishes canary and stable builds via npm
dist-tags
> - But there was no automated verification that published packages
actually landed with the correct dist-tags, and broken canary publishes
could silently ship to users
> - This PR adds a registry verification script that checks published
packages match their expected dist-tags, and wires it into PR CI so
regressions are caught before merge
> - The benefit is release integrity is verified automatically, and
broken dist-tag states are caught early

## What Changed

- Added `scripts/verify-release-registry-state.mjs` — verifies that
published npm packages have correct dist-tag assignments and detects
orphaned or mispointed tags
- Added `scripts/verify-release-registry-state.test.mjs` — test coverage
for the verification logic
- Updated `scripts/release.sh` to include canary dist-tag safety checks
before publishing
- Updated `.github/workflows/pr.yml` to run registry verification as a
CI step
- Updated `doc/PUBLISHING.md` and `doc/RELEASING.md` with the new
verification workflow

## Verification

- `pnpm test` — all tests pass including new verification script tests
- `node scripts/verify-release-registry-state.mjs` — runs against the
live npm registry and reports current state
- CI: the new PR workflow step runs on every PR push

## Risks

- Low risk. This is additive CI and tooling — no runtime code changes.
The registry verification is read-only (queries npm, does not publish).
The release script changes add safety checks that abort before
publishing if state is unexpected.

## Model Used

Codex GPT 5.4 high via Paperclip.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-04-29 15:56:20 -07:00
Devin Foley 40782f703d Fix release packaging for standalone public packages (#4494)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies, and the
sandbox-provider work just moved E2B into a standalone publishable
plugin package.
> - That plugin is intentionally excluded from the root pnpm workspace
so it can model third-party install behavior without forcing lockfile
churn in the main repo.
> - The merged architecture change exposed a follow-up release problem:
the canary publish workflow tried to publish `@paperclipai/plugin-e2b`,
but the tarball had no `dist/` payload because standalone public
packages were not being built in the release path.
> - That means the release pipeline needed a packaging fix in core
release tooling, not another architectural change in the sandbox
provider itself.
> - This pull request adds a generic release step for public packages
that live outside the pnpm workspace, instead of hardcoding E2B-specific
behavior into the release script.
> - The benefit is that standalone publishable packages can be built and
packed correctly during release, including future sandbox-provider
plugins that follow the same pattern.

## What Changed

- Added `scripts/build-standalone-public-packages.mjs` to discover
public packages outside the pnpm workspace, run a clean package-local
install, and build them before publish.
- Updated `scripts/release.sh` to invoke that helper immediately after
the normal workspace build step.
- Kept the behavior generic by driving off the existing public package
map and pnpm workspace patterns rather than special-casing
`@paperclipai/plugin-e2b`.

## Verification

- `rm -rf packages/plugins/sandbox-providers/e2b/dist`
- `node ./scripts/build-standalone-public-packages.mjs`
- `cd packages/plugins/sandbox-providers/e2b && npm pack --dry-run`
- Confirm the tarball now includes the rebuilt `dist/` files instead of
only `README.md` / `package.json`

## Risks

- Low risk: this only changes the release build path for public packages
outside the pnpm workspace.
- The helper performs a clean package-local install for each standalone
public package, so release time may increase slightly as more such
packages are added.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex via `codex_local`
- Model ID: `gpt-5.4`
- Reasoning effort: `high`
- Context window observed in runtime session metadata: `258400` tokens
- Capabilities used: terminal tool execution, git, GitHub CLI, and local
build/test inspection

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-04-25 12:16:23 -07:00
dotta 3e0e15394a chore: switch release calver to mdd patch 2026-03-18 07:57:36 -05:00
dotta 21f7adbe45 fix: advance canary after partial publish 2026-03-17 16:31:38 -05:00
repro 9176218d16 fix: validate canary release path in CI 2026-03-17 15:35:59 -05:00
Dotta 48326da83f fix: restore release script executable bit 2026-03-17 14:09:16 -05:00
Dotta 21c1235277 chore: automate canary and stable releases 2026-03-17 14:08:55 -05:00
Dotta 964e04369a fixes verification 2026-03-12 12:55:26 -05:00
Dotta 873535fbf0 verify the packages actually make it to npm 2026-03-12 12:42:00 -05:00
Dotta 469bfe3953 chore: add release train workflow 2026-03-09 13:55:30 -05:00
Dotta 422f57b160 chore: use public-gh for manual release flow 2026-03-09 10:33:56 -05:00
Dotta 31c947bf7f fix: publish canaries in changesets pre mode 2026-03-09 10:23:04 -05:00
Dotta f5bf743745 fix: support older git in release cleanup 2026-03-09 10:11:46 -05:00
Dotta df94c98494 chore: add release preflight workflow 2026-03-09 09:06:45 -05:00
Dotta a7cfd9f24b chore: formalize release workflow 2026-03-09 08:49:42 -05:00
Dotta 77e04407b9 fix(publish): always bundle ui-dist into server package 2026-03-09 07:21:33 -05:00
Dotta 048e2b1bfe Remove legacy OpenClaw adapter and keep gateway-only flow 2026-03-07 18:50:25 -06:00
Dotta e693e3d466 feat(release): auto-create GitHub Release on publish
Add create_github_release helper to release script that creates a
GitHub Release via gh CLI after tagging, using release notes from
releases/vX.Y.Z.md if available or auto-generated notes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 16:50:25 -06:00
Dotta 35e4897256 Merge pull request #141 from aaaaron/integrate-opencode-pr62
Integrate opencode pr62 & pr104
2026-03-06 11:32:03 -06:00
Dotta 7c07b16f80 feat(release): add --canary and --promote flags to release.sh
Support two-phase canary release flow:
- --canary: publishes packages under @canary npm tag, skips git commit/tag
- --promote <version>: moves canary packages to @latest, then commits and tags

Both flags work with --dry-run. Existing behavior unchanged when neither flag is passed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 16:03:49 -06:00
Konan69 6a101e0da1 Add OpenCode provider integration and strict model selection 2026-03-05 15:24:20 +01:00
Dotta a4da932d8d fix: remove stale server/ui-dist from git add in release.sh
The release script cleaned up temporary build artifacts (ui-dist, skills)
before staging files for the commit, then tried to git add server/ui-dist
which no longer existed.  Remove it from the git add list since these
artifacts should never be committed — they're only needed during publish.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 16:14:28 -06:00
Dotta f4a5b00116 fix: bundle skills directory into npm packages for runtime discovery
The claude-local, codex-local adapters and the server all resolve a
skills/ directory using __dirname-relative paths that only work inside
the monorepo.  When installed from npm the paths point outside the
package and cause ENOENT on readdir/readFile.

- Update both adapter execute.ts files to try a published-path
  candidate (../../skills from dist/) before falling back to the
  monorepo dev path (../../../../../skills from src/).
- Update server readSkillMarkdown() to try the published path first.
- Add "skills" to the files array in server, claude-local, and
  codex-local package.json so npm includes them.
- Update release.sh to copy the repo-root skills/ into each package
  before publish, and clean up after.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 16:06:12 -06:00
Dotta 09d2ef1a37 fix: restore docs deleted in v0.2.3 release, add Paperclip branding
- Restored docs/ directory that was accidentally deleted by `git add -A`
  in the v0.2.3 release script
- Replaced generic "P" favicon with actual paperclip icon using brand
  primary color (#2563EB)
- Added light/dark logo SVGs for Mintlify navbar (paperclip icon + wordmark)
- Updated docs.json with logo configuration for dark/light mode
- Fixed release.sh to stage only release-related files instead of `git add -A`
  to prevent sweeping unrelated changes into release commits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:49:43 -06:00
Dotta d18312d6de fix: bundle UI dist into server package for npm publishing
The server's static-ui mode resolves the UI dist path relative to its
own directory. In the monorepo it finds ../../ui/dist, but when published
to npm the UI package isn't available.

- server/src/app.ts: try ../ui-dist (published) then ../../ui/dist (dev),
  gracefully degrade to API-only if neither exists
- server/package.json: include ui-dist/ in published files
- scripts/release.sh: build UI and copy dist to server/ui-dist before
  publishing, clean up in restore step

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:45:45 -06:00
Dotta c11d2d3eda fix: copy repo README to CLI for npm publishing
build-npm.sh now copies the root README.md into cli/ so the npm package
page shows the same README as the GitHub repo. release.sh cleans up the
copy after publishing during the restore step.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:05:41 -06:00
Dotta 24b393b08f fix: release script grep -P macOS compat and CLI typecheck actor errors
- Replace grep -P (Perl regex, unavailable on macOS) with sed for
  extracting the CLI version string in release.sh
- Add triple-slash reference to server/src/types/express.d.ts in
  server/src/index.ts so CLI's tsc picks up the Express Request
  augmentation when following workspace links

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:58:55 -06:00
Dotta defccdd4d9 feat: integrate Changesets for multi-package npm publishing
Migrate from single-bundle CLI publishing to publishing all @paperclipai/*
packages individually via Changesets. This fixes the "Cannot find package
@paperclipai/server" error when installing from npm.

Changes:
- Add @changesets/cli with fixed versioning (all packages share version)
- Make 7 packages publishable (shared, adapter-utils, db, 3 adapters, server)
- Add build scripts, publishConfig, and files fields to all packages
- Mark @paperclipai/server as external in CLI esbuild config
- Simplify CLI importServerEntry() to use string-literal dynamic import
- Add generate-npm-package-json support for external workspace packages
- Create scripts/release.sh for one-command releases
- Remove old bump-and-publish.sh and version-bump.sh
- All packages start at version 0.2.0

Usage: ./scripts/release.sh patch|minor|major [--dry-run]

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:46:16 -06:00