feat: add npx CLI with monorepo, CI/CD, and ephemeral worker architecture (#256)

* feat: integrate npx CLI, CI/CD, and ephemeral worker architecture

Bring in changes from shannon-npx: npx-distributable CLI package (cli/),
semantic-release CI/CD workflows, ephemeral per-scan worker containers,
TOML config support, setup wizard, and workspace management.

Preserves all shannon-only changes: security hardening (localhost-bound
ports, MCP env allowlist, path traversal guard), updated benchmarks
(XBEN 19/31/35/44), README assets, and prompt injection disclaimer.

Applies security hardening to cli/infra/compose.yml as well.

* refactor: migrate to Turborepo + pnpm + Biome monorepo

Restructure into apps/worker, apps/cli, packages/mcp-server with
Turborepo task orchestration, pnpm workspaces, Biome linting/formatting,
and tsdown CLI bundling.

Key changes:
- src/ -> apps/worker/src/, cli/ -> apps/cli/, mcp-server/ -> packages/mcp-server/
- prompts/ and configs/ moved into apps/worker/
- npm replaced with pnpm, package-lock.json replaced with pnpm-lock.yaml
- Dockerfile updated for pnpm-based builds
- CLI logs command rewritten with chokidar for cross-platform reliability
- Router health checking added for auto-detected router mode
- Centralized path resolution via apps/worker/src/paths.ts

* fix: resolve all biome warnings and formatting issues

- Remove unnecessary non-null assertions where values are guaranteed
- Replace array index access with .at() for safer element retrieval
- Use local variables to avoid repeated process.env lookups
- Replace any types with unknown in functional utilities
- Use nullish coalescing for TOTP hash byte access
- Auto-format security patches to match biome config

* fix: pin pnpm to 10.12.1 in Dockerfile for catalog support

* fix: handle Esc cancellation in Bedrock setup flow

Replace p.group() with individual prompts and per-field cancel checks,
matching the pattern used by all other provider setup flows.

* feat: add optional model customization to Anthropic setup

* fix: resolve Docker bind mount permission errors on Linux

Use entrypoint-based UID remapping instead of --user flag so the
container's pentest user matches the host UID/GID, keeping bind-mounted
volumes writable. Git config moved to --system level to survive remapping.

* fix: show resumed workflow ID in splash screen URL

When resuming a workflow, the Temporal Web UI link pointed to the old
(terminated) workflow ID. Now extracts "New Workflow ID" from the resume
header in workflow.log, falling back to the original ID for fresh scans.

* style: fix biome formatting in docker.ts

* fix: align TypeScript config types with JSON Schema

- SuccessCondition.type: use schema values (url_contains,
  element_present, url_equals_exactly, text_contains) instead of
  stale values (url, cookie, element, redirect)
- Authentication.login_flow: mark optional to match schema which
  does not require it

* feat: mark GitHub release as latest during rollback

* fix: use native ARM64 runners for Docker multi-platform builds

Replace QEMU emulation with parallel native builds using a matrix
strategy (ubuntu-latest for amd64, ubuntu-24.04-arm for arm64).
Each platform pushes by digest, then a merge job creates the
multi-arch manifest list before signing with cosign.

* fix: resolve SessionMutex race condition with 3+ concurrent waiters

* fix: skip POSIX permission check on Windows

writeFileSync mode option is ignored on Windows, so config.toml
gets 0o666 and the guard rejects it.

* fix: resolve unsubstituted placeholders in report prompt

Remove unused {{GITHUB_URL}} placeholder and wire up {{AUTH_CONTEXT}}
with structured auth context (login type, username, URL, MFA status).

* fix: remove duplicate environment gate from merge-docker job

Move DOCKERHUB_USERNAME from vars to secrets so merge-docker can access
credentials without its own environment scope. This eliminates the
redundant double approval since build-docker already gates on
release-publish.

* fix: replace POSIX sleep binary with cross-platform async sleep

execFileSync('sleep') is unavailable on Windows. Use node:timers/promises
setTimeout instead, making ensureInfra async.

* fix: use session.json for workflow ID on resume instead of parsing workflow.log

On resume, workflow.log already exists with stale headers from the
previous run. The CLI poll found '====' immediately and extracted the
old workflow ID, producing a wrong Temporal Web UI URL.

Read the workflow ID from session.json instead — the worker writes
resume attempts there atomically. For fresh runs, poll until
originalWorkflowId appears. For resumes, poll until a new
resumeAttempts entry is appended.

* feat: add custom base URL support for Anthropic-compatible proxies

Support ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN to route SDK requests
through LiteLLM or any Anthropic-compatible proxy. Adds TUI wizard
option, TOML config mapping, credential validation, and preflight
endpoint reachability check via SDK query.

* fix: remove environment gates and add NPM_TOKEN to publish step

* feat: add beta release and rollback workflows with cosign signing

* fix: remove redundant checkout and pnpm steps from beta release workflow

* docs: normalize README commands to mode-neutral shorthand

Add a substitution note after Quick Start sections so all subsequent
examples use bare `shannon` instead of mixing `./shannon` and
`npx @keygraph/shannon`. Mode-specific commands (build, update,
uninstall) get inline annotations. Also fixes a broken command in the
Custom Base URL section.

* fix: remove redundant `update` command

Image is already auto-pulled by `ensureImage()` during `start` when the
pinned version tag is missing locally. Manual `update` was unnecessary.

* docs: add CLI package README stub

* docs: update README setup instructions for dual CLI modes

* docs: update announcement banner to npx availability

* feat: migrate from MCP tools to CLI based tools (#252)

* feat: migrate from MCP tools to CLI tools

* fix: restore browser action emoji formatters for CLI output

Adapt formatBrowserAction for playwright-cli commands, replacing the old
mcp__playwright__browser_* tool name matching removed during migration.

* fix: mount credential file to fixed container path for Vertex AI

GOOGLE_APPLICATION_CREDENTIALS was forwarded as-is to the container,
causing the relative host path to resolve against the repo mount
instead of the credentials mount. Now both local and npx modes mount
the resolved file to /app/credentials/google-sa-key.json and rewrite
the env var to match.

* feat: add git awareness and optional description field to config

* fix: drop redundant --ipc host flag from worker container

* fix: align announcement banner URL with main branch

* feat: add target URL reachability preflight check (#254)

* Moving asset benchmark graph image to this folder

* Move benchmark results to benchmark repo

Windows Defender flags exploit code in the pentest reports as false positives, forcing every Windows user to add a Defender exclusion just to clone Shannon.

* Updated README

* fix: case-insensitive grep for semantic-release version probe

* fix: harden supply chain security (#255)

* fix: patch smol-toml and tsdown vulnerabilities

Update smol-toml 1.6.0→1.6.1 (DoS via recursive comment parsing) and
tsdown 0.21.2→0.21.5 (picomatch ReDoS + method injection).

* fix: pin all unpinned dependency versions in Dockerfile

Pins subfinder v2.13.0, WhatWeb v0.6.3 (switched from git clone to
release tarball), schemathesis 4.13.0, addressable 2.8.9,
claude-code 2.1.84, and playwright-cli 0.1.1 for reproducible builds.

* fix: pin GitHub Actions to commit SHAs for supply chain security

* fix: pin GitHub Actions to commit SHAs in beta and rollback workflows
This commit is contained in:
ezl-keygraph
2026-03-27 02:34:29 +05:30
committed by GitHub
parent 0d172f5e32
commit bc8fd203ed
4058 changed files with 7774 additions and 1189080 deletions
+3 -378
View File
@@ -1,378 +1,3 @@
#!/bin/bash
# Shannon CLI - AI Penetration Testing Framework
set -e
# Prevent MSYS from converting Unix paths (e.g. /repos/my-repo) to Windows paths
case "$OSTYPE" in
msys*) export MSYS_NO_PATHCONV=1 ;;
esac
# Detect Podman vs Docker and set compose files accordingly
# Podman doesn't support host-gateway, so we only include the Docker override for actual Docker
COMPOSE_BASE="docker-compose.yml"
if command -v podman &>/dev/null; then
# Podman detected (either native or via Docker Desktop shim) - use base config only
COMPOSE_OVERRIDE=""
else
# Docker detected - include extra_hosts override for Linux localhost access
COMPOSE_OVERRIDE="-f docker-compose.docker.yml"
fi
COMPOSE_FILE="$COMPOSE_BASE"
# Load .env if present
if [ -f .env ]; then
set -a
source .env
set +a
fi
show_help() {
cat << 'EOF'
███████╗██╗ ██╗ █████╗ ███╗ ██╗███╗ ██╗ ██████╗ ███╗ ██╗
██╔════╝██║ ██║██╔══██╗████╗ ██║████╗ ██║██╔═══██╗████╗ ██║
███████╗███████║███████║██╔██╗ ██║██╔██╗ ██║██║ ██║██╔██╗ ██║
╚════██║██╔══██║██╔══██║██║╚██╗██║██║╚██╗██║██║ ██║██║╚██╗██║
███████║██║ ██║██║ ██║██║ ╚████║██║ ╚████║╚██████╔╝██║ ╚████║
╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═══╝╚═╝ ╚═══╝ ╚═════╝ ╚═╝ ╚═══╝
AI Penetration Testing Framework
Usage:
./shannon start URL=<url> REPO=<name> Start a pentest workflow
./shannon workspaces List all workspaces
./shannon logs ID=<workflow-id> Tail logs for a specific workflow
./shannon stop Stop all containers
./shannon help Show this help message
Options for 'start':
REPO=<name> Folder name under ./repos/ (e.g. REPO=repo-name)
CONFIG=<path> Configuration file (YAML)
OUTPUT=<path> Output directory for reports (default: ./audit-logs/)
WORKSPACE=<name> Named workspace (auto-resumes if exists, creates if new)
PIPELINE_TESTING=true Use minimal prompts for fast testing
ROUTER=true Route requests through claude-code-router (multi-model support)
Options for 'stop':
CLEAN=true Remove all data including volumes
Examples:
./shannon start URL=https://example.com REPO=repo-name
./shannon start URL=https://example.com REPO=repo-name WORKSPACE=q1-audit
./shannon start URL=https://example.com REPO=repo-name CONFIG=./config.yaml
./shannon start URL=https://example.com REPO=repo-name OUTPUT=./my-reports
./shannon workspaces
./shannon logs ID=example.com_shannon-1234567890
./shannon stop CLEAN=true
Monitor workflows at http://localhost:8233
EOF
}
# Parse KEY=value arguments into variables
parse_args() {
for arg in "$@"; do
case "$arg" in
URL=*) URL="${arg#URL=}" ;;
REPO=*) REPO="${arg#REPO=}" ;;
CONFIG=*) CONFIG="${arg#CONFIG=}" ;;
OUTPUT=*) OUTPUT="${arg#OUTPUT=}" ;;
ID=*) ID="${arg#ID=}" ;;
CLEAN=*) CLEAN="${arg#CLEAN=}" ;;
PIPELINE_TESTING=*) PIPELINE_TESTING="${arg#PIPELINE_TESTING=}" ;;
REBUILD=*) REBUILD="${arg#REBUILD=}" ;;
ROUTER=*) ROUTER="${arg#ROUTER=}" ;;
WORKSPACE=*) WORKSPACE="${arg#WORKSPACE=}" ;;
esac
done
}
# Check if Temporal is running and healthy
is_temporal_ready() {
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE exec -T temporal \
temporal operator cluster health --address localhost:7233 2>/dev/null | grep -q "SERVING"
}
# Ensure containers are running with correct mounts
ensure_containers() {
# If custom OUTPUT_DIR is set, always refresh worker to ensure correct volume mount
# Docker compose will only recreate if the mount actually changed
if [ -n "$OUTPUT_DIR" ]; then
echo "Ensuring worker has correct output mount..."
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE up -d worker 2>/dev/null || true
fi
# Quick check: if Temporal is already healthy, we're good
if is_temporal_ready; then
return 0
fi
# Need to start containers
echo "Starting Shannon containers..."
if [ "$REBUILD" = "true" ]; then
# Force rebuild without cache (use when code changes aren't being picked up)
echo "Rebuilding with --no-cache..."
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE build --no-cache worker
fi
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE up -d --build
# Wait for Temporal to be ready
echo "Waiting for Temporal to be ready..."
for i in $(seq 1 30); do
if is_temporal_ready; then
echo "Temporal is ready!"
return 0
fi
if [ "$i" -eq 30 ]; then
echo "Timeout waiting for Temporal"
exit 1
fi
sleep 2
done
}
cmd_start() {
parse_args "$@"
# Validate required vars
if [ -z "$URL" ] || [ -z "$REPO" ]; then
echo "ERROR: URL and REPO are required"
echo "Usage: ./shannon start URL=<url> REPO=<name>"
exit 1
fi
# Check for API key (Bedrock, Vertex, router, and custom base URL modes can bypass this)
if [ -z "$ANTHROPIC_API_KEY" ] && [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
if [ -n "$ANTHROPIC_BASE_URL" ] && [ -n "$ANTHROPIC_AUTH_TOKEN" ]; then
# Custom base URL mode — use auth token as API key for SDK initialization
echo "Using custom base URL: $ANTHROPIC_BASE_URL"
elif [ "$CLAUDE_CODE_USE_BEDROCK" = "1" ]; then
# Bedrock mode — validate required AWS credentials
MISSING=""
[ -z "$AWS_REGION" ] && MISSING="$MISSING AWS_REGION"
[ -z "$AWS_BEARER_TOKEN_BEDROCK" ] && MISSING="$MISSING AWS_BEARER_TOKEN_BEDROCK"
[ -z "$ANTHROPIC_SMALL_MODEL" ] && MISSING="$MISSING ANTHROPIC_SMALL_MODEL"
[ -z "$ANTHROPIC_MEDIUM_MODEL" ] && MISSING="$MISSING ANTHROPIC_MEDIUM_MODEL"
[ -z "$ANTHROPIC_LARGE_MODEL" ] && MISSING="$MISSING ANTHROPIC_LARGE_MODEL"
if [ -n "$MISSING" ]; then
echo "ERROR: Bedrock mode requires the following env vars in .env:$MISSING"
exit 1
fi
elif [ "$CLAUDE_CODE_USE_VERTEX" = "1" ]; then
# Vertex AI mode — validate required GCP credentials
MISSING=""
[ -z "$CLOUD_ML_REGION" ] && MISSING="$MISSING CLOUD_ML_REGION"
[ -z "$ANTHROPIC_VERTEX_PROJECT_ID" ] && MISSING="$MISSING ANTHROPIC_VERTEX_PROJECT_ID"
[ -z "$ANTHROPIC_SMALL_MODEL" ] && MISSING="$MISSING ANTHROPIC_SMALL_MODEL"
[ -z "$ANTHROPIC_MEDIUM_MODEL" ] && MISSING="$MISSING ANTHROPIC_MEDIUM_MODEL"
[ -z "$ANTHROPIC_LARGE_MODEL" ] && MISSING="$MISSING ANTHROPIC_LARGE_MODEL"
if [ -n "$MISSING" ]; then
echo "ERROR: Vertex AI mode requires the following env vars in .env:$MISSING"
exit 1
fi
# Validate service account key file (must be inside ./credentials/ for Docker mount)
if [ -z "$GOOGLE_APPLICATION_CREDENTIALS" ]; then
echo "ERROR: Vertex AI mode requires GOOGLE_APPLICATION_CREDENTIALS in .env"
echo " Place your service account key in ./credentials/ and set:"
echo " GOOGLE_APPLICATION_CREDENTIALS=./credentials/gcp-sa-key.json"
exit 1
fi
if [ ! -f "$GOOGLE_APPLICATION_CREDENTIALS" ]; then
echo "ERROR: Service account key file not found: $GOOGLE_APPLICATION_CREDENTIALS"
echo " Download a key from the GCP Console (IAM > Service Accounts > Keys)"
exit 1
fi
elif [ "$ROUTER" = "true" ] && { [ -n "$OPENAI_API_KEY" ] || [ -n "$OPENROUTER_API_KEY" ]; }; then
# Router mode with alternative provider - set a placeholder for SDK init
export ANTHROPIC_API_KEY="router-mode"
else
echo "ERROR: Set ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN in .env"
echo " (or use CLAUDE_CODE_USE_BEDROCK=1 for AWS Bedrock,"
echo " CLAUDE_CODE_USE_VERTEX=1 for Google Vertex AI,"
echo " or ROUTER=true with OPENAI_API_KEY or OPENROUTER_API_KEY)"
exit 1
fi
fi
# Determine container path for REPO
# - If REPO is already a container path (/benchmarks/*, /repos/*), use as-is
# - Otherwise, treat as a folder name under ./repos/ (mounted at /repos in container)
case "$REPO" in
/benchmarks/*|/repos/*)
CONTAINER_REPO="$REPO"
;;
*)
if [ ! -d "./repos/$REPO" ]; then
echo "ERROR: Repository not found at ./repos/$REPO"
echo ""
echo "Place your target repository under the ./repos/ directory"
exit 1
fi
CONTAINER_REPO="/repos/$REPO"
;;
esac
# Handle custom OUTPUT directory
# Export OUTPUT_DIR for docker-compose volume mount BEFORE starting containers
if [ -n "$OUTPUT" ]; then
# Create output directory with write permissions for container user (UID 1001)
mkdir -p "$OUTPUT"
chmod 777 "$OUTPUT"
export OUTPUT_DIR="$OUTPUT"
fi
# Handle ROUTER flag - start claude-code-router for multi-model support
if [ "$ROUTER" = "true" ]; then
# Check if router is already running
if docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE --profile router ps router 2>/dev/null | grep -q "running"; then
echo "Router already running, skipping startup..."
else
echo "Starting claude-code-router..."
# Check for provider API keys
if [ -z "$OPENAI_API_KEY" ] && [ -z "$OPENROUTER_API_KEY" ]; then
echo "WARNING: No provider API key set (OPENAI_API_KEY or OPENROUTER_API_KEY). Router may not work."
fi
# Start router with profile
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE --profile router up -d router
# Give router a few seconds to start (health check disabled for now - TODO: debug later)
echo "Waiting for router to start..."
sleep 5
fi
# Set ANTHROPIC_BASE_URL to route through router
export ANTHROPIC_BASE_URL="http://router:3456"
# Set auth token to match router's APIKEY
export ANTHROPIC_AUTH_TOKEN="shannon-router-key"
fi
# Ensure audit-logs directory exists with write permissions for container user (UID 1001)
mkdir -p ./audit-logs ./credentials
chmod 777 ./audit-logs
# Ensure repo deliverables directory is writable by container user (UID 1001)
if [ -d "./repos/$REPO" ]; then
mkdir -p "./repos/$REPO/deliverables"
chmod 777 "./repos/$REPO/deliverables"
fi
# Ensure containers are running (starts them if needed)
ensure_containers
# Build optional args
ARGS=""
[ -n "$CONFIG" ] && ARGS="$ARGS --config $CONFIG"
# Pass container path for output (where OUTPUT_DIR is mounted)
# Also pass display path so client can show the host path to user
if [ -n "$OUTPUT" ]; then
ARGS="$ARGS --output /app/output --display-output $OUTPUT"
fi
[ "$PIPELINE_TESTING" = "true" ] && ARGS="$ARGS --pipeline-testing"
[ -n "$WORKSPACE" ] && ARGS="$ARGS --workspace $WORKSPACE"
# Run the client to submit workflow
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE exec -T worker \
node dist/temporal/client.js "$URL" "$CONTAINER_REPO" $ARGS
}
cmd_logs() {
parse_args "$@"
if [ -z "$ID" ]; then
echo "ERROR: ID is required"
echo "Usage: ./shannon logs ID=<workflow-id>"
exit 1
fi
# Auto-discover the workflow log file
# 1. Check default location first
# 2. Search common output directories
# 3. Fall back to find command
WORKFLOW_LOG=""
if [ -f "./audit-logs/${ID}/workflow.log" ]; then
WORKFLOW_LOG="./audit-logs/${ID}/workflow.log"
else
# For resume workflow IDs (e.g. workspace_resume_123), check the original workspace
WORKSPACE_ID="${ID%%_resume_*}"
if [ "$WORKSPACE_ID" != "$ID" ] && [ -f "./audit-logs/${WORKSPACE_ID}/workflow.log" ]; then
WORKFLOW_LOG="./audit-logs/${WORKSPACE_ID}/workflow.log"
fi
# For named workspace IDs (e.g. workspace_shannon-123), check the workspace name
if [ -z "$WORKFLOW_LOG" ]; then
WORKSPACE_ID="${ID%%_shannon-*}"
if [ "$WORKSPACE_ID" != "$ID" ] && [ -f "./audit-logs/${WORKSPACE_ID}/workflow.log" ]; then
WORKFLOW_LOG="./audit-logs/${WORKSPACE_ID}/workflow.log"
fi
fi
if [ -z "$WORKFLOW_LOG" ]; then
# Search for the workflow directory (handles custom OUTPUT paths)
FOUND=$(find . -maxdepth 3 -path "*/${ID}/workflow.log" -type f 2>/dev/null | head -1)
if [ -n "$FOUND" ]; then
WORKFLOW_LOG="$FOUND"
fi
fi
fi
if [ -n "$WORKFLOW_LOG" ]; then
echo "Tailing workflow log: $WORKFLOW_LOG"
tail -f "$WORKFLOW_LOG"
else
echo "ERROR: Workflow log not found for ID: $ID"
echo ""
echo "Possible causes:"
echo " - Workflow hasn't started yet"
echo " - Workflow ID is incorrect"
echo ""
echo "Check the Temporal Web UI at http://localhost:8233 for workflow details"
exit 1
fi
}
cmd_workspaces() {
# Ensure containers are running (need worker to execute node)
ensure_containers
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE exec -T worker \
node dist/temporal/workspaces.js
}
cmd_stop() {
parse_args "$@"
if [ "$CLEAN" = "true" ]; then
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE --profile router down -v
else
docker compose -f "$COMPOSE_FILE" $COMPOSE_OVERRIDE --profile router down
fi
}
# Main command dispatch
case "${1:-help}" in
start)
shift
cmd_start "$@"
;;
logs)
shift
cmd_logs "$@"
;;
workspaces)
shift
cmd_workspaces
;;
stop)
shift
cmd_stop "$@"
;;
help|--help|-h|*)
show_help
;;
esac
#!/usr/bin/env node
process.env.SHANNON_LOCAL = '1';
import('./apps/cli/dist/index.mjs');