feat: backport config-driven run scoping and report filtering
Cherry-pick of upstream Shannon PR #326. Adds vuln_classes subset selection, exploit toggle, code_path avoid enforcement via SDK deny rules, deterministic findings rendering when exploit is disabled, report filtering (min_severity, min_confidence, guidance), and rules_of_engagement config field. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
@@ -105,6 +105,8 @@ If you only confirmed a vulnerability without attempting to weaponize it, you ha
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<starting_context>
|
||||
- Your **actionable exploitation queue** is located at `.shannon/deliverables/auth_exploitation_queue.json`. This is your to-do list, containing the precise targets identified by the analysis phase. You MUST process every vulnerability in this queue.
|
||||
|
||||
|
||||
@@ -92,6 +92,8 @@ After exhaustive bypass attempts, determine:
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<starting_context>
|
||||
- Your **actionable exploitation queue** is located at `.shannon/deliverables/authz_exploitation_queue.json`. This is your to-do list, containing the precise targets identified by the analysis phase. You MUST process every vulnerability in this queue.
|
||||
|
||||
|
||||
@@ -69,6 +69,8 @@ Remember: An unproven vulnerability is worse than no finding at all - it wastes
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<starting_context>
|
||||
- Your **actionable exploitation queue** is located at `.shannon/deliverables/injection_exploitation_queue.json`. This is your to-do list, containing the precise targets identified by the analysis phase. You MUST process every vulnerability in this queue.
|
||||
|
||||
|
||||
@@ -92,6 +92,8 @@ After exhaustive bypass attempts, determine:
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<starting_context>
|
||||
- Your **actionable exploitation queue** is located at `.shannon/deliverables/ssrf_exploitation_queue.json`. This is your to-do list, containing the precise targets identified by the analysis phase. You MUST process every vulnerability in this queue.
|
||||
|
||||
|
||||
@@ -90,6 +90,8 @@ After exhaustive bypass attempts, determine:
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<starting_context>
|
||||
- Your **actionable exploitation queue** is located at `.shannon/deliverables/xss_exploitation_queue.json`. This is your to-do list, containing the precise targets identified by the analysis phase. You MUST process every vulnerability in this queue.
|
||||
|
||||
|
||||
@@ -31,6 +31,14 @@ Read `.gitignore` and run `git ls-files --others --ignored --exclude-standard --
|
||||
|
||||
{{DESCRIPTION}}
|
||||
|
||||
<scope>
|
||||
Downstream vulnerability analysis will cover these classes: {{VULN_CLASSES_TESTED}}.
|
||||
</scope>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<system_architecture>
|
||||
**PENTESTING WORKFLOW - YOUR POSITION:**
|
||||
|
||||
|
||||
@@ -20,6 +20,10 @@ Filesystem:
|
||||
- {{REPO_PATH}}/.shannon/scratchpad/ (read-write) - screenshots, scripts, scratch work, etc.
|
||||
</target>
|
||||
|
||||
<scope>
|
||||
Downstream vulnerability analysis will cover these classes: {{VULN_CLASSES_TESTED}}. Map only what supports these classes.
|
||||
</scope>
|
||||
|
||||
<rules>
|
||||
Rules to Avoid:
|
||||
{{RULES_AVOID}}
|
||||
@@ -28,10 +32,14 @@ Areas to Focus On:
|
||||
{{RULES_FOCUS}}
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<scope_boundaries>
|
||||
# Penetration Test Scope & Boundaries
|
||||
|
||||
|
||||
@@ -9,11 +9,11 @@ Technical leadership (CTOs, CISOs, Engineering VPs) who need both technical accu
|
||||
</audience>
|
||||
|
||||
<objective>
|
||||
The orchestrator has already concatenated all specialist reports into `comprehensive_security_assessment_report.md`.
|
||||
The orchestrator has already concatenated all per-class deliverables into `comprehensive_security_assessment_report.md`. Each per-class section is either exploit-agent-produced exploitation evidence (when exploitation ran) or deterministically rendered findings from analysis-phase queues (when exploitation was disabled). The cleanup rules below apply uniformly to either source.
|
||||
Your task is to:
|
||||
1. Read this existing concatenated report
|
||||
2. Add an Executive Summary (vulnerability overview) at the top
|
||||
3. Clean up ALL exploitation evidence sections by removing hallucinated content
|
||||
2. Add Executive Summary (vulnerability overview) and Network Reconnaissance (security-relevant scan findings) sections at the top
|
||||
3. Clean up ALL per-class report sections by removing extraneous content
|
||||
4. Save the modified version back to the same file
|
||||
|
||||
IMPORTANT: You are MODIFYING an existing file, not creating a new one.
|
||||
@@ -32,8 +32,13 @@ Filesystem:
|
||||
<context>
|
||||
Authentication Context:
|
||||
{{AUTH_CONTEXT}}
|
||||
|
||||
Vulnerability classes tested in this run: {{VULN_CLASSES_TESTED}}
|
||||
Exploitation: {{EXPLOITATION}}
|
||||
</context>
|
||||
|
||||
{{REPORT_FILTERS_BLOCK}}
|
||||
|
||||
<input_files>
|
||||
You will analyze the following deliverable files:
|
||||
- `.shannon/deliverables/pre_recon_deliverable.md` - Initial reconnaissance and technology stack
|
||||
@@ -51,34 +56,31 @@ MODIFY the existing file `.shannon/deliverables/comprehensive_security_assessmen
|
||||
## Executive Summary
|
||||
- Target: {{WEB_URL}}
|
||||
- Assessment Date: {current date}
|
||||
- Scope: Authentication, XSS, SQL and Command Injection, SSRF, Authorization testing
|
||||
- Scope: {{VULN_CLASSES_TESTED}}
|
||||
- Exploitation: {{EXPLOITATION}}
|
||||
|
||||
## Summary by Vulnerability Type
|
||||
|
||||
{For each vulnerability type below, examine the comprehensive_security_assessment_report.md sections and provide a summary. If no section exists for that type or no vulnerabilities are listed, explicitly state that none were found.}
|
||||
|
||||
**Authentication Vulnerabilities:**
|
||||
{Check for "Authentication Exploitation Evidence" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No authentication vulnerabilities were found."}
|
||||
{{VULN_SUMMARY_SUBSECTIONS}}
|
||||
|
||||
**Authorization Vulnerabilities:**
|
||||
{Check for "Authorization Exploitation Evidence" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No authorization vulnerabilities were found."}
|
||||
## Network Reconnaissance
|
||||
{Extract security-relevant findings from automated scanning tools:
|
||||
- Open ports and exposed services from nmap
|
||||
- Subdomain discoveries from subfinder that could expand attack surface
|
||||
- Security headers or misconfigurations detected by whatweb
|
||||
- Any other security-relevant findings from the automated tools
|
||||
SKIP stack details - technical leaders know their infrastructure}
|
||||
|
||||
**Cross-Site Scripting (XSS) Vulnerabilities:**
|
||||
{Check for "XSS Exploitation Evidence" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No XSS vulnerabilities were found."}
|
||||
|
||||
**SQL/Command Injection Vulnerabilities:**
|
||||
{Check for "Injection Exploitation Evidence" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No SQL or command injection vulnerabilities were found."}
|
||||
|
||||
**Server-Side Request Forgery (SSRF) Vulnerabilities:**
|
||||
{Check for "SSRF Exploitation Evidence" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No SSRF vulnerabilities were found."}
|
||||
|
||||
2. KEEPING the existing exploitation evidence sections but CLEANING them according to the rules below
|
||||
2. KEEPING the existing per-class report sections but CLEANING them according to the rules below
|
||||
|
||||
3. The final structure should be:
|
||||
- Executive Summary (new)
|
||||
- All existing exploitation evidence sections (cleaned)
|
||||
- Network Reconnaissance (new)
|
||||
- All existing per-class report sections (cleaned)
|
||||
|
||||
IMPORTANT: Do NOT reorder the existing exploitation evidence sections. Maintain the exact order they appear in the concatenated report. Only remove sections that do not match the defined criteria above.
|
||||
IMPORTANT: Do NOT reorder the existing per-class report sections. Maintain the exact order they appear in the concatenated report. Only remove sections that do not match the defined criteria above.
|
||||
|
||||
</deliverable_instructions>
|
||||
|
||||
@@ -91,11 +93,12 @@ IMPORTANT: Do NOT reorder the existing exploitation evidence sections. Maintain
|
||||
2. Create the Executive Summary content:
|
||||
- Executive Summary: Technical overview with actionable findings for engineering leaders
|
||||
|
||||
3. Clean the exploitation evidence sections from `.shannon/deliverables/comprehensive_security_assessment_report.md` by applying these rules:
|
||||
3. Clean the per-class report sections in `.shannon/deliverables/comprehensive_security_assessment_report.md` by applying these rules:
|
||||
- KEEP these specific section headings:
|
||||
NOTE: these sections will contain vulnerability lists with IDs matching pattern `### [TYPE]-VULN-[NUMBER]`
|
||||
* `# [Type] Exploitation Evidence`
|
||||
* `## Successfully Exploited Vulnerabilities`
|
||||
* `# [Type] {{REPORT_VULN_HEADING}}`
|
||||
* `## {{REPORT_VULN_SUBHEADING}}`
|
||||
{{REPORT_FILTER_RULES}}
|
||||
- REMOVE ANY OTHER SECTIONS (even if they contain vulnerability IDs), such as:
|
||||
* `## Potential Vulnerabilities (Validation Blocked)` (All agents)
|
||||
* Standalone "Recommendations" sections
|
||||
@@ -107,11 +110,11 @@ IMPORTANT: Do NOT reorder the existing exploitation evidence sections. Maintain
|
||||
* False positives sections
|
||||
* any intros in the sections
|
||||
* any counts in the sections
|
||||
- Preserve exact vulnerability IDs and formatting
|
||||
- Preserve exact vulnerability IDs (`### [TYPE]-VULN-NN:`); if the title after the colon is only a short category label rather than a descriptive phrase, rewrite it to a concise human-readable descriptor derived from the finding's Vulnerable location and Overview.
|
||||
|
||||
4. Combine the content:
|
||||
- Place the Executive Summary and Network Reconnaissance sections at the top
|
||||
- Follow with the cleaned exploitation evidence sections
|
||||
- Follow with the cleaned per-class report sections
|
||||
- Save as the modified `.shannon/deliverables/comprehensive_security_assessment_report.md`
|
||||
|
||||
CRITICAL: You are modifying the existing concatenated report at `.shannon/deliverables/comprehensive_security_assessment_report.md` IN-PLACE, not creating a separate file.
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
<code_path_rules>
|
||||
Source-code routing. Each rule is tagged `[FILE]` (literal path) or `[GLOB]` (pattern). All paths are repository-relative.
|
||||
|
||||
How to apply (focus rules):
|
||||
- For `[FILE]` entries — delegate analysis to the Task tool.
|
||||
- For `[GLOB]` entries — invoke the Glob tool to enumerate matches, then delegate analysis of every match to the Task tool.
|
||||
|
||||
Avoid — out of scope. Skip entirely; the tool layer will block any access attempts.
|
||||
{{CODE_RULES_AVOID}}
|
||||
|
||||
Focus — priority work assignments. Analyze every entry.
|
||||
{{CODE_RULES_FOCUS}}
|
||||
</code_path_rules>
|
||||
@@ -0,0 +1,3 @@
|
||||
<rules_of_engagement>
|
||||
{{RULES_OF_ENGAGEMENT}}
|
||||
</rules_of_engagement>
|
||||
@@ -20,10 +20,14 @@ Success criterion: A complete, code-backed analysis of every potential authentic
|
||||
@include(shared/_rules.txt)
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<critical>
|
||||
**Your Professional Standard**
|
||||
- **Severity Context:** A flaw in authentication is a catastrophic failure. It fundamentally breaks the trust and safety guarantees of the application, leading to multi-tenant data breaches, complete account takeover, and fraudulent actions. These are the most critical vulnerabilities you can find.
|
||||
|
||||
@@ -19,10 +19,14 @@ Success criterion: A complete, code-backed analysis of every potential authoriza
|
||||
@include(shared/_rules.txt)
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<critical>
|
||||
**Your Professional Standard**
|
||||
- **Severity Context:** A flaw in authorization is a critical security failure. It allows privilege escalation, horizontal and vertical access control bypass, and unauthorized data access. These vulnerabilities enable attackers to access resources they shouldn't, escalate privileges, and compromise multi-tenant data isolation.
|
||||
|
||||
@@ -20,10 +20,14 @@ Success criterion: Complete source-to-sink traces detailing path, sanitizers, si
|
||||
@include(shared/_rules.txt)
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<critical>
|
||||
**Your Professional Standard**
|
||||
- **Severity Context:** A structural flaw in a backend command, whether an SQL query or a shell command, is one of the most severe classes of vulnerability in a SaaS application. It undermines the foundational trust of the system by creating the *potential* for data exposure (SQLi) or direct server compromise (Command Injection).
|
||||
|
||||
@@ -19,10 +19,14 @@ Success criterion: A complete source-to-sink trace for every identified SSRF vul
|
||||
@include(shared/_rules.txt)
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<critical>
|
||||
**Your Professional Standard**
|
||||
- **Severity Context:** An SSRF flaw is a breach of network segmentation and trust boundaries. It allows attackers to leverage the application server as a proxy to access internal services, cloud metadata endpoints, or perform reconnaissance of internal networks. These vulnerabilities can lead to data exposure, privilege escalation, and complete network compromise.
|
||||
|
||||
@@ -19,10 +19,14 @@ Success criterion: Live confirmation of XSS execution for every vulnerability th
|
||||
@include(shared/_rules.txt)
|
||||
</rules>
|
||||
|
||||
@include(shared/_code-path-rules.txt)
|
||||
|
||||
<login_instructions>
|
||||
{{LOGIN_INSTRUCTIONS}}
|
||||
</login_instructions>
|
||||
|
||||
@include(shared/_rules-of-engagement.txt)
|
||||
|
||||
<critical>
|
||||
**Your Professional Standard**
|
||||
- Severity Context: An XSS flaw is a breach of trust between the user and the application. It grants an attacker the ability to execute arbitrary code within a victim's browser, leading to session hijacking, credential theft, and complete compromise of the user's account and data within the application.
|
||||
|
||||
Reference in New Issue
Block a user