Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .skillspector-baseline.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
# See docs/SUPPRESSION.md for the full reference. All identifiers below are
# placeholders — replace them with your own rule ids, paths, and reasons.

version: 1
version: 2
scanner_version: "X.Y.Z" # generated automatically; do not edit

# Glob rules — human-authored, drift-tolerant (survive line/wording changes).
# A finding is suppressed when EVERY field a rule sets glob-matches it.
Expand All @@ -31,7 +32,7 @@ rules:
# Fingerprints — exact, machine-generated suppressions (one per accepted
# finding). Regenerate with `skillspector baseline` when a skill changes.
fingerprints:
- hash: "sha256:0123456789abcdef"
- hash: "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
rule_id: "SDI-2"
file: "example-skill/SKILL.md"
reason: "Accepted: reads its own environment ($EXAMPLE_TOKEN) for context"
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
### 2.5.0 (Friday, July 24, 2026)
### Features/Bug Fixes
* feat(report): add canonical inspection-ledger reporting, including JSON and SARIF execution-completeness status
* fix(cli): make recursive child-scan failures fail the combined command and JSON report
* fix(oss): exclude internal inspection-ledger plans and design documents from public snapshots

### 2.4.4 (Thursday, July 23, 2026)
### Features/Bug Fixes
* fix(anthropic): re-apply ANTHROPIC_BASE_URL override reverted by 2.4.3 snapshot (#301)
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,8 @@ skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-supp

A baseline can also use drift-tolerant glob rules (by rule id, file path, or
message) — see [`.skillspector-baseline.example.yaml`](.skillspector-baseline.example.yaml).
Exact fingerprint baselines are evidence-bound: changing the scanned source or
SkillSpector version keeps the finding active until it is reviewed again.

### LLM Analysis

Expand Down Expand Up @@ -560,6 +562,7 @@ Issues (2)
| `OPENAI_BASE_URL` | Override the OpenAI endpoint (e.g. point at Ollama). | Optional |
| `SKILLSPECTOR_REASONING_EFFORT` | Optional provider- and model-dependent reasoning-effort setting. Non-empty values are trimmed and passed through unchanged; unset or blank preserves provider-default behavior. | Optional |
| `ANTHROPIC_API_KEY` | Credential for the Anthropic provider (`SKILLSPECTOR_PROVIDER=anthropic`). | Required for LLM analysis when `SKILLSPECTOR_PROVIDER=anthropic` |
| `ANTHROPIC_BASE_URL` | Override the native Anthropic endpoint (default: `https://api.anthropic.com`). | Optional |
| `ANTHROPIC_PROXY_ENDPOINT_URL` | Full endpoint URL for the Anthropic proxy provider (Vertex-style raw-predict). | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
| `ANTHROPIC_PROXY_API_KEY` | Bearer token for the Anthropic proxy provider. | Required when `SKILLSPECTOR_PROVIDER=anthropic_proxy` |
| `ANTHROPIC_PROXY_API_VERSION` | `anthropic_version` value sent in the request body (default: `vertex-2023-10-16`). | Optional |
Expand Down
84 changes: 84 additions & 0 deletions contrib/batch_scan/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,47 @@ def sorted_results(results: list[dict[str, object]]) -> list[dict[str, object]]:
)


def _completeness(entry: dict[str, object]) -> dict[str, object]:
"""Return a child scan's public completeness projection, never its raw ledger."""
value = entry.get("analysis_completeness")
return value if isinstance(value, dict) else {}


def _inspection_summary(results: list[dict[str, object]]) -> dict[str, int]:
"""Aggregate public child completeness while keeping transport errors separate."""
completed_results = [result for result in results if not result.get("error")]
return {
"failed_executions": sum(
1 for result in completed_results if result.get("execution_successful") is False
),
"incomplete_skills": sum(
1 for result in completed_results if not _completeness(result).get("is_complete", True)
),
"partially_inspected_files": sum(
int(_completeness(result).get("partially_inspected_files", 0) or 0)
for result in completed_results
),
"entirely_uninspected_files": sum(
int(_completeness(result).get("entirely_uninspected_files", 0) or 0)
for result in completed_results
),
}


def _exception_groups(results: list[dict[str, object]]) -> list[tuple[str, list[dict[str, object]]]]:
"""Collect every public exception by child skill, without sampling rows."""
groups: list[tuple[str, list[dict[str, object]]]] = []
for result in sorted_results(results):
exceptions = _completeness(result).get("ledger_exceptions", [])
if not isinstance(exceptions, list) or not exceptions:
continue
safe_exceptions = [exception for exception in exceptions if isinstance(exception, dict)]
if safe_exceptions:
name = str(result.get("skill", {}).get("name", "unknown"))
groups.append((name, safe_exceptions))
return groups


# ═══════════════════════════════════════════════════════════════════
# Terminal (Rich)
# ═══════════════════════════════════════════════════════════════════
Expand Down Expand Up @@ -85,6 +126,14 @@ def _format_terminal(results: list[dict[str, object]]) -> str:
capture.print(f"[bold]Total:[/bold] {total} skill(s) scanned")
if errs:
capture.print(f"[red]Errors:[/red] {errs}")
inspection = _inspection_summary(results)
capture.print(
"[bold]Inspection:[/bold] "
f"{inspection['failed_executions']} failed execution(s), "
f"{inspection['incomplete_skills']} incomplete skill(s), "
f"{inspection['partially_inspected_files']} partial file(s), "
f"{inspection['entirely_uninspected_files']} entirely uninspected file(s)"
)
if non_en:
capture.print(
f"[bold]Multilingual:[/bold] {non_en} non-English skill(s) "
Expand Down Expand Up @@ -157,6 +206,14 @@ def _format_terminal(results: list[dict[str, object]]) -> str:
capture.print(
f"[green]{low_count} skill(s)[/green] with LOW risk — likely safe"
)
for skill_name, exceptions in _exception_groups(results):
capture.print(f"[bold]Ledger exceptions — {skill_name}[/bold]")
for exception in exceptions:
capture.print(
" - "
f"{exception.get('reason_code', 'unknown')} "
f"{exception.get('path', '')}: {exception.get('message', '')}"
)
capture.print()

return capture.export_text()
Expand Down Expand Up @@ -235,6 +292,13 @@ def _format_terminal_plain(results: list[dict[str, object]]) -> str:
f" {skill.get('name', '?'):40s} "
f"{risk.get('score', 0):>3}/100 {risk.get('severity', 'LOW'):<8s}"
)
for skill_name, exceptions in _exception_groups(results):
lines.append(f"Ledger exceptions — {skill_name}")
for exception in exceptions:
lines.append(
f" - {exception.get('reason_code', 'unknown')} "
f"{exception.get('path', '')}: {exception.get('message', '')}"
)
return "\n".join(lines)


Expand All @@ -258,6 +322,8 @@ def _format_json(results: list[dict[str, object]]) -> str:
"risk_assessment": r.get("risk_assessment", {}),
"components": r.get("components", []),
"issues": r.get("issues", []),
"execution_successful": r.get("execution_successful", "error" not in r),
"analysis_completeness": _completeness(r),
"scan_mode": r.get("scan_mode", "multilingual-enhanced"),
"enhancements": r.get("enhancements", {}),
}
Expand Down Expand Up @@ -292,6 +358,7 @@ def _format_json(results: list[dict[str, object]]) -> str:
"gap_fill_applied": gap_fill_skills,
"gap_fill_findings": gap_fill_total,
},
"inspection_completeness": _inspection_summary(results),
},
"skills": entries,
"metadata": {
Expand Down Expand Up @@ -351,6 +418,11 @@ def _format_markdown(results: list[dict[str, object]]) -> str:
lines.append(f"| 🔴 HIGH | {high} |")
lines.append(f"| 🟡 MEDIUM | {medium} |")
lines.append(f"| 🟢 LOW | {low_count} |")
inspection = _inspection_summary(results)
lines.append(f"| Failed executions | {inspection['failed_executions']} |")
lines.append(f"| Incomplete skills | {inspection['incomplete_skills']} |")
lines.append(f"| Partially inspected files | {inspection['partially_inspected_files']} |")
lines.append(f"| Entirely uninspected files | {inspection['entirely_uninspected_files']} |")
lines.append("")

lines.append("## Skills by Risk Score\n")
Expand Down Expand Up @@ -408,5 +480,17 @@ def _format_markdown(results: list[dict[str, object]]) -> str:
lines.append("")
lines.append("")

exception_groups = _exception_groups(results)
if exception_groups:
lines.append("## Ledger Exceptions\n")
for skill_name, exceptions in exception_groups:
lines.append(f"### {skill_name}\n")
for exception in exceptions:
lines.append(
f"- **{exception.get('reason_code', 'unknown')}** "
f"`{exception.get('path', '')}`: {exception.get('message', '')}"
)
lines.append("")

lines.append(f"\n*Generated by SkillSpector v{_skillspector_version}*")
return "\n".join(lines)
2 changes: 2 additions & 0 deletions contrib/batch_scan/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,8 @@ def entry_from_result(
for c in component_metadata # type: ignore[union-attr]
],
"issues": issues,
"analysis_completeness": result.get("analysis_completeness") or {},
"execution_successful": bool(result.get("execution_successful", True)),
"scan_mode": "multilingual-enhanced",
"enhancements": {
"gap_fill_applied": gap_fill_applied,
Expand Down
66 changes: 66 additions & 0 deletions contrib/batch_scan/tests/test_inspection_reporting.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Batch propagation tests for canonical inspection completeness."""

from __future__ import annotations

import json
from pathlib import Path

from contrib.batch_scan.reports import _format_json, _format_markdown, _format_terminal
from contrib.batch_scan.runner import entry_from_result


def test_entry_from_result_preserves_analysis_completeness(tmp_path: Path) -> None:
completeness = {
"execution_successful": False,
"ledger_exceptions": [{"reason_code": "read_error", "path": "x.py"}],
"scope_exclusions": [],
"analyzer_statuses": [],
}
entry = entry_from_result(
{
"analysis_completeness": completeness,
"execution_successful": False,
"risk_score": 0,
"risk_severity": "LOW",
"risk_recommendation": "CAUTION",
"component_metadata": [],
"manifest": {"name": "broken"},
"filtered_findings": [],
},
tmp_path,
tmp_path,
)

assert entry["analysis_completeness"] == completeness
assert entry["execution_successful"] is False


def test_batch_formats_preserve_every_child_ledger_exception() -> None:
entry = {
"skill": {"name": "ledger-skill", "language": "en"},
"risk_assessment": {"score": 0, "severity": "LOW", "recommendation": "CAUTION"},
"components": [],
"issues": [{"id": "P1", "finding_id": "finding-batch-1"}],
"analysis_completeness": {
"execution_successful": False,
"ledger_exceptions": [
{"reason_code": "read_error", "path": "a.py", "message": "could not read"},
{"reason_code": "syntax_error", "path": "b.py", "message": "could not parse"},
],
"scope_exclusions": [],
"analyzer_statuses": [],
},
"execution_successful": False,
}

payload = json.loads(_format_json([entry]))
exceptions = payload["skills"][0]["analysis_completeness"]["ledger_exceptions"]
assert [item["reason_code"] for item in exceptions] == ["read_error", "syntax_error"]
assert payload["skills"][0]["issues"][0]["finding_id"] == "finding-batch-1"

for rendered in (_format_terminal([entry]), _format_markdown([entry])):
assert "read_error" in rendered
assert "syntax_error" in rendered
10 changes: 5 additions & 5 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ All targets assume the virtual environment is **already created and activated**.
| `show_suppressed` | When True, baseline-suppressed findings are listed in the report (still excluded from the risk score) |
| `suppressed_findings` | List of `SuppressedFinding` (finding + reason) produced by the report node |
| `findings` | All raw findings from analyzers (reducer: `operator.add`) |
| `filtered_findings` | Findings after meta_analyzer |
| `filtered_findings` | Report-stage compatibility projection selected from `effective_finding_ids` |
| `model_config` | Optional model IDs per node (e.g. default, meta_analyzer) |
| `risk_severity` | Severity band from risk score: LOW, MEDIUM, HIGH, CRITICAL |
| `risk_recommendation` | SAFE, CAUTION, or DO_NOT_INSTALL (from report node) |
Expand Down Expand Up @@ -128,7 +128,7 @@ There are no conditional edges: after `resolve_input` → `build_context`, all a
| **resolve_input** | Consumes `input_path` or `skill_path`; resolves URLs/zips/files via InputHandler; sets `skill_path` and (when needed) `temp_dir_for_cleanup` | [resolve_input.py](../src/skillspector/nodes/resolve_input.py) |
| **build_context** | Reads `skill_path`, populates `components`, `file_cache`, `ast_cache`, `manifest`, `component_metadata`, `has_executable_scripts` | [build_context.py](../src/skillspector/nodes/build_context.py) |
| **Analyzers** | 22 nodes; each returns `AnalyzerNodeResponse` (list of `Finding`). State reducer appends to `findings`. | [nodes/analyzers/__init__.py](../src/skillspector/nodes/analyzers/__init__.py) (`ANALYZER_NODE_IDS`, `ANALYZER_NODES`) |
| **meta_analyzer** | Per-file LLM filter/enrich of `findings` → `filtered_findings` via `LLMMetaAnalyzer`; one LLM call per file (or per chunk for oversized files); token budgets from `constants.py`; falls back when `use_llm` is False | [meta_analyzer.py](../src/skillspector/nodes/meta_analyzer.py), [llm_analyzer_base.py](../src/skillspector/nodes/llm_analyzer_base.py) |
| **meta_analyzer** | Per-file LLM filter/enrich of canonical `findings`; emits ordered `effective_finding_ids` for report selection. One LLM call per file (or per chunk for oversized files); token budgets from `constants.py`; falls back when `use_llm` is False. | [meta_analyzer.py](../src/skillspector/nodes/meta_analyzer.py), [llm_analyzer_base.py](../src/skillspector/nodes/llm_analyzer_base.py) |
| **report** | Applies baseline suppression (`state["baseline"]`), then builds SARIF 2.1.0, computes `risk_score`, `risk_severity`, `risk_recommendation` from the non-suppressed findings; writes `report_body` from `output_format` (terminal/json/markdown/sarif) | [report.py](../src/skillspector/nodes/report.py) |

---
Expand All @@ -145,7 +145,7 @@ There are no conditional edges: after `resolve_input` → `build_context`, all a
| `llm_utils.py` | `chat_completion()` for OpenAI-compatible / NVIDIA Inference API |
| `cli.py` | Typer app: `scan` (with input resolution, `--format`, `--no-llm`), `--version` |
| `input_handler.py` | Resolves Git URL, file URL, .zip, single file, or directory to a local directory path |
| `suppression.py` | Baseline / false-positive suppression: `Baseline`, `SuppressionRule`, `load_baseline`, `partition_findings`, `finding_fingerprint`, `build_baseline_dict` (see [SUPPRESSION.md](SUPPRESSION.md)) |
| `suppression.py` | Baseline / false-positive suppression: `Baseline`, `SuppressionRule`, `load_baseline`, `partition_findings`, `finding_fingerprint`, `build_baseline_dict`; exact v2 fingerprints require the scanner version and source `file_cache` (see [SUPPRESSION.md](SUPPRESSION.md)) |
| `__init__.py` | Package version (from pyproject.toml via `importlib.metadata`) |
| `sarif_models.py` | SARIF 2.1.0 Pydantic models and `validate_sarif_report()` |
| **nodes/** | |
Expand Down Expand Up @@ -207,7 +207,7 @@ result = graph.invoke({
# Or: graph.stream(...)
```

Optional state keys: `mode`, `model_config`, `output_format`, `use_llm`. The result includes `findings`, `filtered_findings`, `sarif_report`, `risk_score`, `risk_severity`, `risk_recommendation`, and `report_body` (formatted string for the requested `output_format`).
Optional state keys: `mode`, `model_config`, `output_format`, `use_llm`. The final report result includes canonical `findings`, the report-projected `filtered_findings`, `sarif_report`, `risk_score`, `risk_severity`, `risk_recommendation`, and `report_body` (formatted string for the requested `output_format`).

---

Expand Down Expand Up @@ -255,7 +255,7 @@ block a merge request.

- **Finding** ([models.py](../src/skillspector/models.py)): `rule_id`, `message`, `severity`, `confidence`, `file`, `start_line`, `end_line`, `category`, `pattern`, `finding`, `explanation`, `remediation`, `code_snippet`, `intent`, `tags`, `context`, `matched_text`. This is the type stored in state and used in SARIF and JSON report output.
- **AnalyzerFinding**: Analyzer-facing type with `Location` and `Severity` enum. Convert to `Finding` via [static_runner.analyzer_finding_to_finding](../src/skillspector/nodes/analyzers/static_runner.py) (or equivalent).
- **SARIF**: [sarif_models.py](../src/skillspector/sarif_models.py) provides Pydantic models for SARIF 2.1.0. The report node builds a `SarifLog` from `filtered_findings`.
- **SARIF**: [sarif_models.py](../src/skillspector/sarif_models.py) provides Pydantic models for SARIF 2.1.0. The report node builds a `SarifLog` from its effective-ID-selected findings.

---

Expand Down
Loading
Loading