You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The skill security scan can pass on a PR and then fail with CRITICAL findings when the identical content is rescanned on the push to main. The scan inputs are fully deterministic in these cases (same skill content hash, same skill-scanner version, same ATR ruleset), but the final verdict depends on the scanner's LLM meta-analyzer ("false positive filtering"), which adjudicates each raw pattern hit and does not produce stable results across runs.
Because the failure lands on the merge commit's Build Skill Artifacts run rather than on the PR, it's easy to miss: the PR shows green, gets merged, and main quietly stops publishing that skill until someone notices and re-runs the job (which usually passes, further hiding the pattern).
Verified identical inputs: both scans used skill-scanner 2.0.13, loaded byte-identical rulesets (all 470 Rule ATR_* log lines match, including ATR_2026_00115), and scanned the same content (same upstream digest 47cc461 and spec hash; the scanner bump to 4.8.4 in chore(deps): update dependency cisco-ai-mcp-scanner to v4.8.4 #934 merged 5 minutes after this run). The pattern rule fired both times; only the meta-analyzer's FP judgment differed.
Re-running the failed merge jobs for both skills subsequently passed with no changes to content, scanner, or ruleset. The same commit fails or passes depending on nothing but the run.
Why main always rescans
The scan-result cache (skill-scan-v1-<skill>-<repo>-<digest>-<spec hash>) works as designed within a PR: the initial commit scans fresh and saves the cache, and the versioning workflow's follow-up commit replays it. But GitHub-hosted caches saved on a PR branch are not readable by push runs on main, so every merge re-runs the scanner from scratch and re-rolls the meta-analyzer, even when the content hash is identical to what the PR gate already approved.
This has been failing main for a while
Failed Build Skill Artifacts runs on main where the failing job was skill-security-scan on a merge of a PR that had passed:
The findings themselves aren't hallucinated; the pattern hits are real and repeatable (env | grep does print values, the vercel skill does echo token-bearing .env content). The instability is entirely in whether the meta-analyzer classifies a borderline hit as a false positive, so findings near that boundary will keep coin-flipping on every fresh scan. Possible directions, not mutually exclusive:
Inherit the PR verdict on main. If the merge commit's skill content hash matches what the PR scan approved, skip the rescan (or run it report-only). Deterministic gate, but main never gets an independent check, and ruleset updates would only apply to skills the next time their content changes.
Take the meta-analyzer out of the blocking path. Block on raw findings plus the allowlist, and treat meta-analysis as advisory annotation. Fully deterministic, but requires triaging/allowlisting the existing borderline pattern hits up front.
Persist adjudications. Once a finding (rule + location + content hash) has been judged a false positive, record that so later scans of identical content don't re-adjudicate it.
Near-term there's no action strictly required: re-runs cleared both failures. But that's the problem in miniature, and the same borderline hits (ATR_2026_00115 in mongodb-mcp-setup, ATR_2026_00063 in vercel-cli-with-tokens) will keep coin-flipping on every fresh scan until they're either allowlisted, fixed upstream, or the blocking path is made deterministic.
Summary
The skill security scan can pass on a PR and then fail with CRITICAL findings when the identical content is rescanned on the push to
main. The scan inputs are fully deterministic in these cases (same skill content hash, sameskill-scannerversion, same ATR ruleset), but the final verdict depends on the scanner's LLM meta-analyzer ("false positive filtering"), which adjudicates each raw pattern hit and does not produce stable results across runs.Because the failure lands on the merge commit's
Build Skill Artifactsrun rather than on the PR, it's easy to miss: the PR shows green, gets merged, and main quietly stops publishing that skill until someone notices and re-runs the job (which usually passes, further hiding the pattern).Evidence: two identical-input flips on 2026-08-31
mongodb-mcp-setup (PR #924, merge run 33443302315):
Meta-analysis complete: 63 false positives removed, 1 findings retained, passed with one MEDIUM finding.Meta-analysis complete: 62 false positives removed, 2 findings retained, failed on[ATR_2026_00115] (CRITICAL) Pattern detected: env | (SKILL.md:33).Rule ATR_*log lines match, including ATR_2026_00115), and scanned the same content (same upstream digest47cc461and spec hash; the scanner bump to 4.8.4 in chore(deps): update dependency cisco-ai-mcp-scanner to v4.8.4 #934 merged 5 minutes after this run). The pattern rule fired both times; only the meta-analyzer's FP judgment differed.vercel-cli-with-tokens (PR #929, merge run 33443330327):
58 false positives removed, 2 findings retained(both MEDIUM), passed.53 false positives removed, 8 findings retained, failed on four[ATR_2026_00063] (CRITICAL) Pattern detected: .envhits (SKILL.md:20, 34, 70, 310).Re-running the failed merge jobs for both skills subsequently passed with no changes to content, scanner, or ruleset. The same commit fails or passes depending on nothing but the run.
Why main always rescans
The scan-result cache (
skill-scan-v1-<skill>-<repo>-<digest>-<spec hash>) works as designed within a PR: the initial commit scans fresh and saves the cache, and the versioning workflow's follow-up commit replays it. But GitHub-hosted caches saved on a PR branch are not readable by push runs onmain, so every merge re-runs the scanner from scratch and re-rolls the meta-analyzer, even when the content hash is identical to what the PR gate already approved.This has been failing main for a while
Failed
Build Skill Artifactsruns on main where the failing job wasskill-security-scanon a merge of a PR that had passed:Options
The findings themselves aren't hallucinated; the pattern hits are real and repeatable (
env | grepdoes print values, the vercel skill does echo token-bearing.envcontent). The instability is entirely in whether the meta-analyzer classifies a borderline hit as a false positive, so findings near that boundary will keep coin-flipping on every fresh scan. Possible directions, not mutually exclusive:Near-term there's no action strictly required: re-runs cleared both failures. But that's the problem in miniature, and the same borderline hits (ATR_2026_00115 in mongodb-mcp-setup, ATR_2026_00063 in vercel-cli-with-tokens) will keep coin-flipping on every fresh scan until they're either allowlisted, fixed upstream, or the blocking path is made deterministic.