Skip to content

skill-security-scan verdicts are nondeterministic between PR and merge, silently failing main builds on approved PRs #940

Description

@danbarr

Summary

The skill security scan can pass on a PR and then fail with CRITICAL findings when the identical content is rescanned on the push to main. The scan inputs are fully deterministic in these cases (same skill content hash, same skill-scanner version, same ATR ruleset), but the final verdict depends on the scanner's LLM meta-analyzer ("false positive filtering"), which adjudicates each raw pattern hit and does not produce stable results across runs.

Because the failure lands on the merge commit's Build Skill Artifacts run rather than on the PR, it's easy to miss: the PR shows green, gets merged, and main quietly stops publishing that skill until someone notices and re-runs the job (which usually passes, further hiding the pattern).

Evidence: two identical-input flips on 2026-08-31

mongodb-mcp-setup (PR #924, merge run 33443302315):

  • Fresh PR scan (00:36 UTC): Meta-analysis complete: 63 false positives removed, 1 findings retained, passed with one MEDIUM finding.
  • Fresh merge scan (21:52 UTC): Meta-analysis complete: 62 false positives removed, 2 findings retained, failed on [ATR_2026_00115] (CRITICAL) Pattern detected: env | (SKILL.md:33).
  • Verified identical inputs: both scans used skill-scanner 2.0.13, loaded byte-identical rulesets (all 470 Rule ATR_* log lines match, including ATR_2026_00115), and scanned the same content (same upstream digest 47cc461 and spec hash; the scanner bump to 4.8.4 in chore(deps): update dependency cisco-ai-mcp-scanner to v4.8.4 #934 merged 5 minutes after this run). The pattern rule fired both times; only the meta-analyzer's FP judgment differed.

vercel-cli-with-tokens (PR #929, merge run 33443330327):

  • Fresh PR scan (00:37 UTC): 58 false positives removed, 2 findings retained (both MEDIUM), passed.
  • Fresh merge scan (21:59 UTC): 53 false positives removed, 8 findings retained, failed on four [ATR_2026_00063] (CRITICAL) Pattern detected: .env hits (SKILL.md:20, 34, 70, 310).

Re-running the failed merge jobs for both skills subsequently passed with no changes to content, scanner, or ruleset. The same commit fails or passes depending on nothing but the run.

Why main always rescans

The scan-result cache (skill-scan-v1-<skill>-<repo>-<digest>-<spec hash>) works as designed within a PR: the initial commit scans fresh and saves the cache, and the versioning workflow's follow-up commit replays it. But GitHub-hosted caches saved on a PR branch are not readable by push runs on main, so every merge re-runs the scanner from scratch and re-rolls the meta-analyzer, even when the content hash is identical to what the PR gate already approved.

This has been failing main for a while

Failed Build Skill Artifacts runs on main where the failing job was skill-security-scan on a merge of a PR that had passed:

Options

The findings themselves aren't hallucinated; the pattern hits are real and repeatable (env | grep does print values, the vercel skill does echo token-bearing .env content). The instability is entirely in whether the meta-analyzer classifies a borderline hit as a false positive, so findings near that boundary will keep coin-flipping on every fresh scan. Possible directions, not mutually exclusive:

  1. Inherit the PR verdict on main. If the merge commit's skill content hash matches what the PR scan approved, skip the rescan (or run it report-only). Deterministic gate, but main never gets an independent check, and ruleset updates would only apply to skills the next time their content changes.
  2. Take the meta-analyzer out of the blocking path. Block on raw findings plus the allowlist, and treat meta-analysis as advisory annotation. Fully deterministic, but requires triaging/allowlisting the existing borderline pattern hits up front.
  3. Persist adjudications. Once a finding (rule + location + content hash) has been judged a false positive, record that so later scans of identical content don't re-adjudicate it.

Near-term there's no action strictly required: re-runs cleared both failures. But that's the problem in miniature, and the same borderline hits (ATR_2026_00115 in mongodb-mcp-setup, ATR_2026_00063 in vercel-cli-with-tokens) will keep coin-flipping on every fresh scan until they're either allowlisted, fixed upstream, or the blocking path is made deterministic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIssue needs initial triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions