Skip to content

perf(codex): memoize Gatekeeper verdicts for standalone CLI binaries - #4080

Closed
dustball wants to merge 6 commits into
steipete:mainfrom
dustball:fix/codex-preflight-assessment-memo
Closed

dustball wants to merge 6 commits into
steipete:mainfrom
dustball:fix/codex-preflight-assessment-memo

Conversation

@dustball

@dustball dustball commented Sep 28, 2026 •

Copy link
Copy Markdown

Fixes #4078.

Summary

Every Codex binary lookup runs CodexLaunchPreflight, which spawns spctl --assess on the native codex executable. Nothing caches the verdict, and Gatekeeper does not cache a rejected (the code is valid but does not seem to be an app) result either, so every lookup re-hashes the whole binary in syspolicyd. On an Intel Mac that is ~2.5 s of CPU per call for the 281 MB x86_64 standalone CLI: ~1.7% of a core at a 5-minute refresh, and ~55% of a core during a 52-minute stretch when lookups ran every ~12 s.

It was found because a fan got loud. CodexBar had an alibi, since its own CPU column was spotless: the work is billed to syspolicyd, which spent the hour confirming, carefully and repeatedly, that codex had still not become an app.

This PR remembers the verdict, bound to the file Gatekeeper actually assessed.

What changed

CodexLaunchPreflight.AssessmentMemo (new, CodexLaunchPreflight+AssessmentMemo.swift) follows the shape of KeychainAccessPreflight.ValidationMemo from #3857: a bounded memo (16 entries) with per-key in-flight sharing.

  • Gatekeeper assesses the inode, not the path. The memo stats the candidate, which follows symlinks, and has spctl assess /.vol/<device>/<inode>. That path names the resolved file directly, so no symlink or directory swapped while spctl runs can change what is assessed. /.vol works on APFS and returns the same verdicts: on the reporting Mac both the Developer ID codex and an ad-hoc CLI give identical output through either path. Where /.vol cannot reach the file, which the memo checks by comparing identities, nothing is memoized and the call behaves exactly as today.
  • Verdicts are keyed by the file's identity: device, inode, size, mtime and ctime. User space cannot set ctime, so a content write, chmod or xattr change (a quarantine attribute arriving included) is always a new identity.
  • A verdict is kept only if the file is unchanged when spctl returns. Every answer, whether a cache hit, a shared in-flight result or a fresh assessment, goes through one deliver step. That step re-checks, immediately before returning, that the caller's path still names that file. Otherwise the caller gets a fresh, unshared assessment of its path, exactly as on main.
  • Only definitive verdicts are kept (first diagnostic line accepted… or rejected…). Timeouts, launch failures and spctl errors stay retryable.
  • Verdicts are reported for the caller's path. The leading /.vol/…: is rewritten, so assessmentDiagnosticText's path stripping behaves as before.
  • App bundles are never memoized. The memo only engages for regular files, so the .app assessment path is byte-for-byte unchanged, per the fix(keychain): keep completed signature validations fresh #3838 ruling that bundle metadata cannot prove sealed resources unchanged. A single Mach-O carries its own signature, so its identity covers everything the assessment reads.
  • The injectable isLaunchCandidateAllowed(path:fileManager:hasExtendedAttribute:spctlAssessment:…) seam is untouched. Only the production entry point routes through the memo, so the existing preflight tests still never launch host binaries.

Review history

ClawSweeper's reviews each found a narrower way to swap the file out from under spctl:

  1. A symlink swapped during the assessment.
  2. A shared in-flight verdict handed to a waiter whose link had moved.
  3. A parent directory moved aside, replaced and restored.
  4. The path changing inside the cache-hit window.

Revision 2 tracked the path's symlinks, and revision 3 revalidated shared verdicts per caller. A draft that also compared every directory's ctime before and after was never pushed. That was abandoned because directory ctimes also move when unrelated entries change, and the suite's own parallel tests, all sharing the temp directory, defeated the cache. Revision 5 stopped following the path and asked about the file, which closed the first three cases at once with a simpler key. The fourth is closed by routing every answer through the same last-moment path re-check.

Decision for a maintainer: how long may a verdict live?

AssessmentMemo.lifetime is one constant, now 5 minutes. That is ClawSweeper's final recommendation, and the conservative end of the range, so approving it as-is is the safe choice. File changes of any kind already invalidate immediately, so the lifetime only bounds one thing: a certificate revoked in place while the file stays untouched.

lifetime assessments per hour, steady state syspolicyd cost on the reporting Mac
none (today) 24 at a 5-min refresh, up to ~600 when lookups spike 1.7% → ~55% of a core
1 hour 1 ~0.07%
15 min 4 ~0.3%
5 min (this PR) ≤ 12 ~0.8%, and one assessment per 5 min during spikes (vs ~600/hour)

If you prefer a longer window, it is a one-line change and the tests use Memo.lifetime symbolically. The spike case, where lookups ran every ~12 s, is the big win at any of these values.

Tests

CodexLaunchPreflightAssessmentMemoTests (17 tests, synthetic temp files only: no spctl, no host binaries, no Keychain):

  • Caching: one assessment for 100 lookups; invalidation on rewrite, on an xattr-only change (with mtime asserted unchanged, so ctime does the work) and on a repointed symlink; expiry; retryable timeouts and spctl errors; directories never memoized; 20 concurrent callers sharing one assessment; capacity eviction; the definitive-verdict classifier; verdicts reported for the caller's path, with the memo shown to have assessed a /.vol/ path.
  • Final decisions through isLaunchCandidateAllowed, all of which keep the unsigned CLI blocked:
    • a leaf link pointed at a signed CLI during the assessment and back
    • an intermediate directory link (current -> release-1) swapped during the assessment and back
    • the round-3 scenario: the parent directory moved aside, a signed tool placed at its pathname while spctl runs, the original restored. One assessment, blocked, and blocked from the cache afterwards.
    • a link retargeted to an unsigned binary while a second lookup waits on the shared assessment: both leader and waiter are blocked
    • the round-4 scenario: a link retargeted to an unsigned CLI inside the cache-hit window (via the memo's onCacheHit test hook): blocked, after one fresh assessment
    • a replaced executable blocked on the very next lookup, and a revocation that leaves the file untouched taking effect when the verdict expires

The existing Codex launch preflight tests in PathBuilderTests all still pass (66 tests across both suites).

Real-machine proof (the Intel iMac that reported #4078)

The redacted trace is also committed as .github/pr-proof/codex-gatekeeper-assessment-memo.log, following the repo's pr-proof convention.

Intel Core i7-4790K, macOS 15.7.7, Codex CLI 0.145.0 standalone (~/.local/bin/codex → ~/.codex/packages/standalone/current/bin/codex, 281 MB x86_64). Counts come from the unified log (log show --predicate 'process == "spctl"'). Private paths are shown relative to ~.

1. How often CodexBar runs spctl, before and after.

build window refreshes spctl runs by CodexBar
official 0.57.0 (main behaviour) 16:50–17:10 5 10, a pair on every refresh
official 0.57.0, elevated lookup cadence 17:00–18:00 — 494
this PR, revision 2 (0.68.1, 160), CODEX_CLI_PATH unset 20:31–20:47 4 (launch + 3) 4, all within five minutes of launch; 0 at the 20:41:30 and 20:46:31 refreshes, which still ran codex

Later revisions change only how the verdict is bound, not how often it is reused. On the cold start after launch, the first three assessments needed more than the existing 5 s spctl timeout (a cold assessment of this binary takes ~11.5 s). They were terminated and, as before this PR, not remembered. The fourth finished in 2.64 s and was remembered.

2. Final effect through the production path, real Gatekeeper (this revision). This is a throwaway local harness, not part of the PR. It calls the public CodexLaunchPreflight.isLaunchCandidateAllowed(path:), so it goes through the memo and the real spctl, on a symlink retargeted between the Developer ID codex and an ad-hoc-signed CLI (spctl: rejected / source=no usable signature).

codex (Developer ID), lookup 1           ALLOWED    2580 ms   spctl on /.vol/<dev>/<inode>, remembered
codex, lookup 2                          ALLOWED       0 ms   memo
codex, lookup 3                          ALLOWED       0 ms   memo
link retargeted to ad-hoc binary         BLOCKED      76 ms   a different file → its own assessment
ad-hoc binary, lookup 2                  BLOCKED       0 ms   memo
link retargeted back to codex            ALLOWED       0 ms   the same file as lookup 1 → its verdict
codex, lookup 5                          ALLOWED       0 ms   memo

Shared in-flight assessment, run in a fresh process so the codex assessment is genuinely in flight: a leader starts on codex, a second lookup joins 300 ms later, and the link is retargeted to the ad-hoc binary 1 s in.

-- link retargeted to ad-hoc binary at 1004 ms, mid-assessment
leader (starts spctl on codex)           BLOCKED  returned at 2637 ms
waiter (joins the same assessment)       BLOCKED  returned at 2637 ms

The unified log shows one spctl on codex (2.47 s), then two short spctl runs on the ad-hoc binary, one fresh assessment per caller. Neither caller received the verdict for the file its path no longer named.

Cache-hit window, the round-4 finding. The window is microseconds wide, so it is reached deterministically through the memo's onCacheHit test hook. The run uses the production decision function (the internal isLaunchCandidateAllowed seam), the production AssessmentMemo, and a real spctl run with the production arguments:

codex (Developer ID), lookup 1                       ALLOWED  9479 ms  spctl runs: 1   (cold)
codex, lookup 2 (cache hit)                          ALLOWED     0 ms  spctl runs: 0
cache hit; link -> ad-hoc inside the hit window      BLOCKED   138 ms  spctl runs: 1
ad-hoc binary, next lookup                           BLOCKED   137 ms  spctl runs: 1

The remembered "allowed" for codex never reached the decision for the ad-hoc binary.

Commands run

Full CI on Xcode 26.6. The upstream run for this PR is waiting on first-contributor approval, so the repository's own ci.yml was run unchanged on the fork's pull request: all 8 jobs green: lint (lint.sh lint-linux), both swift-test-macos shards (Xcode 26.6, which includes lint.sh lint-macos and the full sharded Swift test run; CodexLaunchPreflightAssessmentMemoTests ran in shard 1 and passed), the Linux x64/arm64/musl CLI builds, and the lint-build-test gate. Same head as this PR.

Locally (Command Line Tools + swift.org 6.3.3):

swift test --filter 'CodexLaunchPreflightAssessmentMemoTests|PathBuilderTests'   # 66 passed
.build/lint-tools/bin/swiftformat --lint <changed files>                         # 0 files require formatting
.build/lint-tools/bin/swiftlint lint --strict --no-cache <changed files>          # 0 violations
CODEXBAR_SIGNING=adhoc ./Scripts/package_app.sh release                           # launch smoke check OK

Transparency note: this machine has Command Line Tools and a swift.org 6.3.3 toolchain, not Xcode. Building locally therefore needed a few local-only workarounds that are not in this PR:

  • desugaring SwiftUI's @Entry and dropping KeyboardShortcuts' #Preview blocks (their macro plugins ship only with Xcode)
  • narrowing the test target to the two Swift Testing suites above, because there is no XCTest, which is also why the full make test could not run here
  • reusing the 0.68.0 widget appex (its build step calls xcodebuild)
  • weak-linking swift_Concurrency, which the Xcode toolchain does on its own

The fork CI run above is the full sharded make test / lint.sh equivalent on Xcode 26.6.

@clawsweeper

clawsweeper Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 28, 2026
@clawsweeper

clawsweeper Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed September 28, 2026, 12:15 AM ET / 04:15 UTC (Revision 6).

ClawSweeper review

What this changes

The branch caches Gatekeeper assessments of standalone Codex CLI files, shares concurrent assessments, and adds regression tests, a native trace, and a changelog entry.

Merge readiness

⛔ Blocked before merge - 8 items remain

The optimization addresses a measured CPU cost and has useful native proof, but a completed cache hit can still return an allowed verdict after the caller’s path changes. The proposed five-minute window for an unchanged file also needs an explicit trust-freshness decision.

Priority: P2
Reviewed head: c9b0edd56383e08456fee1f2866174277c20ef5c
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The measured performance gain and native trace are useful, but the remaining cache-hit security race and its missing final-effect proof prevent merge readiness.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: Authority-chain proof required: the committed Intel Mac trace exercises the production preflight with real spctl, shows fewer assessments, and blocks an unsigned target after a pre-lookup swap and an in-flight swap. It does not show the nearest forbidden case: a path change after the initial stat of a completed cache hit but before that verdict reaches the final decision. No stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🦐 gold shrimp (3/6) Security review found an item that needs attention.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: Authority-chain proof required: the committed Intel Mac trace exercises the production preflight with real spctl, shows fewer assessments, and blocks an unsigned target after a pre-lookup swap and an in-flight swap. It does not show the nearest forbidden case: a path change after the initial stat of a completed cache hit but before that verdict reaches the final decision. No stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 8 items Introduced cache-hit path: A completed entry returns its stored assessment after the initial file stat, without the caller-path recheck used for in-flight and fresh results.
Final decision: The production preflight consumes the memoized assessment and uses it to allow or block a native candidate.
Native proof: The committed Intel Mac trace reports fewer assessments and blocked results after a link retarget before lookup and during an in-flight assessment. It does not record a retarget between the initial stat and return of a completed cache hit.
Findings 1 actionable finding [P1] Revalidate the caller path on a completed cache hit
Security Needs attention Cached allowance can outlive its caller path: The cache-hit branch lacks the final file-identity check used by other branches, so an unsigned replacement can receive the earlier file’s allowed result.
Unapproved trust-freshness interval: An unchanged binary can retain an allowance for five minutes after an external trust revocation; the acceptable policy window is a maintainer choice.

How this fits together

CodexBar locates a Codex executable for usage probes and local session scanning. Its launch preflight checks candidate files and passes an allowed or blocked decision to those callers.

flowchart LR
  A[Codex binary lookup] --> B[Launch preflight]
  B --> C[File identity]
  C --> D{Cached verdict valid?}
  D -->|Yes| E[Gatekeeper verdict]
  D -->|No| F[Gatekeeper assessment]
  F --> E
  E --> G[Allow or block candidate]
Loading

Decision needed

Question Recommendation
Is five minutes an acceptable maximum reuse window for a Codex CLI Gatekeeper verdict when system trust changes but the file does not? Approve five minutes: Accept the bounded revocation-detection delay after the cache-hit race is repaired and proved.

Why: File identity cannot detect in-place certificate revocation, and the shared preflight is used by local session scanning as well as usage lookup.

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: Authority-chain proof required: the committed Intel Mac trace exercises the production preflight with real spctl, shows fewer assessments, and blocks an unsigned target after a pre-lookup swap and an in-flight swap. It does not show the nearest forbidden case: a path change after the initial stat of a completed cache hit but before that verdict reaches the final decision. No stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Revalidate the caller path on a completed cache hit (P1) - After the initial stat, another thread can retarget the caller’s path before this cached return. The stored allowed verdict then reaches the native launch decision for a different, unsigned file. Recheck the caller’s identity here, as the in-flight and fresh-result paths already do, and cover the final blocked decision.
  • Resolve security concern: Cached allowance can outlive its caller path - The cache-hit branch lacks the final file-identity check used by other branches, so an unsigned replacement can receive the earlier file’s allowed result.
  • Resolve security concern: Unapproved trust-freshness interval - An unchanged binary can retain an allowance for five minutes after an external trust revocation; the acceptable policy window is a maintainer choice.
  • Resolve merge risk (P1) - A completed cached allowance can be returned for a different file if the caller’s path changes after the initial stat and before the cache-hit return.
  • Resolve merge risk (P1) - A certificate revoked while the file remains untouched can retain an allowed verdict for up to five minutes; the acceptable freshness window has not received an owner ruling.
  • Complete next step (P2) - Repair the completed cache-hit race, add redacted final-decision proof for the unsigned stat-to-return swap, and obtain the owner’s ruling on five-minute verdict reuse; updating the PR body should trigger re-review, or a maintainer can request @clawsweeper re-review.
  • Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

  • [P1] Revalidate the caller path on a completed cache hit — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:74-76
  • [high] Cached allowance can outlive its caller path — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:74
  • [medium] Unapproved trust-freshness interval — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:27
Agent review details

Security

Needs attention: A completed cache hit can transfer an allowed Gatekeeper verdict to a retargeted executable, and the unchanged-file revocation window needs approval.

Review metrics

Metric Value Why it matters
Code and test delta production +183/−1 lines, tests +494 lines The production growth implements bounded sharing and file binding, while the larger test addition exercises concurrency and replacement cases.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #4078
Summary: This PR is the open candidate fix for the linked repeated-Gatekeeper-assessment issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

  1. Repair the cache-hit guard (recommended)
    Revalidate the caller path before a completed verdict is returned and show an unsigned swap is blocked at the final decision.
  2. Pause for trust policy
    Hold the PR if the owner does not accept a bounded stale-verdict interval.

Technical review

Best possible solution:

Recheck the caller’s file identity immediately before returning a completed cache hit, prove the unsigned target is blocked at the final decision, and use an owner-approved short expiry for unchanged files.

Do we have a high-confidence way to reproduce the issue?

Yes for the reported performance problem: current main starts a fresh spctl assessment on each native candidate lookup, and the reporter supplied Intel Mac timing and process counts. The cache-hit race is source-reproducible; this read-only review did not execute macOS code.

Is this the best way to solve the issue?

Not yet. A bounded memo fits the repeated-assessment problem, but the completed-hit path must validate its caller before its verdict is used, and the trust lifetime needs approval.

Full review comments:

  • [P1] Revalidate the caller path on a completed cache hit — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:74-76
    After the initial stat, another thread can retarget the caller’s path before this cached return. The stored allowed verdict then reaches the native launch decision for a different, unsigned file. Recheck the caller’s identity here, as the in-flight and fresh-result paths already do, and cover the final blocked decision.
    Confidence: 0.94

Overall correctness: patch is incorrect
Overall confidence: 0.92

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 579f68406855.

Labels

Label changes:

  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Authority-chain proof required: the committed Intel Mac trace exercises the production preflight with real spctl, shows fewer assessments, and blocks an unsigned target after a pre-lookup swap and an in-flight swap. It does not show the nearest forbidden case: a path change after the initial stat of a completed cache hit but before that verdict reaches the final decision. No stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • remove proof: sufficient: Current real behavior proof status is insufficient, not sufficient.
  • remove status: ⏳ waiting on author: Current PR status label is status: 📣 needs proof.

Label justifications:

  • P2: The PR targets measurable background CPU use with a bounded Codex CLI preflight change.
  • merge-risk: 🚨 security-boundary: Caching a Gatekeeper allowance changes when a candidate’s trust verdict is refreshed, and the completed-hit path has a reachable stale-path race.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Authority-chain proof required: the committed Intel Mac trace exercises the production preflight with real spctl, shows fewer assessments, and blocks an unsigned target after a pre-lookup swap and an in-flight swap. It does not show the nearest forbidden case: a path change after the initial stat of a completed cache hit but before that verdict reaches the final decision. No stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

Security concerns:

  • [high] Cached allowance can outlive its caller path — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:74
    The cache-hit branch lacks the final file-identity check used by other branches, so an unsigned replacement can receive the earlier file’s allowed result.
    Confidence: 0.94
  • [medium] Unapproved trust-freshness interval — Sources/CodexBarCore/CodexLaunchPreflight+AssessmentMemo.swift:27
    An unchanged binary can retain an allowance for five minutes after an external trust revocation; the acceptable policy window is a maintainer choice.
    Confidence: 0.85

What I checked:

Likely related people:

  • Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Oleksiy Akimov: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Revalidate completed cache hits and add final-effect proof that the nearest unsigned target is blocked after a stat-to-return swap.
  • Obtain the owner’s ruling on the five-minute trust-verdict lifetime.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)
  • reviewed 2026-09-28T00:53:45.665Z sha 6b56ae7 :: needs real behavior proof before merge. :: [P1] Bind the cached verdict to the executable assessed
  • reviewed 2026-09-28T01:51:44.103Z sha d6de435 :: needs real behavior proof before merge. :: [P1] Revalidate the path before returning an in-flight verdict
  • reviewed 2026-09-28T02:08:44.856Z sha 8b8d230 :: needs real behavior proof before merge. :: [P1] Bind the verdict across ordinary directory replacements
  • reviewed 2026-09-28T02:36:27.656Z sha 1b3c5a8 :: blocked before merge. :: none
  • reviewed 2026-09-28T03:30:13.855Z sha c9b0edd :: needs real behavior proof before merge. :: [P1] Revalidate the caller path on a completed cache hit

@dustball
dustball force-pushed the fix/codex-preflight-assessment-memo branch from 6b56ae7 to d6de435 Compare September 28, 2026 01:45
@dustball
dustball marked this pull request as ready for review September 28, 2026 01:47
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. proof: sufficient Contributor real behavior proof is sufficient. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 28, 2026
dustball and others added 5 commits September 27, 2026 22:25
Every Codex binary lookup ran `spctl --assess` on the native executable, and
nothing cached the verdict. Each assessment re-hashes the whole binary in
syspolicyd (~2.5 s of CPU for a 281 MB x86_64 codex), so the cost scaled with
refresh cadence: ~1.7% of a core at a 5-minute refresh, ~55% when lookups ran
every few seconds.

Remember definitive verdicts (accepted/rejected) for regular files, keyed by
path plus device, inode, size, mtime and ctime, for up to an hour. ctime is
not settable from user space, so a content write, chmod or xattr change
(quarantine included) always re-assesses. Timeouts and spctl errors stay
retryable, concurrent callers share one in-flight assessment, and app bundles
are never memoized, matching the steipete#3838 rule that bundle metadata cannot prove
sealed resources unchanged.

Fixes steipete#4078

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Review found that the memo read the file identity before `spctl` ran and
stored the verdict under it, so a symlink swapped during the assessment and
swapped back would leave another executable's verdict in the cache.

The key now resolves the path itself, recording every symlink it crosses
(a link cannot be retargeted in place, so a swap changes its inode or ctime
even when reverted), plus the resolved file's identity. A verdict is kept only
when the key read after `spctl` returns equals the one read before.

Adds regressions for a leaf link and an intermediate directory link swapped
during assessment, and a final-effect test through isLaunchCandidateAllowed:
a replaced executable is blocked on the next lookup, and a revocation that
leaves the file untouched takes effect when the verdict expires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
…te lifetime

Review round 2 found that a caller joining an in-flight assessment returned the
leader's verdict without rechecking its own path, so a link retargeted to a
forbidden executable mid-assessment could be allowed on the old target's
verdict. The leader already declined to cache that verdict but still published
it to waiters.

Every caller now gets a verdict only when the path still names the assessed
file after `spctl` returns; otherwise it runs a fresh, unshared assessment.
Adds a final-decision regression through isLaunchCandidateAllowed where the
link is retargeted to an unsigned binary during a shared assessment: both the
leader and the waiter are blocked.

Also shortens the verdict lifetime from one hour to 15 minutes, per the
review's recommendation on the certificate-revocation window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Review round 3 found that a parent directory moved aside, replaced while spctl
ran, and restored could leave a different tool's verdict cached for the
original, because the path spctl traversed could change under it.

Instead of tracking every link and directory on the path, the memo now has
Gatekeeper assess `/.vol/<device>/<inode>`, which names the resolved file
directly: nothing on the path can change what was assessed. Verdicts are
keyed by that file's identity (device, inode, size, mtime, ctime), kept only
if it is unchanged when spctl returns, and handed to a caller only while the
caller's path still names that file; otherwise the caller gets a fresh,
unshared assessment. Verdicts are reported for the caller's path. Where /.vol
cannot reach the file, nothing is memoized.

Replaces the path-walking key from the previous revision (simpler, and it
also removes the noise that directory timestamps would have added). Adds the
reviewer's directory-swap scenario and link swaps during assessment as
final-decision tests: the unsigned CLI stays blocked in every case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Adopt ClawSweeper's recommended five-minute bound on how long a cached
Gatekeeper verdict can outlive an in-place certificate revocation. The
elevated-cadence case that motivated steipete#4078 still collapses to one
assessment per five minutes.

Adds .github/pr-proof/codex-gatekeeper-assessment-memo.log with the Intel Mac
before/after spctl counts and the production-path harness output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
@dustball

Copy link
Copy Markdown
Author

@steipete this is ready for your review. It's a perf fix under Merge by Default, bounded to the Codex launch preflight.

  • ClawSweeper's findings are all addressed. Its three race findings are fixed in code: the final design has Gatekeeper assess /.vol/<device>/<inode>, so the verdict belongs to the exact file assessed. Each finding has a final-decision regression test. Its latest review reported Findings: None and proof 5/6.
  • One decision for you: the verdict lifetime. It is now 5 minutes, ClawSweeper's recommendation and the conservative option. The trade-off table is in the PR body if you'd rather go longer; it is a one-line change.
  • CI. The upstream run is waiting on first-contributor approval (run). The same ci.yml on the same head passed all 8 jobs on the fork, including both Xcode 26.6 macOS shards: run.
  • Proof. Real Intel Mac before/after counts and a production-path harness with the real spctl, committed in .github/pr-proof/codex-gatekeeper-assessment-memo.log.

Maintainer edits are enabled. Squash as you like; the five commits are the review history, not five features.

@clawsweeper clawsweeper Bot added status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed proof: sufficient Contributor real behavior proof is sufficient. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 28, 2026
Review of revision 5 found that the cache-hit branch returned a remembered
verdict after the initial stat without re-checking that the caller's path
still names that file, unlike the shared and fresh branches.

All three branches now return through one `deliver` step that re-checks the
caller's path immediately before answering, and otherwise runs a fresh,
unshared assessment. Adds a final-decision regression through
isLaunchCandidateAllowed with a test hook that retargets the link to an
unsigned CLI inside the cache-hit window: the decision is blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
@dustball dustball closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex binary preflight runs an uncached spctl --assess on every lookup (~2.5 s of syspolicyd CPU each; ~55% of a core when refresh cadence rises)

1 participant