Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs real behavior proof before merge. Reviewed September 28, 2026, 12:15 AM ET / 04:15 UTC (Revision 6). ClawSweeper reviewWhat this changesThe branch caches Gatekeeper assessments of standalone Codex CLI files, shares concurrent assessments, and adds regression tests, a native trace, and a changelog entry. Merge readiness⛔ Blocked before merge - 8 items remain The optimization addresses a measured CPU cost and has useful native proof, but a completed cache hit can still return an allowed verdict after the caller’s path changes. The proposed five-minute window for an unchanged file also needs an explicit trust-freshness decision. Priority: P2 Review scores
Verification
How this fits togetherCodexBar locates a Codex executable for usage probes and local session scanning. Its launch preflight checks candidate files and passes an allowed or blocked decision to those callers. flowchart LR
A[Codex binary lookup] --> B[Launch preflight]
B --> C[File identity]
C --> D{Cached verdict valid?}
D -->|Yes| E[Gatekeeper verdict]
D -->|No| F[Gatekeeper assessment]
F --> E
E --> G[Allow or block candidate]
Decision needed
Why: File identity cannot detect in-place certificate revocation, and the shared preflight is used by local session scanning as well as usage lookup. Before merge
Findings
Agent review detailsSecurityNeeds attention: A completed cache hit can transfer an allowed Gatekeeper verdict to a retargeted executable, and the unchanged-file revocation window needs approval. Review metrics
Root-cause clusterRelationship: Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Recheck the caller’s file identity immediately before returning a completed cache hit, prove the unsigned target is blocked at the final decision, and use an owner-approved short expiry for unchanged files. Do we have a high-confidence way to reproduce the issue? Yes for the reported performance problem: current main starts a fresh spctl assessment on each native candidate lookup, and the reporter supplied Intel Mac timing and process counts. The cache-hit race is source-reproducible; this read-only review did not execute macOS code. Is this the best way to solve the issue? Not yet. A bounded memo fits the repeated-assessment problem, but the completed-hit path must validate its caller before its verdict is used, and the trust lifetime needs approval. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 579f68406855. LabelsLabel changes:
Label justifications:
EvidenceSecurity concerns:
What I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (5 earlier review cycles)
|
6b56ae7 to
d6de435
Compare
Every Codex binary lookup ran `spctl --assess` on the native executable, and nothing cached the verdict. Each assessment re-hashes the whole binary in syspolicyd (~2.5 s of CPU for a 281 MB x86_64 codex), so the cost scaled with refresh cadence: ~1.7% of a core at a 5-minute refresh, ~55% when lookups ran every few seconds. Remember definitive verdicts (accepted/rejected) for regular files, keyed by path plus device, inode, size, mtime and ctime, for up to an hour. ctime is not settable from user space, so a content write, chmod or xattr change (quarantine included) always re-assesses. Timeouts and spctl errors stay retryable, concurrent callers share one in-flight assessment, and app bundles are never memoized, matching the steipete#3838 rule that bundle metadata cannot prove sealed resources unchanged. Fixes steipete#4078 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Review found that the memo read the file identity before `spctl` ran and stored the verdict under it, so a symlink swapped during the assessment and swapped back would leave another executable's verdict in the cache. The key now resolves the path itself, recording every symlink it crosses (a link cannot be retargeted in place, so a swap changes its inode or ctime even when reverted), plus the resolved file's identity. A verdict is kept only when the key read after `spctl` returns equals the one read before. Adds regressions for a leaf link and an intermediate directory link swapped during assessment, and a final-effect test through isLaunchCandidateAllowed: a replaced executable is blocked on the next lookup, and a revocation that leaves the file untouched takes effect when the verdict expires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
…te lifetime Review round 2 found that a caller joining an in-flight assessment returned the leader's verdict without rechecking its own path, so a link retargeted to a forbidden executable mid-assessment could be allowed on the old target's verdict. The leader already declined to cache that verdict but still published it to waiters. Every caller now gets a verdict only when the path still names the assessed file after `spctl` returns; otherwise it runs a fresh, unshared assessment. Adds a final-decision regression through isLaunchCandidateAllowed where the link is retargeted to an unsigned binary during a shared assessment: both the leader and the waiter are blocked. Also shortens the verdict lifetime from one hour to 15 minutes, per the review's recommendation on the certificate-revocation window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Review round 3 found that a parent directory moved aside, replaced while spctl ran, and restored could leave a different tool's verdict cached for the original, because the path spctl traversed could change under it. Instead of tracking every link and directory on the path, the memo now has Gatekeeper assess `/.vol/<device>/<inode>`, which names the resolved file directly: nothing on the path can change what was assessed. Verdicts are keyed by that file's identity (device, inode, size, mtime, ctime), kept only if it is unchanged when spctl returns, and handed to a caller only while the caller's path still names that file; otherwise the caller gets a fresh, unshared assessment. Verdicts are reported for the caller's path. Where /.vol cannot reach the file, nothing is memoized. Replaces the path-walking key from the previous revision (simpler, and it also removes the noise that directory timestamps would have added). Adds the reviewer's directory-swap scenario and link swaps during assessment as final-decision tests: the unsigned CLI stays blocked in every case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Adopt ClawSweeper's recommended five-minute bound on how long a cached Gatekeeper verdict can outlive an in-place certificate revocation. The elevated-cadence case that motivated steipete#4078 still collapses to one assessment per five minutes. Adds .github/pr-proof/codex-gatekeeper-assessment-memo.log with the Intel Mac before/after spctl counts and the production-path harness output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
1b3c5a8 to
c9b0edd
Compare
|
@steipete this is ready for your review. It's a perf fix under Merge by Default, bounded to the Codex launch preflight.
Maintainer edits are enabled. Squash as you like; the five commits are the review history, not five features. |
Review of revision 5 found that the cache-hit branch returned a remembered verdict after the initial stat without re-checking that the caller's path still names that file, unlike the shared and fresh branches. All three branches now return through one `deliver` step that re-checks the caller's path immediately before answering, and otherwise runs a fresh, unshared assessment. Adds a final-decision regression through isLaunchCandidateAllowed with a test hook that retargets the link to an unsigned CLI inside the cache-hit window: the decision is blocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014giiyt6RgBarhNPpjn2Xiz
Fixes #4078.
Summary
Every Codex binary lookup runs
CodexLaunchPreflight, which spawnsspctl --assesson the nativecodexexecutable. Nothing caches the verdict, and Gatekeeper does not cache arejected (the code is valid but does not seem to be an app)result either, so every lookup re-hashes the whole binary insyspolicyd. On an Intel Mac that is ~2.5 s of CPU per call for the 281 MB x86_64 standalone CLI: ~1.7% of a core at a 5-minute refresh, and ~55% of a core during a 52-minute stretch when lookups ran every ~12 s.It was found because a fan got loud. CodexBar had an alibi, since its own CPU column was spotless: the work is billed to
syspolicyd, which spent the hour confirming, carefully and repeatedly, thatcodexhad still not become an app.This PR remembers the verdict, bound to the file Gatekeeper actually assessed.
What changed
CodexLaunchPreflight.AssessmentMemo(new,CodexLaunchPreflight+AssessmentMemo.swift) follows the shape ofKeychainAccessPreflight.ValidationMemofrom #3857: a bounded memo (16 entries) with per-key in-flight sharing.stats the candidate, which follows symlinks, and hasspctlassess/.vol/<device>/<inode>. That path names the resolved file directly, so no symlink or directory swapped whilespctlruns can change what is assessed./.volworks on APFS and returns the same verdicts: on the reporting Mac both the Developer IDcodexand an ad-hoc CLI give identical output through either path. Where/.volcannot reach the file, which the memo checks by comparing identities, nothing is memoized and the call behaves exactly as today.chmodor xattr change (a quarantine attribute arriving included) is always a new identity.spctlreturns. Every answer, whether a cache hit, a shared in-flight result or a fresh assessment, goes through onedeliverstep. That step re-checks, immediately before returning, that the caller's path still names that file. Otherwise the caller gets a fresh, unshared assessment of its path, exactly as onmain.accepted…orrejected…). Timeouts, launch failures andspctlerrors stay retryable./.vol/…:is rewritten, soassessmentDiagnosticText's path stripping behaves as before..appassessment path is byte-for-byte unchanged, per the fix(keychain): keep completed signature validations fresh #3838 ruling that bundle metadata cannot prove sealed resources unchanged. A single Mach-O carries its own signature, so its identity covers everything the assessment reads.isLaunchCandidateAllowed(path:fileManager:hasExtendedAttribute:spctlAssessment:…)seam is untouched. Only the production entry point routes through the memo, so the existing preflight tests still never launch host binaries.Review history
ClawSweeper's reviews each found a narrower way to swap the file out from under
spctl:Revision 2 tracked the path's symlinks, and revision 3 revalidated shared verdicts per caller. A draft that also compared every directory's ctime before and after was never pushed. That was abandoned because directory ctimes also move when unrelated entries change, and the suite's own parallel tests, all sharing the temp directory, defeated the cache. Revision 5 stopped following the path and asked about the file, which closed the first three cases at once with a simpler key. The fourth is closed by routing every answer through the same last-moment path re-check.
Decision for a maintainer: how long may a verdict live?
AssessmentMemo.lifetimeis one constant, now 5 minutes. That is ClawSweeper's final recommendation, and the conservative end of the range, so approving it as-is is the safe choice. File changes of any kind already invalidate immediately, so the lifetime only bounds one thing: a certificate revoked in place while the file stays untouched.syspolicydcost on the reporting MacIf you prefer a longer window, it is a one-line change and the tests use
Memo.lifetimesymbolically. The spike case, where lookups ran every ~12 s, is the big win at any of these values.Tests
CodexLaunchPreflightAssessmentMemoTests(17 tests, synthetic temp files only: nospctl, no host binaries, no Keychain):spctlerrors; directories never memoized; 20 concurrent callers sharing one assessment; capacity eviction; the definitive-verdict classifier; verdicts reported for the caller's path, with the memo shown to have assessed a/.vol/path.isLaunchCandidateAllowed, all of which keep the unsigned CLI blocked:current -> release-1) swapped during the assessment and backspctlruns, the original restored. One assessment, blocked, and blocked from the cache afterwards.onCacheHittest hook): blocked, after one fresh assessmentThe existing
Codex launch preflighttests inPathBuilderTestsall still pass (66 tests across both suites).Real-machine proof (the Intel iMac that reported #4078)
The redacted trace is also committed as
.github/pr-proof/codex-gatekeeper-assessment-memo.log, following the repo'spr-proofconvention.Intel Core i7-4790K, macOS 15.7.7, Codex CLI 0.145.0 standalone (
~/.local/bin/codex→~/.codex/packages/standalone/current/bin/codex, 281 MB x86_64). Counts come from the unified log (log show --predicate 'process == "spctl"'). Private paths are shown relative to~.1. How often CodexBar runs
spctl, before and after.spctlruns by CodexBarCODEX_CLI_PATHunsetcodexLater revisions change only how the verdict is bound, not how often it is reused. On the cold start after launch, the first three assessments needed more than the existing 5 s
spctltimeout (a cold assessment of this binary takes ~11.5 s). They were terminated and, as before this PR, not remembered. The fourth finished in 2.64 s and was remembered.2. Final effect through the production path, real Gatekeeper (this revision). This is a throwaway local harness, not part of the PR. It calls the public
CodexLaunchPreflight.isLaunchCandidateAllowed(path:), so it goes through the memo and the realspctl, on a symlink retargeted between the Developer IDcodexand an ad-hoc-signed CLI (spctl:rejected/source=no usable signature).Shared in-flight assessment, run in a fresh process so the
codexassessment is genuinely in flight: a leader starts oncodex, a second lookup joins 300 ms later, and the link is retargeted to the ad-hoc binary 1 s in.The unified log shows one
spctloncodex(2.47 s), then two shortspctlruns on the ad-hoc binary, one fresh assessment per caller. Neither caller received the verdict for the file its path no longer named.Cache-hit window, the round-4 finding. The window is microseconds wide, so it is reached deterministically through the memo's
onCacheHittest hook. The run uses the production decision function (the internalisLaunchCandidateAllowedseam), the productionAssessmentMemo, and a realspctlrun with the production arguments:The remembered "allowed" for
codexnever reached the decision for the ad-hoc binary.Commands run
Full CI on Xcode 26.6. The upstream run for this PR is waiting on first-contributor approval, so the repository's own
ci.ymlwas run unchanged on the fork's pull request: all 8 jobs green:lint(lint.sh lint-linux), bothswift-test-macosshards (Xcode 26.6, which includeslint.sh lint-macosand the full sharded Swift test run;CodexLaunchPreflightAssessmentMemoTestsran in shard 1 and passed), the Linux x64/arm64/musl CLI builds, and thelint-build-testgate. Same head as this PR.Locally (Command Line Tools + swift.org 6.3.3):
Transparency note: this machine has Command Line Tools and a swift.org 6.3.3 toolchain, not Xcode. Building locally therefore needed a few local-only workarounds that are not in this PR:
@Entryand droppingKeyboardShortcuts'#Previewblocks (their macro plugins ship only with Xcode)make testcould not run herexcodebuild)swift_Concurrency, which the Xcode toolchain does on its ownThe fork CI run above is the full sharded
make test/lint.shequivalent on Xcode 26.6.