fix(ops): keep remote builds off ACK system nodes - #706
Conversation
Cloud Assistant validation builds hard-coded an ack-system node and /tmp, exhausting its 40 GiB system disk. Require the research worker /work task root with scoped Cargo/Rustup/target state and EXIT cleanup.
|
Warning Review limit reached
Next review available in: 40 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0df9ab8c7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,7 @@ | |||
| owner: codex | |||
| contract: remote-build-workspace-guard | |||
| worktree: /Users/proerror/Documents/monday/.worktrees/codex/remote-build-workspace-guard | |||
There was a problem hiding this comment.
This committed ownership file is not the record that the preflight gate reads: .github/scripts/agent-worktree-preflight.sh resolves it with git rev-parse --git-path agent-worktree.yml and then requires the recorded worktree/branch to match the current managed worktree. Because this tracked file contains one developer's /Users/... path and branch, every checkout now carries a stale ownership tuple while the actual gate still expects a separate worktree-private record, which undermines the ownership guard agents are instructed to use before edits/commits. Remove this from the committed tree and keep the record in the git-private path instead.
AGENTS.md reference: AGENTS.md:L77-L78
Useful? React with 👍 / 👎.
| @@ -0,0 +1,7 @@ | |||
| owner: codex | |||
| contract: remote-build-workspace-guard | |||
| worktree: /Users/proerror/Documents/monday/.worktrees/codex/remote-build-workspace-guard | |||
There was a problem hiding this comment.
This committed ownership file is not the record that the preflight gate reads: .github/scripts/agent-worktree-preflight.sh resolves it with git rev-parse --git-path agent-worktree.yml and then requires the recorded worktree/branch to match the current managed worktree. Because this tracked file contains one developer's /Users/... path and branch, every checkout now carries a stale ownership tuple while the actual gate still expects a separate worktree-private record, which undermines the ownership guard agents are instructed to use before edits/commits. Remove this from the committed tree and keep the record in the git-private path instead.
Useful? React with 👍 / 👎.
| export TMPDIR="$task_root/tmp" | ||
| export CARGO_HOME="$task_root/cargo" | ||
| export RUSTUP_HOME="$task_root/rustup" | ||
| export CARGO_TARGET_DIR="$task_root/target" | ||
| mkdir -p "$TMPDIR" "$CARGO_HOME" "$RUSTUP_HOME" "$CARGO_TARGET_DIR" |
There was a problem hiding this comment.
When this skill is used to build rust_hft/prediction-markets on a research worker with sccache installed, the repo's Cargo config invokes scripts/rustc-wrapper.sh, which execs sccache; because this contract only redirects TMPDIR/CARGO_HOME/RUSTUP_HOME/CARGO_TARGET_DIR and never sets SCCACHE_DIR (or disables the wrapper), sccache keeps its documented default local cache under the user's home directory on the system disk, leaving mutable build artifacts outside /work/monday-builds and outside the cleanup/readback check. Add SCCACHE_DIR="$task_root/sccache" or turn the wrapper off for these runs.
Useful? React with 👍 / 👎.
1 participant
Change contract
Require every remote Rust build or validation to run on the Monday research worker's
/workdisk, with isolated Cargo/Rustup/target state and cleanup on exit. Reject ACK system nodes and/tmpfallback.Issue relationship
None
Out of scope
Does not change the running cluster, provision disks, or delete the still-active historical
/tmp/monday-695.Hd06hItask root.Dependencies and merge order
None
Focused validation
.github/scripts/agent-worktree-preflight.sh checkgit diff --checki-6wegi12vjys3jxrym41b(role=ack-system) and acceptedi-6weh9o5xrq4xy8pb1a4q(role=research-worker)./work/monday-buildstask-root lifecycle: created scoped state, exited, and read back an empty parent./workretained about 93 GiB free.Rollout and rollback
No service rollout. The emergency cleanup and live workspace preflight were performed separately; this change makes the safe remote-build contract canonical. Roll back by reverting this single commit if the operating contract needs replacement.
Scope exception
None