Lock RLM harness to a run-start commit, keep nano-rlm rename - #2376
Lock RLM harness to a run-start commit, keep nano-rlm rename#2376samsja wants to merge 3 commits into
Conversation
This reverts commit 5787d1d.
rlm-harness was renamed to nano-rlm on GitHub; point RLM_REPO and the version docstring at the new name. Default version stays main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve branch/tag versions to a commit SHA host-side, once per run, so every rollout installs the same harness even if the ref moves mid-run. The default stays main: runs always start from the latest harness, but a push to nano-rlm can no longer change a live run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| if self._resolved_version is None: | ||
| version = self.config.version | ||
| if not COMMIT_RE.fullmatch(version): | ||
| proc = await asyncio.create_subprocess_exec( |
There was a problem hiding this comment.
🟠 High rlm/harness.py:87
With the default version="main", _resolve_version raises FileNotFoundError on hosts without a git executable before the runtime can install git, so no RLM rollout can be set up. Avoid requiring host-side git for version resolution, or detect its absence and raise a controlled configuration error.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/rlm/harness.py around line 87:
With the default `version="main"`, `_resolve_version` raises `FileNotFoundError` on hosts without a `git` executable before the runtime can install git, so no RLM rollout can be set up. Avoid requiring host-side git for version resolution, or detect its absence and raise a controlled configuration error.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ce80a5c. Configure here.
| version, | ||
| ) | ||
| self._resolved_version = version | ||
| return self._resolved_version |
There was a problem hiding this comment.
Version lock does not span rollouts
High Severity
_resolved_version and _version_lock are stored via self, so each new harness instance resolves independently. Eval episodes construct a fresh Agent (and thus a fresh RLMHarness) per rollout, so a moving main can still yield different commits mid-run — the inconsistency this change aims to prevent.
Reviewed by Cursor Bugbot for commit ce80a5c. Configure here.
| f"{stderr.decode().strip()[-500:]}" | ||
| ) | ||
| if stdout.strip(): | ||
| version = stdout.split()[0].decode() |
There was a problem hiding this comment.
Annotated tags resolve to tag objects
Medium Severity
For annotated tags, git ls-remote emits the tag-object OID first and the peeled commit on a following ^{} line. Taking stdout.split()[0] locks the tag object, and git checkout of that OID fails because it is not a tree-ish.
Reviewed by Cursor Bugbot for commit ce80a5c. Configure here.
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. This PR introduces new async version resolution logic with subprocess calls. Multiple unresolved comments identify potential bugs: the lock may not span rollouts as intended, annotated tags may fail, and missing git on host causes crashes. These issues warrant human review before merge. You can customize Macroscope's approvability policy. Learn more. |
samsja
changed the title
1 participant
Summary
Replaces the #2374/#2375 stack as a single PR on main. Net effect vs current main:
RLMHarnessConfig.versiongoes from pinned commit41739cfback tomainversionoverrides skip resolution and pass through untouchedrlm-harness->nano-rlmrename from Pin RLM harness to nano-rlm #2373 is keptWhy
Implements the behavior agreed in Slack: always use the latest harness, but the harness must not change once a run has started. Tracking
mainper-rollout meant a push to nano-rlm mid-eval could make rollouts of the same run use different harness versions; a frozen default commit goes stale. Resolvingmainat run start gives latest and internally consistent, with explicit pins still available for reproducibility.Validation
uv run ruff check/ruff format --check— passuv run pytest tests/v1 -k rlm— pass (e2e skipped, needsPRIME_API_KEY)mainresolves to a 40-char SHA, repeated + concurrent calls return the same commit, full-SHA overrides pass through unresolved🤖 Generated with Claude Code
Note
Lock RLM harness to a resolved commit at run start and rename repo to nano-rlm
_resolve_versiontoRLMHarnessin harness.py that resolves branch/tag refs to a 40-hex commit once per run usinggit ls-remoteagainst thenano-rlmrepo.asyncio.Lockto ensure single-flight resolution, caching the result in_resolved_versionfor concurrent rollouts.RLMHarness.setupnow checks out the resolved commit instead of the original (potentially moving) ref.versioninRLMHarnessConfigchanges from a pinned commit to'main'.mainunless they explicitly set a version.Macroscope summarized ce80a5c.