Extend native tool interception across the harness fleet - #2392
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| if (hook.hook_event_name === "PreToolUse") { | ||
| return { | ||
| hookSpecificOutput: { | ||
| hookEventName: "PreToolUse", | ||
| permissionDecision: "deny", | ||
| permissionDecisionReason: contentText(content), | ||
| }, | ||
| }; |
There was a problem hiding this comment.
🟡 Medium claude_code/tool_hook.mjs:190
A PreToolUse rewrite with array content is converted into a newline-joined string, so text arrays lose their shape and image parts become JSON text; Claude therefore records a result different from the policy-approved message. Reject non-string pre-tool rewrites instead of passing them through contentText, allowing the hook to fail closed.
if (hook.hook_event_name === "PreToolUse") {
+ if (typeof content !== "string") {
+ throw new Error("Claude cannot preserve the rewritten PreToolUse content");
+ }
return {
hookSpecificOutput: {
hookEventName: "PreToolUse",
permissionDecision: "deny",
permissionDecisionReason: contentText(content),
},
};🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/claude_code/tool_hook.mjs around lines 190-197:
A `PreToolUse` rewrite with array content is converted into a newline-joined string, so text arrays lose their shape and image parts become JSON text; Claude therefore records a result different from the policy-approved message. Reject non-string pre-tool rewrites instead of passing them through `contentText`, allowing the hook to fail closed.
| } | ||
| client = AsyncOpenAI(base_url=args.base_url, api_key=args.api_key) | ||
| tool_client = ( | ||
| httpx.AsyncClient(timeout=httpx.Timeout(None, connect=5.0)) |
There was a problem hiding this comment.
🟠 High browser_use/program.py:373
run_tool_hook waits indefinitely when the interception endpoint accepts the connection but never responds, so a stalled service hangs the browser-use rollout instead of failing within a bounded timeout. httpx.Timeout(None, connect=5.0) disables the read, write, and pool timeouts; set a finite overall timeout while retaining the 5-second connect timeout.
| httpx.AsyncClient(timeout=httpx.Timeout(None, connect=5.0)) | |
| httpx.AsyncClient(timeout=httpx.Timeout(60.0, connect=5.0)) |
Also found in 1 other location(s)
verifiers/v1/harnesses/null/program.py:203
tool_clientuseshttpx.Timeout(None, connect=5.0), disabling read/write/pool timeouts. An interception endpoint that accepts a connection and then stalls causesrun_tool_hookto wait indefinitely, hanging the null-harness rollout instead of failing closed within a bounded time.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/browser_use/program.py around line 373:
`run_tool_hook` waits indefinitely when the interception endpoint accepts the connection but never responds, so a stalled service hangs the browser-use rollout instead of failing within a bounded timeout. `httpx.Timeout(None, connect=5.0)` disables the read, write, and pool timeouts; set a finite overall timeout while retaining the 5-second connect timeout.
Also found in 1 other location(s):
- verifiers/v1/harnesses/null/program.py:203 -- `tool_client` uses `httpx.Timeout(None, connect=5.0)`, disabling read/write/pool timeouts. An interception endpoint that accepts a connection and then stalls causes `run_tool_hook` to wait indefinitely, hanging the null-harness rollout instead of failing closed within a bounded time.
| converted.every((part) => part.type === "text") && | ||
| converted.reduce((size, part) => size + part.text.length, 0) > | ||
| CLAUDE_MAX_INLINE_TEXT |
There was a problem hiding this comment.
🟡 Medium claude_code/tool_hook.mjs:89
An MCP result containing an image and more than 50,000 characters of text is approved inline even though Claude persists oversized text after the hook, so the model-visible result diverges from the canonical trace. The size check is gated by converted.every((part) => part.type === "text"); enforce the text-length limit across mixed image/text content as well.
- converted.every((part) => part.type === "text") &&
- converted.reduce((size, part) => size + part.text.length, 0) >
+ converted.reduce(
+ (size, part) => size + (part.type === "text" ? part.text.length : 0),
+ 0,
+ ) >🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/claude_code/tool_hook.mjs around lines 89-91:
An MCP result containing an image and more than 50,000 characters of text is approved inline even though Claude persists oversized text after the hook, so the model-visible result diverges from the canonical trace. The size check is gated by `converted.every((part) => part.type === "text")`; enforce the text-length limit across mixed image/text content as well.
| hook.tool_response.isImage === false && | ||
| hook.tool_response.persistedOutputPath === undefined | ||
| ) { | ||
| return claudeContent(hook.tool_response.stdout, hook.tool_name); |
There was a problem hiding this comment.
🟡 Medium claude_code/tool_hook.mjs:113
The structured Bash allow path accepts stdout values from 30,001 through 50,000 characters, even though Bash preservation is limited to 30,000, so Claude can persist or transform the approved result and diverge from the canonical trace. This path sends stdout through claudeContent, which applies the generic 50,000-character limit; enforce CLAUDE_BASH_MAX_INLINE_TEXT before that call.
- return claudeContent(hook.tool_response.stdout, hook.tool_name);
+ if (hook.tool_response.stdout.length > CLAUDE_BASH_MAX_INLINE_TEXT) {
+ throw new Error(`Claude will persist ${hook.tool_name}'s output after its hook`);
+ }
+ return claudeContent(hook.tool_response.stdout, hook.tool_name);🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/claude_code/tool_hook.mjs around line 113:
The structured Bash allow path accepts `stdout` values from 30,001 through 50,000 characters, even though Bash preservation is limited to 30,000, so Claude can persist or transform the approved result and diverge from the canonical trace. This path sends `stdout` through `claudeContent`, which applies the generic 50,000-character limit; enforce `CLAUDE_BASH_MAX_INLINE_TEXT` before that call.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b68ad58. Configure here.
| hook.tool_response.isImage === false && | ||
| hook.tool_response.persistedOutputPath === undefined | ||
| ) { | ||
| return claudeContent(hook.tool_response.stdout, hook.tool_name); |
There was a problem hiding this comment.
Claude Bash hook drops stderr
High Severity
claudeToolContent only forwards Bash stdout into the /tool after-hook, while the revived tool-interception example emits its failure marker on stderr only. On Claude Code that marker never reaches policy or the prepared result, so failure observation and the Claude e2e reward checks cannot pass.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b68ad58. Configure here.
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR introduces a significant new feature - native tool interception across multiple harnesses - with substantial runtime behavior changes and new integration points. Additionally, there are unresolved High-severity findings including potential bugs with stderr handling and indefinite timeout issues. Not approved because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
412a3f1 to
d22f55d
Compare
1 participant
Overview
Stacked on #2372. Extends synchronous native tool interception from Bash + Pi to every harness that owns a synchronous execution boundary, and documents why the rest cannot gate tools natively. Every wired hook blocks a proposed call before execution and runs post-result policy before the harness advances — a hook that cannot reach interception (or cannot preserve an approved result byte-exact) fails the rollout loudly instead of letting harness state diverge from the canonical trace.
Details
beforehook gates each call (a rewrite skips execution and records the policy's result),afterhook runs before the result is appended; the tool-policy bearer arrives over stdin viarun_with_input, never argv or env.tool_executionmiddleware (pre-execution gate) and atransform_tool_resulthook (post-execution policy). Transport failures and stops hard-exit the agent process, so a vetoed tool can never run. Project plugins are disabled so no task-owned code joins the credential-bearing process.settingSourcesis pinned touserso tasks cannot merge their own hooks. Deny carries a rewrite as the permission reason; PostToolUse rewrites viaupdatedToolOutputwith the content mappings bounded to shapes the pinned release preserves.before_tool_calland runs the result middleware; pre-execution rewrites are applied to the emitted synthetic result so OpenClaw records exactly the approved content. The plugin allowlist keeps workspace plugins out of the Gateway.install_tool_hookbootstrap (one-shot private credentials file, delivered over stdin) reused by Pi, Hermes, and OpenClaw; both Node hook stacks setNODE_USE_ENV_PROXYsofetchreaches interception under restricted networking.tool-interceptionexample environment (block → rewrite → observe-failure in one deterministic rollout) and the e2e rows for bash, claude-code, hermes, openclaw, and pi, including the restricted-network variants.Validation
ruff check/ruff format/pre-commit run --all-filesclean;pytest tests/v1 -m "not e2e"andtests/test_interception_utils.pypass.test_native_tool_interception,..._restricted) need a model key + Docker and were not run here; they are the per-pinned-version verification gate for the three CLI hooks.Note
High Risk
Large cross-harness change touching security-sensitive tool gating, credential handoff, and pinned third-party hook contracts; incorrect interception or hook mapping could let tools run when blocked or diverge traces from policy.
Overview
Extends synchronous native tool interception beyond Bash/Pi so rollouts can block tools before execution, rewrite results before the agent sees them, and observe failures on every harness that owns a real tool boundary. Harnesses that cannot gate tools natively get short in-code notes (Codex, Kimi Code, Pool, mini-swe-agent, Terminus, RLM).
In-repo chat loops (
null,browser-use) now call the rollout/toolpolicy on before and after each tool call; a pre-execution rewrite skips running the tool. The policy bearer is delivered via stdin (run_with_input), not argv or env.ACP agents wire native hooks/plugins per stack: Claude Code wraps the ACP adapter with SDK Pre/Post hooks and pins
settingSourcesto user; Hermes installs a generated plugin with fail-closedtool_execution/transform_tool_result; OpenClaw loads a Gateway plugin forbefore_tool_calland result middleware with an allowlisted plugin set. Sharedinstall_tool_hookdelivers one-shot credentials privately for Pi, Hermes, and OpenClaw.Adds the
tool-interceptionexample environment (three-step deterministic rollout + reward checks) and e2e rows for bash, claude-code, hermes, openclaw, and pi, including restricted Docker networking variants. Testconftestexposes the example package onPYTHONPATHalongside fixtures.Reviewed by Cursor Bugbot for commit b68ad58. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Extend native tool interception to browser-use, claude-code, hermes-agent, null, and openclaw harnesses
SUPPORTS_TOOL_INTERCEPTION=Trueandconfigure_tool_interceptionto browser-use, claude-code, hermes-agent, null, and openclaw harnesses; secrets are delivered securely over stdin using a length-prefix protocol.install_tool_hookhelper that writes hook source and credentials to a private 0600 file, returning aVF_TOOL_INTERCEPTION_CONFIGenv var;piharness is refactored to use it.tool-interceptionrollout environment with a deterministic 3-step task that exercises blocking, rewriting, and failure observation, plus e2e tests covering both standard and restricted (network-isolated) Docker placements.📊 Macroscope summarized b68ad58. 22 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.