Skip to content

fix(escalation): require fresh consistent failure evidence - #637

Draft
pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-fresh-evidence
Draft

pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-fresh-evidence

Conversation

@pst2154

@pst2154 pst2154 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • deduplicate equivalent Terminus raw-command and structured-tool serialization only in the escalation judge transcript
  • require fresh, same-category failure evidence across distinct turns before switching
  • preserve the maximum-one-switch and post-switch latch invariants
  • add focused Rust coverage and a 40-task paired SWE-Atlas report

Benchmark evidence

On 20 RF and 20 TW tasks, the patched router scored 15/40 versus 10/40 for the original router and reduced switches from 9 to 5. Under the documented cached-token-aware synthetic rates, estimated cost fell from $1.171/task to $0.889/task. Direct GLM also scored 15/40 at $0.501/task, so this PR claims a false-positive fix and improvement over the original escalation policy, not superiority over the best fixed model.

Full methodology, task-level observations, formulas, limitations, and validation evidence are in benchmark/SWE_ATLAS_ESCALATION_REPORT.md.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace --all-features
  • uv run ruff check .
  • uv run mypy switchyard
  • uv run maturin develop --uv
  • uv run pytest tests/ -v

All passed on Rust 1.96.1. Pytest reported 115 passed, 2 deselected, and 2 subtests passed.

Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Signed-off-by: Alex Steiner <asteiner@nvidia.com>
@pst2154
pst2154 force-pushed the codex/escalation-fresh-evidence branch from 979c498 to c6f2dfc Compare September 18, 2026 22:32
@linj-glitch

Copy link
Copy Markdown
Contributor

I reviewed the latest rebased head. The fresh-evidence and same-category confirmation design makes sense, and the transcript-only Terminus deduplication is a good fit for the observed false positive. A few comments before merge:

  1. The current head does not compile under cargo check -p switchyard-libsy. crates/libsy/src/algorithms/escalation.rs:170 calls JudgeClassifier::verdict without the new judge_models argument, and crates/libsy/src/algorithms/util/escalation.rs:382 calls collect_text without its new tool_commands argument. These look like omissions from the latest rebase; the validation recorded in the report appears to predate that head.

  2. crates/libsy/src/algorithms/escalation.rs:177-182 logs the judge-generated reason at INFO. Bounding and stripping control characters prevents log injection but does not prevent the reason from echoing sensitive transcript content. I would keep category and new_evidence at INFO and either omit the reason or retain it only at DEBUG.

  3. I read the benchmark as good evidence for fixing the concrete duplicate-serialization false positive, but not yet as evidence that escalation beats a fixed model generally. It is one stochastic trial per task; patched escalation ties direct GLM at 15/40 while remaining materially more expensive, and some switches occur very late. The report mostly states these limitations clearly.

Overall, the #637 direction is sound. The two build errors and logging level are the actionable items I found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants