fix: let bare Tools compete directly in delegate selection (not just as a last resort) - #195
Open
imaustink wants to merge 2 commits into
Open
fix: let bare Tools compete directly in delegate selection (not just as a last resort)#195imaustink wants to merge 2 commits into
imaustink wants to merge 2 commits into
Conversation
"Is airvinyl running okay" matched claude-code-swe-agent -- the only
Agent in the catalog, description broad enough ("runs bash...") to
loosely overlap via embedding similarity -- instead of the ssh tool,
and hung on that agent's Claude identity-link gate (a known class of
pre-existing bug in this repo: fix/claude-auth-submit-hang et al.).
Nothing about that path is specific to ssh; any bare Tool with no
wrapping Skill is exposed to the same mis-routing risk once a broad
Agent exists in the catalog.
Adds ssh-skill (mirroring cluster-debug-skill's shape: toolRefs: [ssh],
no allowedRoles of its own per ADR 0011), so selectDelegate's combined
skill+agent choice sees a strong, specific match for SSH-shaped
requests and picks it over the vague agent match. Its markdown also
gives authored guidance for turning an open-ended "is it healthy?"
into a concrete sequence of read-only diagnostic calls (uptime, df -h,
free -m, then a named service if implied) -- separately addressing
why a vague request was declining at the ActionPlanner stage even when
the ssh tool WAS the right fallback candidate.
Verified with a real kind cluster + kubectl apply --dry-run=server.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg8b9pPWm91nLbDnB6ECJh
…tion Fixes the actual root cause behind ssh-skill being needed at all: a bare Tool with no wrapping Skill was only ever considered as a last resort (noMatchFallback's selectFallbackTool), invoked only once skillCandidates AND agentCandidates both came up empty. Any Agent whose description loosely overlapped a request via embedding similarity alone -- not because it was actually the better fit, but because a Tool never got the chance to compete at all -- would win by existing as a candidate. That's what let claude-code-swe-agent (the only Agent in the catalog) absorb "SSH into X" requests instead of the ssh Tool, hanging on its identity-link gate. Adds a `retrieveTools` graph node (embedding query + ToolFitChecker, reusing selectFallbackTool's own two-stage relevance gate) alongside retrieveSkills/retrieveAgents, and extends DelegateSelector to a three-way choice among skills/agents/tools in one combined decision (docs/adr/0036). Guarded on deps.delegateSelector being configured, so non-NATS deployments pay no extra cost and keep the exact old skill-only + selectFallbackTool behavior. ssh-skill (this branch's other commit) stays -- it still adds value a bare Tool alone can't: authored guidance for turning an open-ended "is it healthy?" into a concrete sequence of diagnostic calls. This fixes the starvation itself so no *future* unwrapped Tool needs a Skill just to be reachable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fg8b9pPWm91nLbDnB6ECJh
imaustink
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #192. After that PR deployed, "Can you SSH into airvinyl and see if it's running all good?" in Open WebUI got a generic LLM fallback with no tool use, and a follow-up with a specific command ("SSH into airvinyl and run uptime") got stuck showing "found 1 agent candidate" indefinitely.
Root cause, confirmed against the live cluster:
claude-code-swe-agentis the onlyAgentin the whole catalog, and its broad description ("Performs software-engineering work on GitHub end-to-end. Runs the Claude Code CLI headless — which has bash, file-read/write, grep, and glob tools...") loosely matched via embedding similarity.selectDelegate(apps/agent-orchestrator/src/agent/graph.ts) only ever considered a bareTool(likessh, with no wrappingSkill) as a last resort — reachable only onceskillCandidatesANDagentCandidatesboth came up empty. Since an Agent candidate always existed, thesshTool never got a chance to compete at all, and dispatching to the agent hung on its Claude identity-link gate (a known, pre-existing bug class in this repo —fix/claude-auth-submit-hanget al. — unrelated to #192's code).Fix (two parts)
1.
ssh-skill(charts/community-components/templates/skill-ssh.yaml) — mirrorscluster-debug-skill's shape (toolRefs: [ssh]). GivesselectDelegatea strong, specific match for SSH-shaped requests so it wins over the vague agent overlap, and its authoredmarkdowngives explicit guidance for turning an open-ended "is it healthy?" into a concrete sequence of read-only diagnostic calls.2. The actual root cause (
docs/adr/0036-tools-compete-directly-in-delegate-selection.md) — a bare Tool was starved out categorically, not because it lost a comparison, but because it was never in the comparison. Any future unwrapped Tool would hit the identical problem the moment a broad Agent (or Skill) exists in the catalog. Fixed by:retrieveToolsnode (apps/agent-orchestrator/src/agent/graph.ts), inserted betweenretrieveAgentsandselectDelegate: runs the same embedding query +ToolFitCheckertwo-stage relevance gateselectFallbackToolalready used, reused rather than reinvented. Guarded ondeps.delegateSelectorbeing configured, so non-NATS deployments pay no extra cost and keep the exact old behavior.DelegateSelector.select(apps/agent-orchestrator/src/agent/delegate-selector.ts) now takes a thirdtoolsparameter and makes one combined three-way choice (skill/agent/tool), with an explicit preference order in its prompt: skill (authored guidance) > bare tool (single well-defined action) > agent (open-ended/multi-step work).selectDelegate's tool branch reuses a new shared helper (planFallbackToolCall, extracted fromselectFallbackTool's own tail) to construct the actual tool call, falling back to the oldnoMatchFallback/selectFallbackToolsafety net if the planner declines or the combined choice comes back empty.ssh-skillis kept alongside the core fix rather than superseded by it — it still adds value a bare Tool alone can't (the authored multi-step diagnostic guidance).Test plan
npm run typecheck --workspace=agent-orchestratornpm run build --workspace=agent-orchestratornpm run test --workspace=agent-orchestrator— 579/579 passing, including 4 new tests covering: tool candidates reach the delegate selector already fit-checked, a tool wins over an agent candidate when the selector says so (agent never launched), a tool that failsToolFitCheckeris excluded from what's offered to the selector, and thenoMatchFallbacksafety net still works when the combined selector picks nothing.helm lint charts/community-componentshelm templateagainstvalues.yamldefaults,values-production.yaml, andvalues-ci-all.yaml(verified againstvalidate-crds.yml's literal "every template rendered" assertion).kindcluster +kubectl apply --dry-run=serveragainst the full rendered catalog —skill.core.controller-agent.dev/ssh-skill created (server dry run).🤖 Generated with Claude Code
https://claude.ai/code/session_01Fg8b9pPWm91nLbDnB6ECJh