Skip to content

feat: add auto approval and auto preset selection - #157

Merged
HugoRCD merged 6 commits into
mainfrom
feat/sdk-auto-approval-presets
Sep 24, 2026
Merged

HugoRCD merged 6 commits into
mainfrom
feat/sdk-auto-approval-presets

Conversation

@HugoRCD

@HugoRCD HugoRCD commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

🔗 Linked issue

None.

📚 Description

Adds 'auto' modes for approval and presets to the SDK and the eve extension. An evaluation model decides per message through AI SDK experimental_evaluate. The default is TypeSafe's Jev ('typesafe-ai/jev' through AI Gateway).

createGithubAgent({ model: 'anthropic/claude-opus-5.5', preset: 'auto', requireApproval: 'auto' })
githubExtension({ preset: 'auto', requireApproval: 'auto' })

requireApproval: 'auto' (createGithubTools, createGithubAgent, eve extension)

  • Before a call to a low-risk write tool (AUTO_APPROVAL_TOOLS: labels, assignees, reactions, comments, review-thread replies, reviewer requests, notification reads, workflow re-runs), the model rates the risk (0–2) and whether the latest user message asked for this exact call. The call runs without approval when risk <= 1 and intent >= 0.6; otherwise the user is asked.
  • Other write tools keep requiring approval. Per-tool values accept 'auto': { updateIssue: 'auto', mergePullRequest: true }.

preset: 'auto' (createGithubAgent, eve extension)

  • One evaluation call per user message, one yes/no question per preset. Presets at or above 0.7 are used, at most two. Read-only repo-explorer is only used when no other preset qualifies. When none qualifies, the most likely preset is used. The full catalog is never exposed.
  • In the eve extension, routing runs once per user message, and tools already called in the session stay registered so a parked approval resumes after routing changes. eve's [Pending approvals] resume message is not treated as a user request.

Failure handling: when the evaluation call fails (model not enabled on AI Gateway, no credits, outage), 'auto' approval asks and preset: 'auto' uses repo-explorer. Both log a github_tools.EVALUATION_FAILED warning with the gateway error as cause.

Configuration: evaluation: { model, maxRisk, minIntent, minPresetProbability, maxPresets } tunes both modes (defaults 1, 0.6, 0.7, 2).

Fix: eve tool output formatters no longer crash on structured GitHub API errors (Cannot read properties of undefined in the next model step for listPullRequestFiles, getFileContent, getRepositoryTree, getPullRequestContext, getCommit, compareCommits).

Compatibility

  • 'auto' needs ai 7.0.105 or later; without experimental_evaluate it throws EVALUATION_UNAVAILABLE. The root entry still loads on ai 6. ai is now a static namespace import, because eve could not bundle the extension with the dynamic import('ai') ("Expected one bundled authored module").
  • createDurableGithubAgent does not accept 'auto'.
  • ToolOptions.needsApproval is widened to the AI SDK type, which also accepts a function. Non-breaking.

Benchmark (examples/eve-bench): eve evals on 12 GitHub tasks with anthropic/claude-opus-5.5 and Jev, writes stubbed. Against the full catalog, preset: 'auto' + requireApproval: 'auto' passed 12/12 tasks with 42% fewer tokens per task (107k to 62k), 34% lower cost ($0.164 to $0.108), and half the approval prompts (8 to 4). On 20 direct approval cases, Jev sent 10/10 unsafe writes to a human and auto-ran 8/10 safe ones. The defaults above were tuned on these runs. One run per configuration; costs vary 2–4× between runs.

Docs: eve extension, presets and approval-control guides, API reference, READMEs, published skill, AGENTS.md, examples/eve now uses both 'auto' modes. Changesets: minor for @github-tools/sdk and @github-tools/eve-extension, patch for the formatter fix.

Verification: pnpm lint, pnpm typecheck, and pnpm test (130 tests) pass. Locally pnpm build fails only in examples/eve, where eve falls back to microsandbox without Docker running; main fails the same way on this machine.

📝 Checklist

  • I have linked an issue or discussion.
  • I have updated the documentation accordingly.

@vercel

vercel Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
github-tools-docs Ready Ready Preview, v0 Sep 24, 2026 8:15am UTC
github-tools-test-agent Ready Ready Preview, v0 Sep 24, 2026 8:15am UTC

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

Comment thread packages/github-tools/src/core/evaluation.ts
Comment thread packages/github-tools/src/core/evaluation.ts
Raise the preset threshold to 0.7, keep repo-explorer out of preset combinations, and loosen approval thresholds to maxRisk 1 / minIntent 0.6. A failed evaluation call now asks for approval or uses repo-explorer and logs EVALUATION_FAILED. Structured GitHub API errors pass through eve formatters.
@HugoRCD
HugoRCD force-pushed the feat/sdk-auto-approval-presets branch from 28c15bb to 1cb6609 Compare September 24, 2026 07:59
@HugoRCD HugoRCD changed the title feat(sdk): add auto approval and auto preset selection feat: add auto approval and auto preset selection Sep 24, 2026
Comment thread AGENTS.md Outdated
@HugoRCD
HugoRCD merged commit 90fd219 into main Sep 24, 2026
15 checks passed
@HugoRCD
HugoRCD deleted the feat/sdk-auto-approval-presets branch September 24, 2026 08:17
@github-actions github-actions Bot mentioned this pull request Sep 24, 2026

This branch was successfully deployed

2 active deployments
Preview – github-tools-docs — c7efea80 Deployed Sep 24, 2026 by vercel[bot]
Preview – github-tools-test-agent — c7efea80 Deployed Sep 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant