Skip to content

refactor(cookbooks): run tools with runTools() and stream raw chunks - #1261

Merged
vishxrad merged 2 commits into
visharad/cookbooks-gateway-conversationsfrom
visharad/cookbooks-run-tools
Sep 29, 2026
Merged

vishxrad merged 2 commits into
visharad/cookbooks-gateway-conversationsfrom
visharad/cookbooks-run-tools

Conversation

@vishxrad

@vishxrad vishxrad commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1258. The cookbooks run their tools with the OpenAI SDK's runTools() instead of the hand-written tool loop, and stream raw completion chunks to Agent Interface, following the self-hosted template (route, page).

Summary

  • Tool loop removed. Each cookbook's src/lib/tool-loop.ts is gone (about 150 lines each). The chat route calls gateway.chat.completions.runTools() with the tool's existing JSON schema plus its executor, typed with the SDK's own runnable-tool types (no casts).
  • Raw chunks to the browser. The route forwards the runner's completion chunks as server-sent events, the format Gateway streams. The page reads them with openAIAdapter() and openAIMessageFormat, as in the template, so the route no longer builds AG-UI events.
  • Same round caps. maxChatCompletions is 3, 4 and 3, matching the old maxRounds.
  • Same first-round errors. The route waits for the first completion before responding (runner.emitted("connect") raced with runner.done()), so a rejected key or rate limit still returns Gateway's HTTP status.
  • Docs. The step 4 and step 5 code samples, READMEs, file tables, and tool module comments describe runTools() and openAIAdapter().

What changes for readers

  • No tool results in Behind the scenes. Chat Completions has no chunk for a tool result, and openAIAdapter() reads only text and tool-call chunks. Behind the scenes shows each call with its arguments, but the call stays at "Calling …" with no result. The docs and Verify steps say so.
  • Tool arguments stream in, as they do from Gateway.
  • Parallel tools. Tools the model calls in one completion run in parallel, and results go back to the model in call order.
  • Tool errors. runTools() ends the run when a tool throws. Each route wraps its executor so the error goes back to the model as the tool result, as the old loop did.
  • Later errors are generic. An error after the first completion ends the stream, and Agent Interface shows "Something went wrong: Failed to fetch" instead of Gateway's message.
  • No forced final answer. The old loop set tool_choice: "none" on its last round. maxChatCompletions just stops, so a run that hits the cap on a tool call ends without an answer.

Docs correction

The pages said "a rejected key, rate limit, or unknown model returns as an HTTP error". An unknown model doesn't: Gateway accepts it with 200 and fails inside the stream, with the old loop too. The pages now say "a rejected key or rate limit".

Test plan

  • tsc --noEmit in all three cookbooks
  • npm run verify in all three cookbooks, on the first commit (before switching to raw chunks)
  • npm run verify after switching to raw chunks
  • Analytics, raw chunks, in the browser: the fastest-laps starter streams the pre-tool text, the query_lap_times call with its arguments, and the table. The call has no result.
  • Document comparison, raw chunks, in the browser: two search_documents calls in one round
  • Analytics, raw chunks, invalid OPENUI_MODEL: Agent Interface shows "Something went wrong: Failed to fetch"
  • Invalid THESYS_API_KEY: /api/chat returns HTTP 401 with Gateway's message (tested on the first commit; this path is unchanged)

🤖 Generated with Claude Code

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
openui-docs Ready Ready Preview Sep 29, 2026 10:03am UTC

Request Review

@vishxrad
vishxrad marked this pull request as ready for review September 29, 2026 07:47
@vishxrad
vishxrad added this pull request to stack #1262 September 29, 2026 07:48
@vishxrad
vishxrad force-pushed the visharad/cookbooks-run-tools branch from 4d89eed to e1efce0 Compare September 29, 2026 09:37
@vishxrad
vishxrad force-pushed the visharad/cookbooks-run-tools branch from e1efce0 to 964f3da Compare September 29, 2026 09:43
@vishxrad vishxrad changed the title refactor(cookbooks): run tools with the OpenAI SDK's runTools() refactor(cookbooks): run tools with runTools() and stream raw chunks Sep 29, 2026
vishxrad and others added 2 commits September 29, 2026 15:30
Replace each cookbook's hand-written Chat Completions tool loop
(src/lib/tool-loop.ts) with gateway.chat.completions.runTools(). The
chat route passes the tool's existing JSON schema with its executor
and translates the runner's events into the same AG-UI events, so
Agent Interface still shows each call and its result.

Behavior differences from the old loop:
- A tool's arguments reach the browser when the completion that calls
  it finishes, instead of streaming in.
- Tools the model calls in one completion run in parallel.
- runTools() ends the run when a tool throws, so each executor returns
  its error to the model as the tool result instead.
- maxChatCompletions has no final text-only round, so a run that hits
  the cap on a tool call ends with a RUN_ERROR instead of an answer.

The route still waits for the first completion before responding, so
a rejected key returns Gateway's HTTP status. The docs no longer say
an unknown model does: Gateway accepts it with 200 and fails inside
the stream, with the old loop too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow the self-hosted template: the chat route forwards the runTools()
runner's completion chunks as server-sent events, the format Gateway
streams, and Agent Interface reads them with openAIAdapter() instead of
AG-UI events built in the route. This drops the event mapping from each
route.

The route still waits for the first completion before responding, so a
rejected key or rate limit returns Gateway's HTTP status.

What changes for readers, and the docs now say so:
- Tool call arguments stream in again.
- Chat Completions has no chunk for a tool result, so Behind the scenes
  shows each call's arguments but not its result.
- A later error ends the stream, and Agent Interface reports that the
  request failed instead of showing Gateway's message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vishxrad
vishxrad force-pushed the visharad/cookbooks-run-tools branch from 1b86cae to e63b315 Compare September 29, 2026 10:01
@vishxrad
vishxrad merged commit e63b315 into main Sep 29, 2026
8 checks passed
@vishxrad
vishxrad deleted the visharad/cookbooks-run-tools branch September 29, 2026 10:11

This branch was successfully deployed

1 active deployment
Preview — e63b3151 Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant