feat(google): native tool calling through Open WebUI's tool loop (1.18.0)

[owndev]

Problem

Native tool calling in the Gemini pipeline was broken on Open WebUI 0.10+ (#169, #155):

• The pipe passed Open WebUI's __tools__ callables to google-genai's automatic function calling (AFC), which runs the tools inside the pipe:
  • direct tools have no callable (KeyError: 'callable');
  • MCP callables have no __signature__ (AttributeError);
  • OpenAPI and MCP callables are all named tool_function (Duplicate function declaration);
  • Open WebUI loads plugins with from __future__ import annotations, so workspace tools carry string annotations that AFC cannot coerce.
• The built-in tools that Open WebUI adds to every browser chat were never declared.
• Tool approval, citations, embeds and the iteration limit of Open WebUI's tool loop were bypassed.
• API clients sending tools got no tool_calls back. Tool messages in the history were turned into user text.

Change (Google Gemini 1.17.0 → 1.18.0)

• No AFC. Tools are declared from body["tools"], the OpenAI tool specs Open WebUI builds for every tool type, as FunctionDeclaration(parameters_json_schema=…) with a sanitised schema. Names Gemini does not accept get a stable hash suffix and are mapped back. AutomaticFunctionCallingConfig(disable=True) everywhere.
• Open WebUI runs the tools. Gemini's function calls are returned as OpenAI tool_calls. On the browser path, Open WebUI's tool loop executes them and calls the pipe again with assistant.tool_calls + role:"tool" messages. That covers built-in, workspace Python, MCP, OpenAPI, terminal and direct tools, plus tool approval. Browser stream=false answers a tool round as a stream so the loop can run.
• Thought signatures travel as reasoning_details and are restored per call id in later rounds and turns. When a client does not echo them, the documented placeholder is used, only for the current turn.
• History conversion. Tool calls and tool results become function_call / function_response parts (responses in call order, {"error": …} for Open WebUI's error results). Unmatched calls and results are dropped with a WARNING. Open WebUI's tool-image message is handled.
• API clients get tool_calls streamed and non-streamed with finish_reason: "tool_calls", including through the openai SDK's stream reader. tool_choice is mapped (none, required, a named function); parallel_tool_calls is ignored.
• Grounding with tools. Gemini 3 on the Gemini API sends Google Search + URL context + functions with include_server_side_tool_invocations. Gemini 2.x and Vertex send grounding only, with no declarations for that request. Image models, background tasks and Legacy mode send no tools.
• Errors. MALFORMED_FUNCTION_CALL / UNEXPECTED_TOOL_CALL give a clear message instead of an empty answer.
• Requirements. google-genai>=1.68.0, google-genai<3, the first version with server-side tool invocations. The docs explain what to do on offline Open WebUI 0.9.0–0.11.3, where pip cannot upgrade the bundled 1.66.0.
• No valve was added, renamed or removed. No-tool chats, image models, Veo, tasks, retries and statuses are unchanged.
• Docs: the docstring, a new "Native tool calling" section in docs/google-gemini-integration.md, and README.

E2E

• New groups:

  • gemini.tools: the browser path through Open WebUI's real tool loop;
  • gemini.toolsapi: API clients.

  With them the gemini suite grows from 81 to 129 checks. On 1.17.0, 42 of the 48 tool checks fail; the commit body of 04f8a86 lists them per root cause.

• New mocks: the MOCKTOOLS: directive in the Gemini mock, which emulates Gemini's 400 validations for pairing, signatures and names; mock_tools.py (an OpenAPI server and an MCP server); a workspace tool.

• Harness fix: the browser session now uses the Socket.IO namespace sid and sends a heartbeat. Before, direct tools and __event_call__ failed with "Client session disconnected".

• Real-API smoke runner (manual, needs a key): GOOGLE_API_KEY=… tests/e2e/smoke/gemini-tools.sh. It runs one scenario per spec risk R1–R15 through the real tool loop and reports PASS, FAIL or SKIP per risk. Retries are bounded, there is a hard request cap, and the key never appears in any output.

Also fixed here: image edit loses the previous image (#194)

Since 1.15.2, a generated image in a browser chat is attached to the answer only as an Open WebUI file. Open WebUI 0.11.x never passes files of earlier assistant messages to a pipe. An edit request ("change the name on the certificate") therefore reached Gemini as text only, and Gemini drew an unrelated image. 1.17.0 and the earlier 1.18.0 state of this branch behaved the same way.

• Saved chats: for a saved chat of the requesting user (or any chat for an admin), the image path loads the message chain from Open WebUI's database, as load_messages_from_db does. It sends the images of earlier user and assistant messages as history images: file ids, /api/v1/files/<id>/content, data: URLs and markdown links.
• Fallback: API clients, temporary chats, background tasks and database errors keep using the request's messages. Earlier model turns are not replayed, so no thought signatures are involved.
• Image limit and dedup: over IMAGE_HISTORY_MAX_REFERENCES the oldest history images are dropped first and current images are never dropped. Duplicates keep their newest position, and an earlier image that is attached again to the current message is not sent twice. Guided regeneration keeps the images of the regenerated message.
• Hardening (side finding): _fetch_file_as_base64 used to read any file id in a message. It now reads a file only for its owner or an admin, on every path including Veo image-to-video, and logs no foreign file id. Image decoding runs in a worker thread.
• E2E: new group gemini.imgedit (19 checks: two-turn edit on Nano Banana 2 and gemini-nano-banana-2.1, uploads, guided regeneration variants, limit, dedup, temporary chat, task, admin, foreign file ids on the browser and API path, a database error, Veo). 15 of the 19 fail on the pre-fix file. A first round of mutation testing left 10 mutants alive; the tests were extended, and all 18 mutants are killed now.

Verification

Run	Result
After merging main 4b80e24 (#195) and #191: all suites, v0.11.4-slim normal and --strict-known, v0.11.3-slim --strict-known	530 PASS, 0 FAIL, 1 KNOWN (infomaniak-name-prefix, no fix yet), 531 checks, each
All suites, v0.11.4-slim, --strict-known (merged with main 3ff6cf9)	363 PASS, 0 FAIL, 1 KNOWN (infomaniak-name-prefix, no fix yet), 364 checks
gemini suite, v0.11.3-slim and v0.11.4-slim, --strict-known	149 PASS, 0 FAIL, 0 KNOWN each (on 0.11.3, pip upgrades the bundled google-genai 1.66.0 on save)
Meta-test (E2E_MOCK_FAULT=500), after the merges	57 PASS, 370 FAIL, 0 KNOWN
Offline unit/integration script, google-genai 1.68.0 and 2.29.0	97/97 each
Mutation testing	first round: 23 surviving mutants, the tests extended; 27/27 mutants killed afterwards
Smoke runner --dry-run-mock	15 PASS, 0 FAIL, 2 SKIP; bad key → exit 2 "API key not valid"; budget cap holds

The design is based on a probe of Open WebUI 0.11.4's tool loop: continuation shapes, reasoning_details round trip, the required index, merged rounds, usage summing. Three reviews followed: spec and API, Open WebUI regressions, and mutation testing. All 19 findings were fixed.

Before release: run the real-API smoke test

Everything above ran against mocks. These are open risks R1–R15 that only Google's servers can answer:

• acceptance of the sanitised schemas;
• the skip_thought_signature_validator placeholder;
• tool combination with replayed server-side parts;
• history without declarations.

Please run GOOGLE_API_KEY=… tests/e2e/smoke/gemini-tools.sh; about 39 small requests, or add --retries 0.

Closes #169
Closes #155
Closes #194

Supersedes #168 (thanks @ctroche99) and #159 (thanks @olivier-lacroix). Both addressed parts of this. This PR moves tool execution to Open WebUI's own tool loop instead of running tools inside the pipe.

🤖 Generated with Claude Code