diff --git a/docs/google-gemini-integration.md b/docs/google-gemini-integration.md index 283d1ac..c1ed008 100644 --- a/docs/google-gemini-integration.md +++ b/docs/google-gemini-integration.md @@ -289,6 +289,8 @@ VERTEX_AI_RAG_STORE="projects/your-project/locations/global/collections/default_ The Google Gemini pipeline supports configurable aspect ratios and resolutions for image generation with **Gemini 3/3.1 image models** (e.g., `gemini-3.1-flash-image`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image`, and the preview IDs `gemini-3.1-flash-image-preview`, `gemini-3-pro-image-preview`, `gemini-3-flash-image-preview`), the **Nano Banana** IDs (e.g., `gemini-nano-banana-2.1`) and the models listed in `GOOGLE_IMAGE_GENERATION_MODELS`. +To let a Gemini *text* model create or edit images in the middle of a chat through Open WebUI's own image engine instead, see [Open WebUI's image generation and code interpreter](#open-webuis-image-generation-and-code-interpreter). + > [!IMPORTANT] > **Model Compatibility**: The `aspect_ratio` and `image_size` parameters (ImageConfig) are **only supported by Gemini 3/3.1 and Nano Banana image models**. Gemini 2.5 image models (e.g., `gemini-2.5-flash-image-preview`) support image generation but do not support these configuration parameters. When using Gemini 2.5 image models, default aspect ratio and resolution will be used automatically. @@ -782,6 +784,35 @@ Gemini accepts function names of up to 64 characters (letters, digits, `_` and ` Image generation models, background tasks (title, tag and follow-up generation) and chats with **Function Calling** set to **Legacy** declare no functions. Legacy mode remains available: Open WebUI then selects and runs the tools itself before it calls the pipeline. +### Open WebUI's image generation and code interpreter + +In Native mode Open WebUI 0.11 offers its own image generation, image editing and code interpreter as built-in tools: `generate_image`, `edit_image` and `execute_code`. The pipeline declares them to a Gemini text model like any other tool; Gemini decides when to call them and Open WebUI runs them. No separate tool or function is needed. + +**Image generation and editing** with Open WebUI's Gemini image engine: + +1. **Admin Settings → Images**: switch on image generation, choose the engine **Gemini**, enter the API base URL `https://generativelanguage.googleapis.com/v1beta` (empty by default unless `GEMINI_API_BASE_URL` is set), a Gemini API key (default: `GEMINI_API_KEY`) and the model. The endpoint method decides the kind of model: `predict` (or empty) for an Imagen model such as `imagen-4.0-generate-001`, `generateContent` for a Gemini image model such as `gemini-2.5-flash-image`. As environment variables: `ENABLE_IMAGE_GENERATION=true`, `IMAGE_GENERATION_ENGINE=gemini`, `IMAGES_GEMINI_API_BASE_URL`, `IMAGES_GEMINI_API_KEY`, `IMAGE_GENERATION_MODEL`, `IMAGES_GEMINI_ENDPOINT_METHOD`. +2. For `edit_image`, also switch on image editing with the engine **Gemini**, the same base URL, a key and a Gemini image model (the edit engine always uses `generateContent`): `ENABLE_IMAGE_EDIT=true`, `IMAGE_EDIT_ENGINE=gemini`, `IMAGES_EDIT_GEMINI_API_BASE_URL`, `IMAGES_EDIT_GEMINI_API_KEY`, `IMAGE_EDIT_MODEL`. Without image editing only `generate_image` is offered. +3. In the chat, pick a Gemini text model and turn on the **Image** toggle of the message input. + +Gemini then calls `generate_image` with a prompt, or `edit_image` with a prompt and the URLs of the images to change. It finds those URLs in the chat: in a saved chat Open WebUI lists the files of each user message in an `` block of that message, and the result of an earlier `generate_image` or `edit_image` call holds the URL of its image. Open WebUI sends the request to its image engine, attaches the new image to the answer and returns its URL to Gemini, which writes the rest of the answer. + +**Code interpreter**: on by default (**Admin Settings → Code Execution**, `ENABLE_CODE_INTERPRETER`). Turn on the **Code Interpreter** toggle of the message input; Gemini can then call `execute_code`: + +- Engine `pyodide` (default): Open WebUI sends the code to the browser tab, which runs it with Pyodide, and returns stdout, stderr and the result to Gemini. Open WebUI adds a note on the Pyodide environment to the system prompt, which the pipeline sends as Gemini's system instruction. An image printed as a `data:image/png;base64,...` line (for example a plot) is uploaded and replaced with a Markdown image link. +- Engine `jupyter`: Open WebUI runs the code on the configured Jupyter server (`CODE_INTERPRETER_JUPYTER_URL`, ...); no browser tab is involved. + +**When Open WebUI offers these tools.** All of the following must hold (Open WebUI 0.11, `get_builtin_tools`): + +- The request comes from the web UI. API clients of `/api/chat/completions` do not get Open WebUI's built-in tools, toggles or not; they send their own `tools` (see [API clients](#api-clients)). +- Function calling is **Native** and the model capability **Builtin Tools** is on. +- Per tool: the global switch (image generation; image editing for `edit_image`; code interpreter), the model's capability (**Image Generation** / **Code Interpreter**) and its **Builtin Tools** category, the chat toggle, and the user permission **Features → Image Generation / Code Interpreter** (admins always have it). Capabilities that are not set count as on; pipeline models have none set until you edit them under **Admin Settings → Models**. + +**Limits** of this setup: + +- Open WebUI's Gemini image engine authenticates with an API key only (`x-goog-api-key` header). It has no Vertex AI login with service account or application default credentials, as the pipeline has; its key and base URL are separate from the pipeline's valves. +- The pipeline's own image models (marked 🎨, e.g. `gemini-3.1-flash-image`) stay available for direct use. Image models support no function calling, so the pipeline sends them no tools: with such a model selected the model makes the image itself, and Open WebUI's image engine is not called, even with the Image toggle on. +- The e2e group `gemini.owuitools` checks these round trips against the Gemini mock, the browser path of the test harness and a Jupyter mock (see the [testing guide](./testing.md#open-webuis-image-and-code-tools-geminiowuitools)). The requests Open WebUI's image engine sends to the real Gemini and Imagen APIs come from Open WebUI, not from this pipeline, and are not tested here. + ### Limitations - Open WebUI ends the tool loop after `CHAT_RESPONSE_MAX_TOOL_CALL_ITERATIONS` rounds (default 256). The pipeline adds no limit of its own. diff --git a/docs/testing.md b/docs/testing.md index 8a4ea68..abea27c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -48,7 +48,7 @@ Typical output (abridged): ```text starting owui-e2e-111759-1234 from ghcr.io/open-webui/open-webui:v0.11.4-slim -Open WebUI healthy after 30s (http://localhost:57765) +Open WebUI healthy after 19s (http://localhost:57765) Open WebUI 0.11.4, suites: gemini, azure, n8n, infomaniak, filters function versions: google_gemini.py 1.18.0, azure_ai_foundry.py 3.0.0, n8n.py 2.3.1, ... === gemini === @@ -57,24 +57,25 @@ function versions: google_gemini.py 1.18.0, azure_ai_foundry.py 3.0.0, n8n.py 2. ... [PASS ] gemini.tools.builtin built-in tool, browser stream=True: Open WebUI runs it, ... ... ---- gemini: 305s +--- gemini: 321s ... === infomaniak === ... [PASS ] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names, and changing it back restores them ... -SUMMARY: 531 PASS, 0 FAIL, 0 KNOWN in 1089s -total runtime: 1123s +SUMMARY: 541 PASS, 0 FAIL, 0 KNOWN in 1112s +total runtime: 1138s output: tests/e2e/out/20261009-111759-owui-e2e-111759-1234 ``` -A full run of all suites took about 19 minutes on a shared 8-CPU Docker host (1120-1169 s; -1123 s: 34 s container start-up, then gemini 305 s, azure 419 s, n8n 71 s, infomaniak -36 s, filters 256 s). Most of it is waiting: the azure `rag` group adds about 5.5 -minutes (query-generation timeouts, the query-generation pause and the 45 s retrieval -limit are waited for; in the meta-test, where every mock answers HTTP 500, its requests -fail at once), the gemini `tools`, `toolsapi` and `imgedit` groups about 2.5 -minutes and the filters `ingest` group (Logs Ingestion API) another 2.5 minutes. +A full run of all suites took about 19 minutes on a shared 8-CPU Docker host (1138 s +and 1177 s; 1138 s: 26 s container start-up, then gemini 321 s, azure 424 s, n8n +71 s, infomaniak 36 s, filters 259 s). Most of it is waiting: the azure `rag` group +adds about 5.5 minutes (query-generation timeouts, the query-generation pause and the +45 s retrieval limit are waited for; in the meta-test, where every mock answers HTTP +500, its requests fail at once), the gemini `tools`, `toolsapi`, `imgedit` and +`owuitools` groups about three minutes and the filters `ingest` group (Logs Ingestion +API) another 2.5 minutes. Network downloads on first use come on top (`pip install google-genai` when the Gemini function is created, the tiktoken encodings the `filters` suite caches before its first scenario). How many checks each suite has @@ -144,7 +145,7 @@ events), a **background title task** and that the server log has no unexpected | Suite | Function(s) | Mock | Groups (`--only .`) and file-specific scenarios | | --- | --- | --- | --- | -| `gemini` | `pipelines/google/google_gemini.py` (+ `google_search_tool`) | `mock_gemini.py` | `models` (image / video indicators, display names #172), `api` (thinking in `
`, full usage), `thinking` (summaries not replayed #176; budget, level, include and strip valves), `browser`, `tasks` (no `
` in task answers, no grounding tools for the tasks of a web_search chat), `image` (image models forced non-stream, exactly one saved file), `images` (thought images skipped, used as fallback, not used after IMAGE_SAFETY; dedup; two final images; image link for API clients; image history, where the limit keeps the current and the newest images and cuts more current images than it allows; optimization), `nano` (`gemini-nano-banana-2.1`), `imgvalve` (`IMAGE_GENERATION_MODELS`), `imgconfig` (ImageConfig valves, user valve, body), `imgtools` (tools per image model with web_search), `nostream` (`GOOGLE_STREAMING_ENABLED=false` with `stream=true`, #170), `video` (Veo: text and video saved, request shape, image-to-video), `grounding` (`google_search_tool` → googleSearch + urlContext, sources, `[1]` citations; no grounding without web_search), `vertex` (Vertex AI Search sources), `errors` (400, 500 with retry, blocked prompt, SAFETY finish, image error status, a streamed answer starting with `data:`), `retry` (`RETRY_COUNT` for streams), `status` (Stop leaves no running status), `valves` (model cache vs. valve changes, safety, whitelist, additional models, system prompt, user headers, API version, params, valve names and defaults), `concurrency` (forwarded user headers belong to the requesting user), `streamimg` (inline image in a stream), `imgedit` (image editing across turns, #194: in a saved chat the image of turn 1, attached as a file only, is sent with the edit request on `gemini-3.1-flash-image` and `gemini-nano-banana-2.1`; the follow-up task gets no images; uploads between generated turns in the default mode, with another image per turn (the mock's `final-image-` trigger) so their order shows; guided regeneration, also of an image-only message and of a text contained in the guidance; `IMAGE_HISTORY_MAX_REFERENCES` keeps the upload of the edit message, else the newest images; an image attached again is sent once and counts at its newest place (with `IMAGE_DEDUP_HISTORY=false` it is sent each time); older saved forms (a data: URL image file, markdown links to a file and to a data: URL; a text file is not sent); a temporary chat and a database error (injected by a test-only filter) take the history from the request; an image file of another user in a non-admin user's chat, in a markdown link on the API path and on the Veo path is not read (logged without its id), an admin continuing the user's chat reads it), `toolsapi` (native tool calling for API clients: client tools → `tool_calls` + `reasoning_details` streamed and non-streamed with `finish_reason` `tool_calls` (also as the openai SDK reads the stream), a text answer to a request with tools (`finish_reason` `stop`), `GOOGLE_STREAMING_ENABLED=false`, `MALFORMED_FUNCTION_CALL` / `UNEXPECTED_TOOL_CALL` streamed and non-streamed, continuation with and without signatures and with `reasoning_details` under `provider_specific_fields`, older turns with a thinking summary, empty arguments, content parts, a numeric id and a `function` that is not an object, `tool_choice` (also an undeclared name and with Search grounding on Gemini 3), name mapping (also a trailing newline) and duplicates, the `default_api.` prefix, schema clean-up, synthetic `owui_` ids, unchanged answers without tools), `tools` (native tool calling through Open WebUI's tool loop: built-in tool streamed and non-streamed, parallel calls (responses in call order), two rounds with summed usage, merged rounds without signatures, text before a call, thinking on / off, workspace Python tool, a tool result with an image, OpenAPI and MCP tool servers, direct tools of the browser, tool approval (approve, reject, two calls), follow-up turns on Gemini 3 / 2.5 / 3 and after an approved call, unknown tool, `MALFORMED_FUNCTION_CALL`, Search grounding with functions on Gemini 3 (the sources of both rounds, the next turn without web search) and grounding only on 2.5, title task, Legacy mode, `builtin_tools` off, `GOOGLE_STREAMING_ENABLED=false`; see [Native tool calling](#native-tool-calling-geminitools-geminitoolsapi)) | +| `gemini` | `pipelines/google/google_gemini.py` (+ `google_search_tool`) | `mock_gemini.py`, `mock_tools.py` (tool servers, Jupyter) | `models` (image / video indicators, display names #172), `api` (thinking in `
`, full usage), `thinking` (summaries not replayed #176; budget, level, include and strip valves), `browser`, `tasks` (no `
` in task answers, no grounding tools for the tasks of a web_search chat), `image` (image models forced non-stream, exactly one saved file), `images` (thought images skipped, used as fallback, not used after IMAGE_SAFETY; dedup; two final images; image link for API clients; image history, where the limit keeps the current and the newest images and cuts more current images than it allows; optimization), `nano` (`gemini-nano-banana-2.1`), `imgvalve` (`IMAGE_GENERATION_MODELS`), `imgconfig` (ImageConfig valves, user valve, body), `imgtools` (tools per image model with web_search), `nostream` (`GOOGLE_STREAMING_ENABLED=false` with `stream=true`, #170), `video` (Veo: text and video saved, request shape, image-to-video), `grounding` (`google_search_tool` → googleSearch + urlContext, sources, `[1]` citations; no grounding without web_search), `vertex` (Vertex AI Search sources), `errors` (400, 500 with retry, blocked prompt, SAFETY finish, image error status, a streamed answer starting with `data:`), `retry` (`RETRY_COUNT` for streams), `status` (Stop leaves no running status), `valves` (model cache vs. valve changes, safety, whitelist, additional models, system prompt, user headers, API version, params, valve names and defaults), `concurrency` (forwarded user headers belong to the requesting user), `streamimg` (inline image in a stream), `imgedit` (image editing across turns, #194: in a saved chat the image of turn 1, attached as a file only, is sent with the edit request on `gemini-3.1-flash-image` and `gemini-nano-banana-2.1`; the follow-up task gets no images; uploads between generated turns in the default mode, with another image per turn (the mock's `final-image-` trigger) so their order shows; guided regeneration, also of an image-only message and of a text contained in the guidance; `IMAGE_HISTORY_MAX_REFERENCES` keeps the upload of the edit message, else the newest images; an image attached again is sent once and counts at its newest place (with `IMAGE_DEDUP_HISTORY=false` it is sent each time); older saved forms (a data: URL image file, markdown links to a file and to a data: URL; a text file is not sent); a temporary chat and a database error (injected by a test-only filter) take the history from the request; an image file of another user in a non-admin user's chat, in a markdown link on the API path and on the Veo path is not read (logged without its id), an admin continuing the user's chat reads it), `toolsapi` (native tool calling for API clients: client tools → `tool_calls` + `reasoning_details` streamed and non-streamed with `finish_reason` `tool_calls` (also as the openai SDK reads the stream), a text answer to a request with tools (`finish_reason` `stop`), `GOOGLE_STREAMING_ENABLED=false`, `MALFORMED_FUNCTION_CALL` / `UNEXPECTED_TOOL_CALL` streamed and non-streamed, continuation with and without signatures and with `reasoning_details` under `provider_specific_fields`, older turns with a thinking summary, empty arguments, content parts, a numeric id and a `function` that is not an object, `tool_choice` (also an undeclared name and with Search grounding on Gemini 3), name mapping (also a trailing newline) and duplicates, the `default_api.` prefix, schema clean-up, synthetic `owui_` ids, unchanged answers without tools), `tools` (native tool calling through Open WebUI's tool loop: built-in tool streamed and non-streamed, parallel calls (responses in call order), two rounds with summed usage, merged rounds without signatures, text before a call, thinking on / off, workspace Python tool, a tool result with an image, OpenAPI and MCP tool servers, direct tools of the browser, tool approval (approve, reject, two calls), follow-up turns on Gemini 3 / 2.5 / 3 and after an approved call, unknown tool, `MALFORMED_FUNCTION_CALL`, Search grounding with functions on Gemini 3 (the sources of both rounds, the next turn without web search) and grounding only on 2.5, title task, Legacy mode, `builtin_tools` off, `GOOGLE_STREAMING_ENABLED=false`; see [Native tool calling](#native-tool-calling-geminitools-geminitoolsapi)), `owuitools` (Open WebUI's built-in `generate_image`, `edit_image` and `execute_code` called by a Gemini text model: declared only with the chat toggles and for browser sessions, `edit_image` only with image editing on; the image engine `gemini` with `generateContent` and with `predict` (Imagen), editing an upload and the image of the previous turn, the image saved with the answer and its URL sent back to Gemini; a pipe image model makes its own image; the code interpreter with pyodide (the browser runs the code) and with Jupyter; see [Open WebUI's image and code tools](#open-webuis-image-and-code-tools-geminiowuitools)) | | `azure` | `pipelines/azure/azure_ai_foundry.py` | `mock_azure.py` (requires `api-version`; answers prompts with a `` block like a plain model, without `context`; query generation, embeddings, tool calls, a ~300 KB and a > 4 MiB stream event, `content: null`), `mock_search.py` (Azure AI Search and an App Service managed identity token endpoint, see below) | `valves` (from 3.0.0 also the `AZURE_AI_SEARCH_*` valves, their defaults, enums and the password input of the key, and no `AZURE_AI_SEARCH_MODE`), `models` (`AZURE_AI_MODEL` lists separated by `;`, `,` or spaces with exact names, `AZURE_AI_PIPELINE_PREFIX`, model from an `*.openai.azure.com` URL, predefined and fallback models), `api` (api-key and Bearer header, path and api-version, allow-listed body: extra client keys dropped, tools forwarded, `stream_options` only for streams; JSON 400 (exactly Azure's message, nothing appended) and text/plain 500 errors), `dotted` (`gpt-4.1`, `Phi-3.5-mini-instruct` reach upstream intact, model in header or body), `browser` (full status sequence, error status with exactly Azure's message), `tasks` (title task, also with Azure AI Search valves, #123), `rag` (the pipe's own Azure AI Search retrieval, 3.0.0, #187; `suites/_azure_rag.py`: API and browser path, stream and non-stream, Foundry `/models` endpoint; search request body per `query_type`, query text rules, embeddings for `deployment_name` (v1 route, gateway prefix, Bearer) and `endpoint` (api-version, own key or access token, chat key only on the same scheme, host and port), integrated vectorizer; `fields_mapping` with `select`, separator and fallback, list-valued titles; `filter`, `top_n_documents` (2 x candidates), `strictness` (per query), the api-version valve, the document budget (auto, also with a large `top_n_documents`, valve, unlimited, dropping a document); prompt placement, one block and one system message, `in_scope` and the tool-results clause, `role_information`, sanitizing (also nested tags, titles and file names), list and image-only content, Open WebUI's tool-images message, an attached file (`` and RAG template around the prompt); citations, `context` event and `message.context`, scores per score type and with `AZURE_AI_INCLUDE_SEARCH_SCORES` off; `[docX]` → links, also split across stream deltas (API and browser, also without a finish chunk; in API streams all text before `[DONE]` and nothing after it), already linked or with parentheses in the URL (stream and non-stream), with a dotted deployment name; links in the history sent back as `[docX]` (also links saved before 2.8.0 with `)` in the URL; a hostile history line in linear time); only referenced sources saved and the show-all valve (also for an answer that cites only a document the search did not return, `[doc9]`, or one next to `[doc1]`); a context event over 128 KiB, an upstream event of ~300 KB (read and passed on) and one over 4 MiB (an error, API and browser); `content: null`; tool rounds (one search, reused only by a tool round of the same message with unchanged valves, no duplicate sources also when the tool round references a document, a non-stream tool call), tools and `stream_options` forwarded, tasks without retrieval and without `data_sources` (also for a saved chat without a websocket session); query generation (follow-ups, `always` / `off`, transcript with Open WebUI's conversation summary, max queries, `` with draft queries, fallback also for JSON nested too deep, partial and all-failed results, merge order by reciprocal rank fusion and reranker score, model in the body, the pause after 3 timeouts in a row: per model, reset by a success, still on after 16 s); auth: key, key valve, access token, system- and user-assigned managed identity (`mi_res_id`), token errors; fail-closed errors with a terminal status (HTTP 500/403/402/404/400/302/429/503, connection, retries, the 45 s retrieval limit, an answer larger than 16 MB, configuration errors, client `data_sources` (API stream and non-stream and the browser path as from an inlet filter: the error naming the removal of On Your Data in 3.0.0, the pipe's ERROR line and the terminal error status, nothing forwarded; an empty list is ignored), `context_length_exceeded` (exactly Azure's message and the budget hint)); Stop during a search and during query generation; `AZURE_AI_SEARCH_MODE=on_your_data` stored by a 2.9.0 pre-release or set in the environment is ignored; `log.debug`: the staged file run in the driver process with every logger at DEBUG logs no key or token, see below; `notice.none`: no On Your Data retirement notice of 2.8.1 in the server log), `logs` (no API key, no search key or token, no citation text, no generated queries, document titles or URLs in the server log at INFO and above) | | `n8n` | `pipelines/n8n/n8n.py` | `mock_n8n.py` | `api` (request payload contract, bearer / Cloudflare headers, usage, `intermediateSteps` tool display with verbosity and truncation, `` blocks, history / `INPUT_FIELD` / `RESPONSE_FIELD` valves, plain text, NDJSON, SSE streams in separate and coalesced writes with plain lines and `event:` / `id:` / `retry:` fields, OpenAI-style chunks, UTF-8 characters split across writes, braces inside strings, a large object trickling in as small writes (server CPU), webhook error), `browser` (saved answer, usage and final status for JSON, NDJSON, UTF-8, an n8n error chunk, a broken stream and a webhook error; chat context sent to the workflow for chat turns vs. background tasks; Stop during a stream and a non-stream request), `tasks` (title task without and with a chat id) | | `infomaniak` | `pipelines/infomaniak/infomaniak.py` | `mock_infomaniak.py` | `models` (llm models only, `NAME_PREFIX`), `api` (product id and bearer key, allow-listed body, SSE stream normal, coalesced into one write and split mid-JSON; OpenAI-style and Infomaniak `error.description` errors with one log line each), `browser` (saved answer, usage and status events for those streams plus no final newline, CRLF, a broken stream and an upstream error; Stop during the stream and while waiting for the response headers), `tasks` | @@ -206,20 +207,21 @@ query generation, errors, client `data_sources` with a key, also streamed); no l contain a key or token. Checks per suite on Open WebUI v0.11.4-slim in strict known mode (observed 2026-10-10, -`main` d3184b9: Azure pipeline 3.0.0, Time Token Tracker 2.7.0 with the Logs Ingestion -API, native tool calling in `google_gemini.py` 1.18.0 with the #194 fix; plus the -`NAME_PREFIX` fix in `infomaniak.py` 2.2.2): +`main` ecac3fb: Azure pipeline 3.0.0, Time Token Tracker 2.7.0 with the Logs Ingestion +API, native tool calling in `google_gemini.py` 1.18.0 with the #194 fix, the +`NAME_PREFIX` fix in `infomaniak.py` 2.2.2; plus the 10 `gemini.owuitools` checks): | Suite | Checks | PASS / KNOWN | | --- | ---: | ---: | -| `gemini` | 149 | 149 / 0 | +| `gemini` | 159 | 159 / 0 | | `azure` | 198 | 198 / 0 | | `n8n` | 51 | 51 / 0 | | `infomaniak` | 32 | 32 / 0 | | `filters` | 101 | 101 / 0 | -| **all** | **531** | **531 / 0** | +| **all** | **541** | **541 / 0** | -No FAIL and no obsolete marker; `v0.11.3-slim` gives the same counts for every suite. +No FAIL and no obsolete marker; `v0.11.3-slim` gives the same counts for every suite +(measured there for the gemini suite with the `owuitools` group too: 159 / 0). No known bug is registered (`harness/known_n8n.py` has no entry): the last one, `infomaniak-name-prefix` (the `NAME_PREFIX` valve was read only once), was fixed in `infomaniak.py` 2.2.2, and its marker went with the fix. The markers of the 62 bugs fixed by #182-#185 were dropped after the merge; their checks stay and @@ -338,6 +340,48 @@ their checks as FAIL (42 of the 48 with 1.17.0). Open WebUI 0.11.4's approval de is asked and the second is not run) are only recorded in the detail of `tools.approval-parallel`, never asserted. +### Open WebUI's image and code tools (`gemini.owuitools`) + +The group lives in `tests/e2e/suites/_gemini_owuitools.py` (run by `suites/gemini.py` +after `tools`). It checks that Open WebUI's own built-in tools `generate_image`, +`edit_image` (Open WebUI's image engine `gemini`) and `execute_code` (code interpreter) +work with a Gemini text model through Open WebUI's tool loop. It switches on **Admin +Settings → Images** with the engine `gemini` against the Gemini mock +(`IMAGES_GEMINI_API_BASE_URL=/v1beta`, keys `mock-images-key` and +`mock-images-edit-key`, by which the mock record tells the engine's requests from the +pipe's), image editing with `gemini-3.1-flash-image` and the code interpreter with the +engine pyodide, and restores both settings at its end. The `MOCKTOOLS:` directive makes +Gemini call the tool: + +| Check | What it shows | +| --- | --- | +| `owuitools.api` | API path with both feature flags on: none of the three tools is declared (Open WebUI adds built-in tools for browser sessions only); `pyodide_note=` records, without asserting it, that Open WebUI still adds its pyodide note to the system prompt | +| `owuitools.declared` | browser path: with `features.image_generation` and `features.code_interpreter` (the chat toggles) all three are declared and Open WebUI's pyodide note is in the system instruction; without them none | +| `owuitools.declared-noedit` | `ENABLE_IMAGE_EDIT=false`: `generate_image` without `edit_image` | +| `owuitools.generate` | `generate_image` with the endpoint method `generateContent` (`gemini-2.5-flash-image`): the engine request (path, key, prompt), the image saved with the answer (message `files`, `chat:message:files`) with the bytes the mock returned, the tool result with its URL in the continuation, the final answer | +| `owuitools.generate-predict` | the same with `predict` (`imagen-4.0-generate-001`, the mock's `:predict` route) | +| `owuitools.edit` | `edit_image` on an upload: the request to Gemini names the file's URL in ``, the edit engine gets the uploaded image, the edited image is saved | +| `owuitools.edit-followup` | `edit_image` in the next turn on the image of `owuitools.generate`: the replayed tool result holds its URL, the edit engine gets that image | +| `owuitools.image-model` | a pipe image model with the Image toggle on: no function declarations, its own image is saved, the engine is not called | +| `owuitools.code` | `execute_code`, engine pyodide: the browser session gets `execute:python` (the code, its session id), its stdout is the tool result; an image line in it is uploaded and linked | +| `owuitools.code-jupyter` | engine jupyter: the Jupyter mock starts a kernel, runs the code and deletes the kernel; no `execute:python`, no pyodide note; stdout and result reach Gemini | + +The Jupyter mock is part of `mocks/mock_tools.py` (`http://127.0.0.1:9111/jupyter/`: +`POST api/kernels`, the kernel websocket `api/kernels//channels`, which answers an +`execute_request` with stdout `JUPYTER-MOCK-STDOUT`, the result `'JUPYTER-MOCK-RESULT'` +and status idle, and `DELETE api/kernels/`; the executed code is recorded with the +path `jupyter:execute`). `BrowserSession` acks `execute:python` with +`python_answer(data)` (default `None`, an empty ack; the web UI acks with the result of +its pyodide worker, `{stdout, stderr, result}`) and records the calls in +`python_calls`. `mock_gemini.py` answers the image engine: the `generateContent` +request of an image model like any image request (without thoughts, the engine asks for +none), `:predict` with `predictions` (the `final-image-` trigger in the prompt picks +the PNG). Details start with `mock_ok=` as in the tool groups; the extra tokens are +`engine=[action:model:key]`, `file_ok=` and `sent_ok=`. Against `google_gemini.py` +1.17.0, which handed the tools to google-genai's automatic function calling, 6 of the +10 checks FAIL (`generate`, `generate-predict`, `edit`, `edit-followup`, `code`, +`code-jupyter`). + ### API path vs. browser path - **API path** (`OWUI.chat`): `POST /api/chat/completions` without `chat_id`, like an @@ -523,11 +567,12 @@ harness/ driver library: owui.py (REST client, API-path chat, valves suite.py, config.py suites/ one module per suite: GROUPS + async def run(t: Suite); _*.py are helper modules of a suite, not suites (_azure_rag.py: the azure rag - group; _gemini_tools.py: gemini.tools / toolsapi) + group; _gemini_tools.py: gemini.tools / toolsapi; + _gemini_owuitools.py: gemini.owuitools) mocks/ aiohttp provider mocks + serve_all.py (127.0.0.1:9101-9104 and :9106 in - the container), tool servers mock_tools.py (OpenAPI :9111, MCP :9112, no - E2E_MOCK_FAULT); mock_la.py (Log Analytics) is started by the filters - suite (:443, :9105, :9107) + the container), tool servers mock_tools.py (OpenAPI and Jupyter :9111, + MCP :9112, no E2E_MOCK_FAULT); mock_la.py (Log Analytics) is started + by the filters suite (:443, :9105, :9107) probe/probe_pipe.py test-only pipe reporting what Open WebUI passes to a pipe probe/workspace_tool.py test-only workspace Python tool (gemini.tools) ``` diff --git a/tests/e2e/harness/browser.py b/tests/e2e/harness/browser.py index 117c49f..276df82 100644 --- a/tests/e2e/harness/browser.py +++ b/tests/e2e/harness/browser.py @@ -18,7 +18,9 @@ ``tool_calls`` (built-in, workspace, MCP and OpenAPI tools on the server; direct tools in the browser, through an ``execute:tool`` socket call that the browser acknowledges with the result) and saves the turn as ``output`` items -(``function_call``, ``function_call_output``, ``reasoning``, ``message``). +(``function_call``, ``function_call_output``, ``reasoning``, ``message``). The +built-in ``execute_code`` tool with the pyodide engine runs the code in the +browser the same way (``execute:python``, see ``python_answer``). """ import asyncio @@ -192,6 +194,13 @@ def __init__(self, owui: OWUI): # is sent as JSON; None makes Open WebUI report unparsable arguments). self.execute_calls: list = [] self.direct_tool_answer = lambda data: NO_DIRECT_ANSWER + # execute:python calls (Open WebUI's execute_code tool with the pyodide + # engine runs the code in the browser) and their answer: a callable(call + # data) -> ack. The web UI acks with its pyodide result, e.g. + # ``{"stdout": "4\n", "stderr": None, "result": None}``; the default None + # is an empty ack (no pyodide here). + self.python_calls: list = [] + self.python_answer = lambda data: None self._heartbeat: Optional[asyncio.Task] = None async def __aenter__(self) -> "BrowserSession": @@ -217,6 +226,9 @@ async def _on_events(data): # Open WebUI waits for this ack (sio.call): the tool result self.execute_calls.append(data) return self.direct_tool_answer(inner.get("data") or {}) + if isinstance(inner, dict) and inner.get("type") == "execute:python": + self.python_calls.append(data) + return self.python_answer(inner.get("data") or {}) return None await self.sio.connect( diff --git a/tests/e2e/mocks/mock_gemini.py b/tests/e2e/mocks/mock_gemini.py index 78f3e09..b28060b 100644 --- a/tests/e2e/mocks/mock_gemini.py +++ b/tests/e2e/mocks/mock_gemini.py @@ -7,6 +7,8 @@ GET /{version}/models model list POST /{version}/models/{model}:generateContent JSON answer POST /{version}/models/{model}:streamGenerateContent SSE answer (?alt=sse) + POST /{version}/models/{model}:predict Imagen (Open WebUI's image engine + "gemini", endpoint method predict) POST /{version}/models/{model}:predictLongRunning Veo: start an operation GET /{version}/models/{model}/operations/{op} Veo: poll the operation GET /{version}/files/{id}:download Veo: download the video @@ -33,6 +35,12 @@ (plus a thought part when thoughts are requested) - Veo: the operation starts pending; the first poll reports ``done: true`` with one video whose Files API ``uri`` google-genai downloads from this mock + - Open WebUI's own image engine "gemini" (built-in tools generate_image / + edit_image): its generateContent requests to an image model get the image + answer above (no thoughts: it asks for none); ``:predict`` (Imagen) answers + ``predictions`` with ``parameters.sampleCount`` (default 1) PNGs, FINAL_PNG + or the ``final-image-`` PNG of the prompt (``instances`` as an object, as + Open WebUI sends it, or as a list) Tools an image model does not support are rejected like the real API does (HTTP 400 INVALID_ARGUMENT; an approximation of the real messages): @@ -99,7 +107,7 @@ "skip_thought_signature_validator"). Each recorded request carries ``model``, ``action`` (generateContent, -streamGenerateContent, predictLongRunning, download), ``tool_kinds`` (keys of +streamGenerateContent, predict, predictLongRunning, download), ``tool_kinds`` (keys of the request's ``tools`` entries, e.g. ``googleSearch``) and, for generate requests, ``status`` (the HTTP status the mock answered with) plus what the tool scenarios assert on (``_tool_view``): ``declared``, ``decl`` (raw schema per @@ -946,6 +954,8 @@ async def model_action(request: web.Request) -> web.StreamResponse: await asyncio.sleep(0.05) await resp.write_eof() return resp + if action == "predict": + return web.json_response(_predict(body)) if action == "predictLongRunning": op = f"op-{next(_op_ids)}" prompt = str(((body.get("instances") or [{}])[0] or {}).get("prompt") or "") @@ -955,6 +965,17 @@ async def model_action(request: web.Request) -> web.StreamResponse: return _not_found(request) +def _predict(body: dict) -> dict: + """Imagen ``:predict`` answer: ``sampleCount`` PNGs for the prompt.""" + instances = body.get("instances") + if isinstance(instances, list): + instances = instances[0] if instances else {} + prompt = str((instances if isinstance(instances, dict) else {}).get("prompt") or "") + count = (body.get("parameters") or {}).get("sampleCount") or 1 + image = {"bytesBase64Encoded": _final_png(prompt), "mimeType": "image/png"} + return {"predictions": [dict(image) for _ in range(max(1, min(int(count), 4)))]} + + async def get_operation(request: web.Request) -> web.Response: await record(request) model, op = request.match_info["model"], request.match_info["op"] diff --git a/tests/e2e/mocks/mock_tools.py b/tests/e2e/mocks/mock_tools.py index 330eebc..a75809e 100644 --- a/tests/e2e/mocks/mock_tools.py +++ b/tests/e2e/mocks/mock_tools.py @@ -1,5 +1,6 @@ """ -Tool server mocks for the native tool calling scenarios (``gemini.tools``). +Tool server mocks for the native tool calling scenarios (``gemini.tools``, +``gemini.owuitools``). These are tools Open WebUI runs itself, not provider APIs: they are NOT behind ``E2E_MOCK_FAULT`` (a provider fault must not break the tools). @@ -10,8 +11,20 @@ GET /lookup?key= operationId lookup.v2 (a name Gemini does not accept: the pipe maps it) - GET /__requests, POST /__reset record of the OpenAPI calls and of - the MCP tool calls (path mcp:) + Jupyter 127.0.0.1:9111 /jupyter/ the subset of a Jupyter server that + Open WebUI's code interpreter engine + "jupyter" uses: POST api/kernels, + the kernel websocket + api/kernels//channels (an + execute_request is answered with + stdout JUPYTER_STDOUT, the result + JUPYTER_RESULT and status idle; the + code is recorded, path + jupyter:execute), DELETE + api/kernels/ + GET /__requests, POST /__reset record of the OpenAPI calls, the + Jupyter requests and the MCP tool + calls (path mcp:) MCP 127.0.0.1:9112 /mcp FastMCP streamable HTTP (the image's ``mcp`` package): mcp_echo, mcp_sum. Optional: when it cannot start, @@ -23,12 +36,14 @@ import argparse import asyncio +import itertools +import json import logging import threading import time import warnings -from aiohttp import web +from aiohttp import WSMsgType, web from common import REQUESTS_KEY, new_app, record @@ -39,8 +54,13 @@ # as the tool name, the pipe declares it as _gemini_function_name("lookup.v2"). LOOKUP_OPERATION = "lookup.v2" +# What the Jupyter mock answers an execute_request with (stdout, text/plain result) +JUPYTER_STDOUT = "JUPYTER-MOCK-STDOUT" +JUPYTER_RESULT = "'JUPYTER-MOCK-RESULT'" + # record of the running process (shared by the OpenAPI app and the MCP thread) _CALLS: list = [] +_kernel_ids = itertools.count(1) def spec(port: int = PORT) -> dict: @@ -135,6 +155,66 @@ async def lookup(request: web.Request) -> web.Response: return web.json_response({"key": request.query.get("key"), "found": True}) +# ------------------------------------------------------------------- Jupyter +async def jupyter_start_kernel(request: web.Request) -> web.Response: + await record(request) + return web.json_response({"id": f"e2e-kernel-{next(_kernel_ids)}", "name": "py"}) + + +async def jupyter_delete_kernel(request: web.Request) -> web.Response: + await record(request) + return web.Response(status=204) + + +def _jupyter_message(parent: dict, msg_type: str, content: dict) -> str: + return json.dumps( + { + "header": {"msg_id": f"mock-{time.time_ns()}", "msg_type": msg_type}, + "parent_header": parent, + "msg_type": msg_type, + "metadata": {}, + "content": content, + "channel": "iopub", + } + ) + + +async def jupyter_channels(request: web.Request) -> web.WebSocketResponse: + """Kernel websocket: answer every execute_request like a kernel would.""" + ws = web.WebSocketResponse() + await ws.prepare(request) + async for message in ws: + if message.type != WSMsgType.TEXT: + continue + try: + data = json.loads(message.data) + except ValueError: + continue + header = data.get("header") or {} + if header.get("msg_type") != "execute_request": + continue + _CALLS.append( + { + "t": time.time(), + "method": "WS", + "path": "jupyter:execute", + "kernel": request.match_info["kernel"], + "body": {"code": (data.get("content") or {}).get("code")}, + } + ) + for msg_type, content in ( + ("status", {"execution_state": "busy"}), + ("stream", {"name": "stdout", "text": JUPYTER_STDOUT + "\n"}), + ( + "execute_result", + {"data": {"text/plain": JUPYTER_RESULT}, "execution_count": 1}, + ), + ("status", {"execution_state": "idle"}), + ): + await ws.send_str(_jupyter_message(header, msg_type, content)) + return ws + + def make_app(port: int = PORT) -> web.Application: app = new_app(fault_injection=False) app[REQUESTS_KEY] = _CALLS # the MCP tools record into the same list @@ -143,6 +223,9 @@ def make_app(port: int = PORT) -> web.Application: app.router.add_get("/weather", weather) app.router.add_post("/convert", convert) app.router.add_get("/lookup", lookup) + app.router.add_post("/jupyter/api/kernels", jupyter_start_kernel) + app.router.add_get("/jupyter/api/kernels/{kernel}/channels", jupyter_channels) + app.router.add_delete("/jupyter/api/kernels/{kernel}", jupyter_delete_kernel) return app diff --git a/tests/e2e/suites/_gemini_owuitools.py b/tests/e2e/suites/_gemini_owuitools.py new file mode 100644 index 0000000..2963cb2 --- /dev/null +++ b/tests/e2e/suites/_gemini_owuitools.py @@ -0,0 +1,662 @@ +""" +Group ``owuitools`` of the gemini suite: Open WebUI's own built-in tools +``generate_image`` / ``edit_image`` (Open WebUI's image engine "gemini") and +``execute_code`` (code interpreter), called by a Gemini *text* model through +Open WebUI's tool loop (native function calling, google_gemini.py 1.18.0). Run +by suites/gemini.py; the leading underscore keeps this module out of the suite +discovery. + +Setup (restored at the end): Admin Settings -> Images with engine "gemini" +pointing at the Gemini mock (``IMAGES_GEMINI_API_BASE_URL`` = /v1beta, an +API key of its own for generation and for editing), image editing switched on; +the code interpreter is on by default (engine pyodide). The pipe models carry +no capabilities, so every capability counts as on (Open WebUI's default). + +Open WebUI 0.11 puts the tools into ``body["tools"]`` only for a browser +session (``session_id``), with function calling not "legacy" and the model +capability ``builtin_tools``, and per tool when the model's builtinTools +category, the global switch (``image_generation.enable``; +``images.edit.enable`` for edit_image; ``code_interpreter.enable``), the model +capability, the request's ``features`` flag (the chat toggle) and the user +permission ``features.`` (admins pass) are all on +(utils/tools.py get_builtin_tools, utils/middleware.py). + +The Gemini mock answers the ``MOCKTOOLS:`` directive with the tool call; Open +WebUI runs the tool: the image engine requests reach the same mock (``:predict`` +for Imagen, ``:generateContent`` for an image model, recorded with their +``x-goog-api-key``), ``execute_code`` asks the browser session +(``execute:python``, answered by ``BrowserSession.python_answer`` the way the +web UI's pyodide worker acks) or the Jupyter mock of mocks/mock_tools.py. + +Every detail starts with ``mock_ok=`` (the preflight of +suites/_gemini_tools.py); the tokens follow ``_bdetail`` there, plus +``engine=[action:model:key]`` (the image engine requests), ``file_ok=`` (the +saved image has the bytes the engine answered) and ``sent_ok=`` (the image the +edit engine got is the one the call named). +""" + +import base64 + +from harness import Suite, short +from harness.config import MOCK_HOST, MOCK_PORTS + +from ._gemini_tools import ( + CALL_0, + PRO3_ID, + Ctx, + _bdetail, + _calls, + _declared, + _final, + _frs, + _json, + _req_tokens, + directive, + preflight, +) +from .gemini import ( + IMAGE_GA, + PRO3, + TEXT, + _body, + _gen, + _png, + _status_closed, + _strings, + _upload_png, +) + +IMG_KEY = "mock-images-key" # Open WebUI's image generation key (not the pipe's) +EDIT_KEY = "mock-images-edit-key" +IMG_MODEL = "gemini-2.5-flash-image" # endpoint method generateContent +IMAGEN = "imagen-4.0-generate-001" # endpoint method predict +EDIT_MODEL = "gemini-3.1-flash-image" # also a pipe model: engine requests are +# told apart by their key +ENGINE_KEYS = (IMG_KEY, EDIT_KEY) +CODE_TOOL = "execute_code" +OWUI_TOOLS = ("generate_image", "edit_image", CODE_TOOL) +IMG_ON = {"image_generation": True} +CODE_ON = {"code_interpreter": True} +# Open WebUI's system prompt addition for the pyodide engine (native mode) +PYODIDE_PROMPT = "Pyodide Environment" +JUPYTER_URL = f"http://{MOCK_HOST}:{MOCK_PORTS['tools']}/jupyter" +# mocks/mock_tools.py's answer to an execute_request +JUPYTER_STDOUT = "JUPYTER-MOCK-STDOUT" +JUPYTER_RESULT = "'JUPYTER-MOCK-RESULT'" +CODE = "print(2 + 2)" +PLOT_PNG = _png(1, 2, 3) # a matplotlib-style image line in the pyodide stdout +UPLOAD_PNG = _png(10, 20, 30) + + +def _engine_key(entry: dict) -> str: + return (entry.get("headers") or {}).get("x-goog-api-key") or "" + + +def _engine(entry: dict) -> bool: + """A request of Open WebUI's image engine (generation or editing): its + own API keys, not the pipe's.""" + return _engine_key(entry) in ENGINE_KEYS + + +def _pipe(model_id: str = PRO3_ID): + """Match the pipe's generate requests to ``model_id``.""" + return lambda e: _gen(e) and e.get("model") == model_id and not _engine(e) + + +def _engine_tokens(entries: list) -> str: + items = [f"{e.get('action')}:{e.get('model')}:{_engine_key(e)}" for e in entries] + return "[" + ",".join(items) + "]" + + +def _engine_text(entry: dict) -> str: + """The prompt the image engine sent (generateContent or predict).""" + body = entry.get("body") if isinstance(entry.get("body"), dict) else {} + if entry.get("action") == "predict": + instances = body.get("instances") + if isinstance(instances, list): + instances = instances[0] if instances else {} + return str((instances if isinstance(instances, dict) else {}).get("prompt")) + return " ".join( + p.get("text", "") + for c in body.get("contents") or [] + for p in c.get("parts") or [] + if isinstance(p, dict) and "text" in p + ) + + +def _engine_images(entry: dict) -> list: + """Base64 of the images the edit engine sent (inline_data parts).""" + body = entry.get("body") if isinstance(entry.get("body"), dict) else {} + images = [] + for content in body.get("contents") or []: + for part in content.get("parts") or []: + data = (part.get("inline_data") or part.get("inlineData") or {}).get("data") + if data: + images.append(data) + return images + + +def _engine_ok(engine: list, path: str, key: str, prompt: str) -> bool: + return ( + len(engine) == 1 + and engine[0].get("path") == path + and _engine_key(engine[0]) == key + and _engine_text(engine[0]) == prompt + ) + + +def _owui_tools(names: list) -> list: + return [n for n in OWUI_TOOLS if n in names] + + +def _system(entry: dict) -> str: + return " ".join(_strings(_body(entry).get("systemInstruction"))) + + +async def _config(owui, path: str) -> dict: + status, data = await owui.api("GET", path) + if status != 200 or not isinstance(data, dict): + raise RuntimeError(f"GET {path}: HTTP {status} {short(data)}") + return data + + +async def _post(owui, path: str, body: dict) -> None: + status, data = await owui.api("POST", path, body) + if status != 200: + raise RuntimeError(f"POST {path}: HTTP {status} {short(data)}") + + +async def _set_images(owui, **changes) -> dict: + """Change Admin Settings -> Images (the full form is sent); returns the + previous settings.""" + previous = await _config(owui, "/api/v1/images/config") + await _post(owui, "/api/v1/images/config/update", {**previous, **changes}) + return previous + + +async def _set_code(owui, **changes) -> dict: + """Change Admin Settings -> Code Execution; returns the previous settings.""" + previous = await _config(owui, "/api/v1/configs/code_execution") + await _post(owui, "/api/v1/configs/code_execution", {**previous, **changes}) + return previous + + +async def _file_b64(owui, url: str) -> str: + """Base64 of an Open WebUI file URL ('' if there is none).""" + if not url: + return "" + r = await owui.request("GET", url) + return base64.b64encode(r.content).decode() if r.status_code == 200 else "" + + +def _first_url(files: list) -> str: + return (files[0] if files else {}).get("url") or "" + + +def _tool_call(name: str, **args) -> str: + return directive([[{"name": name, "args": args}]]) + + +# ================================================================= owuitools +async def owuitools(t: Suite, mock) -> None: + ctx = await preflight(t, mock) + images_before = code_before = None + try: + images_before = await _set_images( + t.owui, + ENABLE_IMAGE_GENERATION=True, + IMAGE_GENERATION_ENGINE="gemini", + IMAGE_GENERATION_MODEL=IMG_MODEL, + IMAGES_GEMINI_API_BASE_URL=mock.url + "/v1beta", + IMAGES_GEMINI_API_KEY=IMG_KEY, + IMAGES_GEMINI_ENDPOINT_METHOD="generateContent", + ENABLE_IMAGE_EDIT=True, + IMAGE_EDIT_ENGINE="gemini", + IMAGE_EDIT_MODEL=EDIT_MODEL, + IMAGES_EDIT_GEMINI_API_BASE_URL=mock.url + "/v1beta", + IMAGES_EDIT_GEMINI_API_KEY=EDIT_KEY, + ) + code_before = await _set_code( + t.owui, ENABLE_CODE_INTERPRETER=True, CODE_INTERPRETER_ENGINE="pyodide" + ) + await api_path(ctx) # before any websocket session of this user + async with t.browser() as b: + await declared(ctx, b) + generated = await generate(ctx, b) + await generate_predict(ctx, b) + await edit(ctx, b) + await edit_followup(ctx, b, generated) + await image_model(ctx, b) + await code(ctx, b) + await code_jupyter(ctx, b) + finally: + if images_before is not None: + await _post(t.owui, "/api/v1/images/config/update", images_before) + if code_before is not None: + await _post(t.owui, "/api/v1/configs/code_execution", code_before) + + +async def api_path(ctx: Ctx) -> None: + """API clients (no session_id) get no hidden built-in tools, toggles or not.""" + await ctx.mock.reset() + r = await ctx.t.owui.chat( + TEXT, + "Hello API with image and code features", + stream=False, + features={**IMG_ON, **CODE_ON}, + ) + reqs = await ctx.mock.requests(_pipe(TEXT.split(".", 1)[1])) + names = _declared(reqs[0]) if reqs else [] + ctx.check( + "owuitools.api", + "API path with features image_generation + code_interpreter: no " + "generate_image / edit_image / execute_code declared (Open WebUI adds its " + "built-in tools for browser sessions only)", + r.status == 200 + and "Hello from mock" in r.content + and len(reqs) == 1 + and not _owui_tools(names), + f"http={r.status} upstream={[e.get('answer') for e in reqs]} " + f"declared_n={len(names)} owui_tools={_owui_tools(names)} " + f"pyodide_note={PYODIDE_PROMPT in _system(reqs[0] if reqs else {})} " + f"content={short(r.content, 120)}", + ) + + +async def _plain_turn(ctx: Ctx, b, label: str, features: dict) -> tuple: + """A turn without tool call; (chat, pipe requests, declared names, pyodide + note in the system instruction).""" + await ctx.mock.reset() + c = await ctx.chat(b, PRO3, f"Hello declared {label}", features=features) + reqs = await ctx.requests() + first = reqs[0] if reqs else {} + return c, reqs, _declared(first), PYODIDE_PROMPT in _system(first) + + +async def declared(ctx: Ctx, b) -> None: + off, off_reqs, off_names, off_note = await _plain_turn(ctx, b, "off", {}) + on, reqs, names, note = await _plain_turn(ctx, b, "on", {**IMG_ON, **CODE_ON}) + ctx.check( + "owuitools.declared", + "browser path: with the Image and Code Interpreter toggles (features) on, " + "generate_image, edit_image and execute_code are declared to Gemini and " + "Open WebUI's pyodide note is in the system instruction; with the toggles " + "off none of them", + on.done + and off.done + and "Hello from mock" in _final(on) + and len(reqs) == 1 + and _owui_tools(names) == list(OWUI_TOOLS) + and note + and len(off_reqs) == 1 + and not _owui_tools(off_names) + and not off_note, + _bdetail( + on, + reqs, + f"owui_tools={_owui_tools(names)} pyodide_note={note} " + f"off_owui_tools={_owui_tools(off_names)} off_pyodide_note={off_note} " + f"off_done={off.done}", + ), + ) + await _set_images(ctx.t.owui, ENABLE_IMAGE_EDIT=False) + try: + c, reqs, names, _ = await _plain_turn(ctx, b, "noedit", IMG_ON) + finally: + await _set_images(ctx.t.owui, ENABLE_IMAGE_EDIT=True) + ctx.check( + "owuitools.declared-noedit", + "image editing switched off (ENABLE_IMAGE_EDIT=false): generate_image is " + "declared, edit_image is not", + c.done + and "Hello from mock" in _final(c) + and _owui_tools(names) == ["generate_image"], + _bdetail(c, reqs, f"owui_tools={_owui_tools(names)}"), + ) + + +async def _image_turn(ctx: Ctx, b, text: str, **kwargs) -> tuple: + """One image tool turn of the Gemini 3 text model; (chat, pipe requests, + image engine requests, saved image as base64).""" + await ctx.mock.reset() + c = await ctx.chat(b, PRO3, text, features=IMG_ON, **kwargs) + entries = await ctx.mock.requests() + reqs = [e for e in entries if _pipe()(e)] + ctx.answered += sum(1 for e in reqs if e.get("status") == 200) + engine = [e for e in entries if _engine(e)] + return c, reqs, engine, await _file_b64(ctx.t.owui, _first_url(c.files)) + + +def _image_saved(c, name: str) -> bool: + """The call ran, its result names the one image saved with the answer + (message files, chat:message:files event), the final answer is saved.""" + result = _json(c.function_outputs.get(CALL_0)) + urls = [i.get("url") for i in result.get("images") or [] if isinstance(i, dict)] + files = [f.get("url") for f in c.files if f.get("type") == "image"] + return ( + c.done + and not c.error + and _calls(c) == [(name, "completed", CALL_0)] + and result.get("status") == "success" + and len(files) == 1 + and urls == files + and "chat:message:files" in c.event_types + and _final(c).startswith(f"MOCK-FINAL {name}=") + and _status_closed(c) + ) + + +def _continued(reqs: list, name: str, url: str, turn: str = "user:text") -> bool: + """The continuation sends the signed call and Open WebUI's tool result (with + the image URL) back to Gemini.""" + second = reqs[1] if len(reqs) > 1 else {} + responses = str([x.get("response") for x in second.get("fr") or []]) + return ( + [e.get("answer") for e in reqs] == ["fc", "final"] + and second.get("kinds") == [turn, "model:fc", "user:fr"] + and [x.get("sig") for x in second.get("fc") or []] == ["issued"] + and _frs(second) == [(CALL_0, name, ["output"])] + and bool(url) + and url in responses + ) + + +def _image_detail(c, reqs: list, engine: list, file_ok: bool, extra: str = "") -> str: + second = reqs[1] if len(reqs) > 1 else {} + return _bdetail( + c, + reqs, + f"engine={_engine_tokens(engine)} " + f"engine_prompt={short(_engine_text(engine[0]), 60) if engine else None} " + f"file_ok={file_ok} files={len(c.files)} " + f"files_event={'chat:message:files' in c.event_types} " + f"statuses_closed={_status_closed(c)} {extra} {_req_tokens(second)}", + ) + + +async def generate(ctx: Ctx, b): + """generate_image with the engine's generateContent method (image model).""" + prompt = "a yellow square final-image-7" + c, reqs, engine, saved = await _image_turn( + ctx, b, _tool_call("generate_image", prompt=prompt) + ) + path = f"/v1beta/models/{IMG_MODEL}:generateContent" + file_ok = saved == _png(255, 255, 7) + ctx.check( + "owuitools.generate", + "generate_image (engine gemini, generateContent): Gemini's call runs Open " + "WebUI's image engine with its own key and the call's prompt, the image is " + "saved with the answer (message files, chat:message:files), the tool result " + "with its URL goes back to Gemini, the final answer is saved", + _image_saved(c, "generate_image") + and _engine_ok(engine, path, IMG_KEY, prompt) + and file_ok + and _continued(reqs, "generate_image", _first_url(c.files)), + _image_detail(c, reqs, engine, file_ok), + ) + return c + + +async def generate_predict(ctx: Ctx, b) -> None: + """generate_image with the engine's predict method (Imagen).""" + await _set_images( + ctx.t.owui, + IMAGE_GENERATION_MODEL=IMAGEN, + IMAGES_GEMINI_ENDPOINT_METHOD="predict", + ) + prompt = "a yellow square final-image-9" + try: + c, reqs, engine, saved = await _image_turn( + ctx, b, _tool_call("generate_image", prompt=prompt) + ) + finally: + await _set_images( + ctx.t.owui, + IMAGE_GENERATION_MODEL=IMG_MODEL, + IMAGES_GEMINI_ENDPOINT_METHOD="generateContent", + ) + path = f"/v1beta/models/{IMAGEN}:predict" + file_ok = saved == _png(255, 255, 9) + ctx.check( + "owuitools.generate-predict", + "generate_image (engine gemini, predict = Imagen): the engine's :predict " + "request carries the call's prompt, the image is saved with the answer, the " + "tool result goes back to Gemini", + _image_saved(c, "generate_image") + and _engine_ok(engine, path, IMG_KEY, prompt) + and file_ok + and _continued(reqs, "generate_image", _first_url(c.files)), + _image_detail(c, reqs, engine, file_ok), + ) + + +async def edit(ctx: Ctx, b) -> None: + """edit_image on an uploaded image: Open WebUI tells Gemini the file's URL + (), Gemini passes it in image_urls, the edit engine gets the + image.""" + item = await _upload_png(ctx.t.owui, UPLOAD_PNG, "photo.png") + file_id = item.get("url") or "" + prompt = "make it blue final-image-11" + c, reqs, engine, saved = await _image_turn( + ctx, + b, + _tool_call("edit_image", prompt=prompt, image_urls=[file_id]), + user_files=[item], + ) + first = reqs[0] if reqs else {} + attached = bool(file_id) and f'url="{file_id}"' in " ".join( + _strings(_body(first).get("contents")) + ) + path = f"/v1beta/models/{EDIT_MODEL}:generateContent" + sent_ok = bool(engine) and _engine_images(engine[0]) == [UPLOAD_PNG] + file_ok = saved == _png(255, 255, 11) + ctx.check( + "owuitools.edit", + "edit_image on an uploaded image: the request to Gemini names the file in " + " (and carries the image), Gemini's call with that URL runs " + "the edit engine (gemini, own key) with the uploaded image, the edited " + "image is saved with the answer, the tool result goes back to Gemini", + _image_saved(c, "edit_image") + and attached + and first.get("kinds") == ["user:text+text+inline"] + and _engine_ok(engine, path, EDIT_KEY, prompt) + and sent_ok + and file_ok + and _continued( + reqs, "edit_image", _first_url(c.files), "user:text+text+inline" + ), + _image_detail( + c, + reqs, + engine, + file_ok, + f"attached={attached} first_kinds={first.get('kinds')} sent_ok={sent_ok}", + ), + ) + + +async def edit_followup(ctx: Ctx, b, generated) -> None: + """The next turn edits the image generate_image made: the replayed tool + result holds its URL, the edit engine gets that image.""" + gen_url = _first_url(generated.files) + if not generated.chat_id or not gen_url: + ctx.check( + "owuitools.edit-followup", + "edit_image in the next turn on the image generate_image made", + False, + f"no generated image to edit: generate done={generated.done} " + f"files={short(generated.files, 120)}", + ) + return + prompt = "add a hat final-image-13" + c, reqs, engine, saved = await _image_turn( + ctx, + b, + _tool_call("edit_image", prompt=prompt, image_urls=[gen_url]), + chat_id=generated.chat_id, + parent_id=generated.message_id, + ) + first = reqs[0] if reqs else {} + replayed = gen_url in str([x.get("response") for x in first.get("fr") or []]) + path = f"/v1beta/models/{EDIT_MODEL}:generateContent" + sent_ok = bool(engine) and _engine_images(engine[0]) == [_png(255, 255, 7)] + file_ok = saved == _png(255, 255, 13) + ctx.check( + "owuitools.edit-followup", + "edit_image in the next turn on the image generate_image made: the " + "replayed turn-1 tool result holds its URL, the edit engine gets that " + "image, the edited image is saved with the new answer", + _image_saved(c, "edit_image") + and first.get("kinds") + == ["user:text", "model:fc", "user:fr", "model:text", "user:text"] + and replayed + and _engine_ok(engine, path, EDIT_KEY, prompt) + and sent_ok + and file_ok, + _image_detail( + c, + reqs, + engine, + file_ok, + f"first_kinds={first.get('kinds')} url_replayed={replayed} " + f"sent_ok={sent_ok}", + ), + ) + + +async def image_model(ctx: Ctx, b) -> None: + """A pipe image model with the Image toggle on: the pipe drops Open WebUI's + tools (image models have no function calling), the model makes the image + itself and Open WebUI's engine is not called.""" + await ctx.mock.reset() + c = await ctx.chat(b, IMAGE_GA, "Draw a cat final-image-15", features=IMG_ON) + entries = await ctx.mock.requests() + reqs = [e for e in entries if _pipe(IMAGE_GA.split(".", 1)[1])(e)] + engine = [e for e in entries if _engine(e)] + kinds = [k for e in reqs for k in e.get("tool_kinds") or []] + file_ok = await _file_b64(ctx.t.owui, _first_url(c.files)) == _png(255, 255, 15) + ctx.check( + "owuitools.image-model", + "pipe image model (gemini-3.1-flash-image) with the Image toggle on: no " + "function declarations, the model's own image is saved, Open WebUI's image " + "engine is not called", + c.done + and not c.error + and len(reqs) == 1 + and reqs[0].get("status") == 200 + and "functionDeclarations" not in kinds + and not engine + and len(c.files) == 1 + and file_ok, + f"http={c.http_status} done={c.done} " + f"error={short(c.error, 120) if c.error else None} " + f"upstream={[e.get('answer') for e in reqs]} tool_kinds={kinds} " + f"engine={_engine_tokens(engine)} file_ok={file_ok} files={len(c.files)}", + ) + + +async def code(ctx: Ctx, b) -> None: + """execute_code, engine pyodide: Open WebUI asks the browser to run the code + (execute:python), the browser's result is the tool result; an image line in + stdout is uploaded and replaced with a markdown image link.""" + stdout = f"4\ndata:image/png;base64,{PLOT_PNG}" + b.python_answer = lambda data: {"stdout": stdout, "stderr": None, "result": None} + before = len(b.python_calls) + try: + await ctx.mock.reset() + c = await ctx.chat(b, PRO3, _tool_call(CODE_TOOL, code=CODE), features=CODE_ON) + reqs = await ctx.requests() + finally: + b.python_answer = lambda data: None + calls = [(x.get("data") or {}).get("data") or {} for x in b.python_calls[before:]] + session_ok = len(calls) == 1 and calls[0].get("session_id") == b.sid + out = str(_json(c.function_outputs.get(CALL_0)).get("stdout") or "") + marker = "![Output Image](" + link = out.split(marker, 1)[1].split(")", 1)[0] if marker in out else "" + plot_ok = bool(link) and await _file_b64(ctx.t.owui, link) == PLOT_PNG + second = reqs[1] if len(reqs) > 1 else {} + responses = str([x.get("response") for x in second.get("fr") or []]) + ctx.check( + "owuitools.code", + "execute_code (engine pyodide): Open WebUI sends the code to the browser " + "session (execute:python), the browser's stdout is the tool result Gemini " + "gets back (an image line uploaded and linked), the final answer is saved", + c.done + and not c.error + and _calls(c) == [(CODE_TOOL, "completed", CALL_0)] + and session_ok + and calls[0].get("code") == CODE + and out.startswith("4\n" + marker + "/api/v1/files/") + and plot_ok + and [e.get("answer") for e in reqs] == ["fc", "final"] + and _frs(second) == [(CALL_0, CODE_TOOL, ["output"])] + and link in responses + and _final(c).startswith(f"MOCK-FINAL {CODE_TOOL}="), + _bdetail( + c, + reqs, + f"python_calls={len(calls)} session_ok={session_ok} " + f"code={short(calls[0].get('code'), 40) if calls else None} " + f"stdout={short(out, 90)} plot_ok={plot_ok} {_req_tokens(second)}", + ), + ) + + +async def code_jupyter(ctx: Ctx, b) -> None: + """execute_code, engine jupyter: the code runs on the Jupyter server + (mocks/mock_tools.py), not in the browser; no pyodide note.""" + tool_mock = ctx.t.mock("tools") + await _set_code( + ctx.t.owui, + CODE_INTERPRETER_ENGINE="jupyter", + CODE_INTERPRETER_JUPYTER_URL=JUPYTER_URL, + CODE_INTERPRETER_JUPYTER_AUTH="", + CODE_INTERPRETER_JUPYTER_TIMEOUT=30, + ) + before = len(b.python_calls) + try: + await ctx.mock.reset() + await tool_mock.reset() + c = await ctx.chat(b, PRO3, _tool_call(CODE_TOOL, code=CODE), features=CODE_ON) + reqs = await ctx.requests() + jupyter = await tool_mock.requests( + lambda e: str(e.get("path", "")).startswith(("/jupyter/", "jupyter:")) + ) + finally: + await _set_code(ctx.t.owui, CODE_INTERPRETER_ENGINE="pyodide") + first = reqs[0] if reqs else {} + note = PYODIDE_PROMPT in _system(first) + executed = [ + (e.get("body") or {}).get("code") + for e in jupyter + if e.get("path") == "jupyter:execute" + ] + paths = [f"{e.get('method')}:{e.get('path')}" for e in jupyter] + result = _json(c.function_outputs.get(CALL_0)) + second = reqs[1] if len(reqs) > 1 else {} + responses = str([x.get("response") for x in second.get("fr") or []]) + ctx.check( + "owuitools.code-jupyter", + "execute_code (engine jupyter): the code runs on the Jupyter server " + "(kernel started, executed, deleted), not in the browser; its stdout and " + "result are the tool result Gemini gets back; no pyodide note", + c.done + and not c.error + and CODE_TOOL in _declared(first) + and not note + and _calls(c) == [(CODE_TOOL, "completed", CALL_0)] + and len(b.python_calls) == before + and executed == [CODE] + and any(p.startswith("DELETE:") for p in paths) + and result.get("stdout") == JUPYTER_STDOUT + and result.get("result") == JUPYTER_RESULT + and JUPYTER_STDOUT in responses + and [e.get("answer") for e in reqs] == ["fc", "final"], + _bdetail( + c, + reqs, + f"jupyter={paths} executed={executed} " + f"browser_calls={len(b.python_calls) - before} pyodide_note={note} " + f"result={short(result, 160)}", + ), + ) diff --git a/tests/e2e/suites/gemini.py b/tests/e2e/suites/gemini.py index fc19187..8168936 100644 --- a/tests/e2e/suites/gemini.py +++ b/tests/e2e/suites/gemini.py @@ -47,6 +47,9 @@ OpenAPI, MCP and direct tools, approval, follow-up turns, unknown tool, malformed call, grounding with tools (and the next turn), task / legacy / no built-in tools, streaming valve off + owuitools Open WebUI's built-in image generation / editing (image engine + "gemini" on the mock) and code interpreter tools called by a + Gemini text model The browser path sends the web UI's default params, i.e. native function calling with Open WebUI's built-in tools (functionDeclarations). The "images" @@ -93,6 +96,7 @@ "imgedit", "toolsapi", "tools", + "owuitools", ) FID = "gemini" PATH = "pipelines/google/google_gemini.py" @@ -464,6 +468,7 @@ async def run(t: Suite) -> None: ("imgedit", imgedit), ("toolsapi", toolsapi_group), ("tools", tools_group), + ("owuitools", owuitools_group), ): if t.selected(group): try: @@ -2615,3 +2620,10 @@ async def tools_group(t: Suite, mock) -> None: from . import _gemini_tools await _gemini_tools.tools(t, mock) + + +# Open WebUI's built-in image and code interpreter tools: suites/_gemini_owuitools.py +async def owuitools_group(t: Suite, mock) -> None: + from . import _gemini_owuitools + + await _gemini_owuitools.owuitools(t, mock)