Repository navigation
test(e2e), docs: Open WebUI's image generation and code interpreter with Gemini - #197
Merged
Merged
Conversation
New gemini group owuitools (suites/_gemini_owuitools.py, 10 checks): a Gemini text model calls Open WebUI's built-in generate_image, edit_image and execute_code through Open WebUI's tool loop (google_gemini.py 1.18.0). The group points Open WebUI's image engine "gemini" at the Gemini mock (generateContent and Imagen predict, image editing) and checks declaration (chat toggles, browser sessions only, edit_image only with image editing on), the engine requests, the saved images, the tool results sent back to Gemini, edits of an upload and of the previous turn's image, a pipe image model with the Image toggle, and the code interpreter with pyodide (the browser session answers execute:python) and with Jupyter. - mock_gemini.py: :predict (Imagen) answer - mock_tools.py: a minimal Jupyter kernel server under /jupyter/ - browser.py: BrowserSession answers execute:python (python_answer, python_calls) - docs/testing.md: the group, the mocks, measured counts (gemini 159, all 541 PASS on v0.11.4-slim; gemini 159 on v0.11.3-slim) +semver: none Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section under "Native tool calling": Open WebUI 0.11's built-in generate_image / edit_image (its Gemini image engine) and execute_code (code interpreter, pyodide or Jupyter) work with a Gemini text model: what to configure, when Open WebUI offers the tools (browser sessions, Native mode, global switches, capabilities, chat toggles, permissions) and the limits (API key only for the built-in engine, the pipeline's own image models get no tools). A pointer from "Image Generation Configuration". +semver: none Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#171 proposes moving Gemini image generation into a separate Tool, and discussion #80 asks whether Open WebUI's code interpreter works with the Gemini manifold. Open WebUI 0.11 has both on its own:
predict, Gemini image models viagenerateContent, plus a Gemini edit engine);In Native mode it offers them as the built-in tools
generate_image,edit_imageandexecute_code. Since pipeline 1.18.0, these reach Gemini like any other tool. This PR verifies that end to end and documents it. No pipeline change.Change
E2E: new gemini group
owuitools(10 checks,suites/_gemini_owuitools.py). Each check proves the mock answered (mock_ok=True).declareddeclared-noeditgenerate_imageis declared.apigenerate/generate-predictgenerate_imageround trip with a Gemini image model and with Imagen. The engine request is recorded, the image is saved with the answer, and its URL goes back to Gemini in the continuation (function call / response pairing, thought signature).edit/edit-followupedit_imageon an uploaded image and on an image generated in the previous turn.image-modelcode/code-jupyterexecute_codewith pyodide (the test browser answersexecute:python, and a printed PNG is uploaded and linked) and with Jupyter (Jupyter mock, no browser call).Against
google_gemini.py1.17.0, 6 of the 10 checks fail. That version ran the tools inside the pipe through automatic function calling, and the follow-up request broke the function call / response pairing.Mocks / harness:
:predictinmock_gemini.py, a Jupyter mock inmock_tools.py, andpython_answer/python_callsforexecute:pythonin the browser session. The group restores the image and code execution settings at the end.Docs: new section in
docs/google-gemini-integration.md→ "Open WebUI's image generation and code interpreter":docs/testing.mdcovers the group, the layout and the counts.Verification
--strict-known(on main ecac3fb)--strict-knownE2E_MOCK_FAULT=500)owuitoolschecks fail there, none is hiddenNot tested: whether the real Gemini/Imagen APIs accept the requests that Open WebUI's image engine sends. Those requests come from Open WebUI, not from this pipeline.
Refs #171, #80
🤖 Generated with Claude Code