Skip to content

test(e2e), docs: Open WebUI's image generation and code interpreter with Gemini - #197

Merged
owndev merged 2 commits into
mainfrom
feature/gemini-builtin-tools-e2e
Oct 10, 2026
Merged

owndev merged 2 commits into
mainfrom
feature/gemini-builtin-tools-e2e

Conversation

@owndev

@owndev owndev commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Why

#171 proposes moving Gemini image generation into a separate Tool, and discussion #80 asks whether Open WebUI's code interpreter works with the Gemini manifold. Open WebUI 0.11 has both on its own:

  • a Gemini image engine (Imagen via predict, Gemini image models via generateContent, plus a Gemini edit engine);
  • the code interpreter (pyodide in the browser, or Jupyter).

In Native mode it offers them as the built-in tools generate_image, edit_image and execute_code. Since pipeline 1.18.0, these reach Gemini like any other tool. This PR verifies that end to end and documents it. No pipeline change.

Change

  • E2E: new gemini group owuitools (10 checks, suites/_gemini_owuitools.py). Each check proves the mock answered (mock_ok=True).

    Check What it shows
    declared All three tools are declared with the toggles on, none with the toggles off; the Pyodide note is added.
    declared-noedit With image editing off, only generate_image is declared.
    api API clients get no built-in tools.
    generate / generate-predict generate_image round trip with a Gemini image model and with Imagen. The engine request is recorded, the image is saved with the answer, and its URL goes back to Gemini in the continuation (function call / response pairing, thought signature).
    edit / edit-followup edit_image on an uploaded image and on an image generated in the previous turn.
    image-model A pipe image model gets no tools and makes the image itself.
    code / code-jupyter execute_code with pyodide (the test browser answers execute:python, and a printed PNG is uploaded and linked) and with Jupyter (Jupyter mock, no browser call).

    Against google_gemini.py 1.17.0, 6 of the 10 checks fail. That version ran the tools inside the pipe through automatic function calling, and the follow-up request broke the function call / response pairing.

  • Mocks / harness: :predict in mock_gemini.py, a Jupyter mock in mock_tools.py, and python_answer / python_calls for execute:python in the browser session. The group restores the image and code execution settings at the end.

  • Docs: new section in docs/google-gemini-integration.md → "Open WebUI's image generation and code interpreter":

    • setup with environment variables;
    • when Open WebUI offers the tools;
    • limits: the built-in engine uses a Gemini API key only (no Vertex AI login), and the pipeline's 🎨 image models stay for direct use.

    docs/testing.md covers the group, the layout and the counts.

Verification

Run Result
All suites, v0.11.4-slim, --strict-known (on main ecac3fb) 541 PASS, 0 FAIL, 0 KNOWN (gemini 159, azure 198, n8n 51, infomaniak 32, filters 101), 1138 s
gemini, v0.11.3-slim and v0.11.4-slim, --strict-known 159 PASS, 0 FAIL, 0 KNOWN (each)
Meta-test (E2E_MOCK_FAULT=500) 57 PASS, 380 FAIL, 0 KNOWN; all 10 owuitools checks fail there, none is hidden
ruff 0.11.10 check + format clean

Not tested: whether the real Gemini/Imagen APIs accept the requests that Open WebUI's image engine sends. Those requests come from Open WebUI, not from this pipeline.

Refs #171, #80

🤖 Generated with Claude Code

owndev and others added 2 commits October 10, 2026 21:20
New gemini group owuitools (suites/_gemini_owuitools.py, 10 checks): a
Gemini text model calls Open WebUI's built-in generate_image, edit_image
and execute_code through Open WebUI's tool loop (google_gemini.py
1.18.0). The group points Open WebUI's image engine "gemini" at the
Gemini mock (generateContent and Imagen predict, image editing) and
checks declaration (chat toggles, browser sessions only, edit_image only
with image editing on), the engine requests, the saved images, the tool
results sent back to Gemini, edits of an upload and of the previous
turn's image, a pipe image model with the Image toggle, and the code
interpreter with pyodide (the browser session answers execute:python)
and with Jupyter.

- mock_gemini.py: :predict (Imagen) answer
- mock_tools.py: a minimal Jupyter kernel server under /jupyter/
- browser.py: BrowserSession answers execute:python (python_answer,
  python_calls)
- docs/testing.md: the group, the mocks, measured counts (gemini 159,
  all 541 PASS on v0.11.4-slim; gemini 159 on v0.11.3-slim)

+semver: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section under "Native tool calling": Open WebUI 0.11's built-in
generate_image / edit_image (its Gemini image engine) and execute_code
(code interpreter, pyodide or Jupyter) work with a Gemini text model:
what to configure, when Open WebUI offers the tools (browser sessions,
Native mode, global switches, capabilities, chat toggles, permissions)
and the limits (API key only for the built-in engine, the pipeline's own
image models get no tools). A pointer from "Image Generation
Configuration".

+semver: none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@owndev
owndev merged commit 2fc7bcc into main Oct 10, 2026
8 checks passed
@owndev
owndev deleted the feature/gemini-builtin-tools-e2e branch October 10, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant