Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,22 +94,23 @@ jobs:
name: Open WebUI ${{ matrix.target.label }}
needs: images
runs-on: ubuntu-latest
timeout-minutes: 45
timeout-minutes: 50
strategy:
fail-fast: false
matrix:
target: ${{ fromJSON(needs.images.outputs.targets) }}
env:
E2E_STRICT_KNOWN: "1"
# Time budget of the driver (25 min); with the start-up and the 60 s grace of
# run.sh it stays well below the step timeout.
E2E_TIMEOUT: "1500"
# Time budget of the driver (30 min, as the run.sh default; a full run takes
# about 17-19 min); with the start-up and the 60 s grace of run.sh it stays
# well below the step timeout.
E2E_TIMEOUT: "1800"
steps:
- name: Checkout
uses: actions/checkout@v7

- name: Run E2E suites
timeout-minutes: 35
timeout-minutes: 40
env:
IMAGE: ${{ matrix.target.image }}
SUITES: ${{ inputs.suites || 'all' }}
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ tests/e2e/check_owui_api.sh [TAG|latest] # static check of open_webui import

- **What to run:** changed `pipelines/google/*` → `gemini`; `pipelines/azure/*` → `azure`; `pipelines/n8n/*` → `n8n`; `pipelines/infomaniak/*` → `infomaniak`; `filters/*` → `filters` (+ `gemini` for `google_search_tool`); `tests/e2e/**` → `all`. New Open WebUI release → `check_owui_api.sh latest`, then `run.sh --image <new tag> all`.
- **Results:** `PASS` / `FAIL` / `KNOWN`. Exit 1 only on FAIL; exit 2 = setup error (Docker/container, leftover `NAME-data` volume without `--force`, an option without value, unknown suite, git ref or `--file` path, `--only` regex matching no group, driver error); 130/143 after Ctrl-C/SIGTERM. KNOWN = fails the way a bug registered in `tests/e2e/harness/known_<area>.py` does (its `evidence` matches the detail, or its log signature shows up with `since=`); a tagged check that fails differently is FAIL. KNOWN does not fail the run; its fixing branch may still be an unmerged/unpublished PR. Markers are **version-gated**: each `KnownIssue` names its `file` and the `fixed_in` version; once the tested file has that version, a failure is FAIL ("regression of known …") and a pass is listed as an obsolete marker ("drop marker"). So when you fix such a bug, bump the file's `version:` and set `fixed_in` to it; drop the markers once the fix is merged. `--only` is a Python regex: `'gemini.(api|image)'`, not `\|`. Output (git-ignored): `tests/e2e/out/<run>/` with `driver.txt`, `results.json`, `summary.md`, `server.log`, `mocks.txt`, staged `functions/` (LF); results are written for interrupted or timed-out runs too (partial).
- **Baseline** (v0.11.4-slim, `--strict-known`): `main` → 0 FAIL, 1 KNOWN (`infomaniak-name-prefix`, no fix yet), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`).
- **Baseline** (v0.11.4-slim, `--strict-known`): `main` → 0 FAIL, 0 KNOWN (no known bug registered), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`).
- **Driver checks:** `<suite>.timeout` (`E2E_SUITE_TIMEOUT`, default 900 s; whole run `E2E_TIMEOUT`, default 1800 s), `<suite>.crash`, `<suite>.no-checks`, `<suite>.interrupted`, `run.server-log` (errors or secrets logged outside every suite's scan window; names the `function_<id>`), `run.mocks-log` (errors in `mocks.txt`). Ctrl-C cleans up within seconds; a killed `run.sh` leaves the container, the volume and an orphan driver in the container, which `--reuse` stops. Suites are discovered from `tests/e2e/suites/` (no registration).
- **Add coverage** for every behaviour change: a `t.check(...)` in `tests/e2e/suites/<suite>.py` (API path `t.owui.chat`, browser path `t.browser().chat`, upstream request via `mock.last()`), mock behaviour in `tests/e2e/mocks/mock_<provider>.py`. Keep `tests/e2e` Ruff-clean. Evidence of a new known bug must not match unrelated failures: the CI meta-test (main's files with `E2E_MOCK_FAULT=500`) allows no KNOWN (`REQUEST_SIDE_KNOWN` in `.github/workflows/e2e.yml`, empty today, is only for bugs that can show nothing but the request sent upstream), so the evidence must include proof that the upstream answered (e.g. `HTTP 200` plus the mock's answer).
- **Gotchas:** `valves/update` REPLACES all valves (use `update_valves`, which merges); refresh `/api/models?refresh=true` after model/filterIds changes; settings in the data volume override env vars; only the browser path (socket.io + saved chat) exercises event emitters, saved content, usage and sources; background tasks (`/api/v1/tasks/*`) call pipes with `__task__` set and `__event_emitter__=None`; the first Gemini install pip-installs `google-genai` (~30-60 s); in Git Bash prefix your own `docker exec`/`docker cp` with `MSYS_NO_PATHCONV=1` per command (do not export it; `run.sh` unsets it because `git -C /c/...` needs path conversion); never remove Docker images, only your containers/volumes.
Expand Down
33 changes: 14 additions & 19 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,9 @@ function versions: google_gemini.py 1.18.0, azure_ai_foundry.py 3.0.0, n8n.py 2.
...
=== infomaniak ===
...
[KNOWN] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names
-> known infomaniak-name-prefix (found by tests/e2e, no issue filed): changing the
NAME_PREFIX valve does not change the model names (the prefix is read once in
__init__); no fix yet
name='Infomaniak: Mixtral Mock'
[PASS ] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names, and changing it back restores them
...
SUMMARY: 530 PASS, 0 FAIL, 1 KNOWN in 1089s
SUMMARY: 531 PASS, 0 FAIL, 0 KNOWN in 1089s
total runtime: 1123s
output: tests/e2e/out/20261009-111759-owui-e2e-111759-1234
```
Expand Down Expand Up @@ -210,24 +206,23 @@ query generation, errors, client `data_sources` with a key, also streamed); no l
contain a key or token.

Checks per suite on Open WebUI v0.11.4-slim in strict known mode (observed 2026-10-10,
`main` 4b80e24 (Azure pipeline 3.0.0) plus the Logs Ingestion API of #188 (2 new `spec`
checks, the `ingest` group with 41 and `log.no-secrets-debug`) and native tool calling
in `google_gemini.py` 1.18.0 with the #194 fix (the `tools` / `toolsapi` groups and the
19 `imgedit` checks)):
`main` d3184b9: Azure pipeline 3.0.0, Time Token Tracker 2.7.0 with the Logs Ingestion
API, native tool calling in `google_gemini.py` 1.18.0 with the #194 fix; plus the
`NAME_PREFIX` fix in `infomaniak.py` 2.2.2):

| Suite | Checks | PASS / KNOWN |
| --- | ---: | ---: |
| `gemini` | 149 | 149 / 0 |
| `azure` | 198 | 198 / 0 |
| `n8n` | 51 | 51 / 0 |
| `infomaniak` | 32 | 31 / 1 |
| `infomaniak` | 32 | 32 / 0 |
| `filters` | 101 | 101 / 0 |
| **all** | **531** | **530 / 1** |
| **all** | **531** | **531 / 0** |

No FAIL and no obsolete marker; `v0.11.3-slim` gives the same counts for every suite.
`harness/known_*.py` registers one known bug, `infomaniak-name-prefix`
(`NAME_PREFIX` is read only once, no fix yet, `fixed_in=""`): the one KNOWN. The markers
of the 62 bugs fixed by #182-#185 were dropped after the merge; their checks stay and
No known bug is registered (`harness/known_n8n.py` has no entry): the last one,
`infomaniak-name-prefix` (the `NAME_PREFIX` valve was read only once), was fixed in
`infomaniak.py` 2.2.2, and its marker went with the fix. The markers of the 62 bugs fixed by #182-#185 were dropped after the merge; their checks stay and
must pass. The `imgedit` group came with its fix and carries no marker either: with
`google_gemini.py` 1.18.0 before the fix (a7fc191) 15 of its 19 checks FAIL. The four
that pass check what did not change: `imgedit.temporary` (the fallback to the
Expand Down Expand Up @@ -362,11 +357,11 @@ is asked and the second is not run) are only recorded in the detail of
- `PASS` – the check held.
- `FAIL` – the check did not hold. The run exits with 1.
- `KNOWN` – the check did not hold because of a **known bug** registered in
`tests/e2e/harness/known_<area>.py` (today only `known_n8n.py` for n8n + Infomaniak;
re-exported by `known.py`): key, summary, issue reference, pull request with the
`tests/e2e/harness/known_<area>.py` (today only `known_n8n.py` for n8n + Infomaniak,
with no entry at the moment; re-exported by `known.py`): key, summary, issue reference, pull request with the
pending fix, evidence, and the function `file` plus the version `fixed_in` that fixes
it. Printed with the bug, e.g. `known infomaniak-name-prefix (found by tests/e2e, no
issue filed): ...; no fix yet`, or for a bug with a pending fix `...; fix pending in
it. Printed with the bug, e.g. `known <key> (found by tests/e2e, no issue
filed): ...; no fix yet`, or for a bug with a pending fix `...; fix pending in
PR #<n> (not merged yet), fixed in <file> <version>`. KNOWN does not fail the run.

A tagged check only counts as KNOWN when the failure **looks like that bug**: one of the
Expand Down
5 changes: 4 additions & 1 deletion pipelines/infomaniak/infomaniak.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
project_url: https://github.com/owndev/Open-WebUI-Functions
funding_url: https://github.com/sponsors/owndev
infomaniak_url: https://own.dev/infomaniak-com-en-hosting-ai-tools
version: 2.2.1
version: 2.2.2
required_open_webui_version: 0.8.0
license: Apache License 2.0
description: A manifold pipeline for interacting with Infomaniak AI Tools.
Expand Down Expand Up @@ -351,6 +351,9 @@ async def pipes(self) -> List[Dict[str, str]]:
Returns:
List of dictionaries containing pipe id and name.
"""
# Open WebUI reads self.name after pipes() to prefix the model names;
# refresh it so a changed NAME_PREFIX valve applies without a restart.
self.name = self.valves.NAME_PREFIX
self.validate_environment()
return await self.get_infomaniak_models()

Expand Down
6 changes: 3 additions & 3 deletions tests/e2e/harness/known.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
Registry of known bugs that make scenarios fail on ``main`` today.

The entries live in one module per area, ``known_<area>.py`` (today only
``known_n8n.py`` for n8n + Infomaniak), and are re-exported here, so suites write
``known.INFOMANIAK_NAME_PREFIX``. A new area module needs its own import at the
bottom of this file.
``known_n8n.py`` for n8n + Infomaniak, with no entry at the moment), and are
re-exported here, so suites write ``known.<NAME>``. A new area module needs its
own import at the bottom of this file.

A failing scenario that carries a ``KnownIssue`` is reported as KNOWN and does
not fail the run, but only when the failure looks like that bug:
Expand Down
21 changes: 6 additions & 15 deletions tests/e2e/harness/known_n8n.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,8 @@
"""Known bugs of pipelines/n8n/n8n.py and pipelines/infomaniak/infomaniak.py
(see ``harness.known``)."""
(see ``harness.known``).

from .known import FOUND_BY_E2E, KnownIssue

INFOMANIAK_FILE = "pipelines/infomaniak/infomaniak.py"

INFOMANIAK_NAME_PREFIX = KnownIssue(
"infomaniak-name-prefix",
"changing the NAME_PREFIX valve does not change the model names (the "
"prefix is read once in __init__)",
"", # no fix yet (follow-up issue)
ref=FOUND_BY_E2E,
evidence=(r"^name='Infomaniak: ",),
file=INFOMANIAK_FILE,
fixed_in="",
)
None is registered right now: the last one, ``infomaniak-name-prefix`` (the
NAME_PREFIX valve was read only once), was fixed in infomaniak.py 2.2.2 and its
check ``infomaniak.models.name-prefix`` must pass. A new entry imports
``FOUND_BY_E2E`` / ``KnownIssue`` from ``.known``.
"""
14 changes: 8 additions & 6 deletions tests/e2e/suites/infomaniak.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tasks background title task
"""

from harness import Suite, known, short
from harness import Suite, short

GROUPS = ("models", "api", "browser", "tasks")
FID = "infomaniak"
Expand Down Expand Up @@ -107,17 +107,19 @@ async def models(t: Suite) -> None:
and str(listed.get(model("mixtral"))).startswith(PREFIX),
f"listed={[(k, v) for k, v in listed.items() if k.startswith(FID + '.')]}",
)
base = str(listed.get(model("mixtral")))[len(PREFIX) :]
await t.owui.update_valves(FID, NAME_PREFIX="IK> ")
listed = {m["id"]: m.get("name") for m in await t.owui.models()}
name = listed.get(model("mixtral"))
await t.owui.update_valves(FID, NAME_PREFIX=PREFIX)
listed = {m["id"]: m.get("name") for m in await t.owui.models()}
restored = listed.get(model("mixtral"))
t.check(
"models.name-prefix",
"NAME_PREFIX valve changes the model names",
str(name).startswith("IK> "),
f"name={name!r}",
known=known.INFOMANIAK_NAME_PREFIX,
"NAME_PREFIX valve changes the model names, and changing it back restores them",
bool(base) and name == f"IK> {base}" and restored == f"{PREFIX}{base}",
f"name={name!r} restored={restored!r}",
)
await t.owui.update_valves(FID, NAME_PREFIX=PREFIX)


async def api(t: Suite, mock) -> None:
Expand Down
Loading