diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 8434f0d..29ceadf 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -94,22 +94,23 @@ jobs: name: Open WebUI ${{ matrix.target.label }} needs: images runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 50 strategy: fail-fast: false matrix: target: ${{ fromJSON(needs.images.outputs.targets) }} env: E2E_STRICT_KNOWN: "1" - # Time budget of the driver (25 min); with the start-up and the 60 s grace of - # run.sh it stays well below the step timeout. - E2E_TIMEOUT: "1500" + # Time budget of the driver (30 min, as the run.sh default; a full run takes + # about 17-19 min); with the start-up and the 60 s grace of run.sh it stays + # well below the step timeout. + E2E_TIMEOUT: "1800" steps: - name: Checkout uses: actions/checkout@v7 - name: Run E2E suites - timeout-minutes: 35 + timeout-minutes: 40 env: IMAGE: ${{ matrix.target.image }} SUITES: ${{ inputs.suites || 'all' }} diff --git a/CLAUDE.md b/CLAUDE.md index e8cf175..f981d29 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,7 +33,7 @@ tests/e2e/check_owui_api.sh [TAG|latest] # static check of open_webui import - **What to run:** changed `pipelines/google/*` → `gemini`; `pipelines/azure/*` → `azure`; `pipelines/n8n/*` → `n8n`; `pipelines/infomaniak/*` → `infomaniak`; `filters/*` → `filters` (+ `gemini` for `google_search_tool`); `tests/e2e/**` → `all`. New Open WebUI release → `check_owui_api.sh latest`, then `run.sh --image all`. - **Results:** `PASS` / `FAIL` / `KNOWN`. Exit 1 only on FAIL; exit 2 = setup error (Docker/container, leftover `NAME-data` volume without `--force`, an option without value, unknown suite, git ref or `--file` path, `--only` regex matching no group, driver error); 130/143 after Ctrl-C/SIGTERM. KNOWN = fails the way a bug registered in `tests/e2e/harness/known_.py` does (its `evidence` matches the detail, or its log signature shows up with `since=`); a tagged check that fails differently is FAIL. KNOWN does not fail the run; its fixing branch may still be an unmerged/unpublished PR. Markers are **version-gated**: each `KnownIssue` names its `file` and the `fixed_in` version; once the tested file has that version, a failure is FAIL ("regression of known …") and a pass is listed as an obsolete marker ("drop marker"). So when you fix such a bug, bump the file's `version:` and set `fixed_in` to it; drop the markers once the fix is merged. `--only` is a Python regex: `'gemini.(api|image)'`, not `\|`. Output (git-ignored): `tests/e2e/out//` with `driver.txt`, `results.json`, `summary.md`, `server.log`, `mocks.txt`, staged `functions/` (LF); results are written for interrupted or timed-out runs too (partial). -- **Baseline** (v0.11.4-slim, `--strict-known`): `main` → 0 FAIL, 1 KNOWN (`infomaniak-name-prefix`, no fix yet), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`). +- **Baseline** (v0.11.4-slim, `--strict-known`): `main` → 0 FAIL, 0 KNOWN (no known bug registered), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`). - **Driver checks:** `.timeout` (`E2E_SUITE_TIMEOUT`, default 900 s; whole run `E2E_TIMEOUT`, default 1800 s), `.crash`, `.no-checks`, `.interrupted`, `run.server-log` (errors or secrets logged outside every suite's scan window; names the `function_`), `run.mocks-log` (errors in `mocks.txt`). Ctrl-C cleans up within seconds; a killed `run.sh` leaves the container, the volume and an orphan driver in the container, which `--reuse` stops. Suites are discovered from `tests/e2e/suites/` (no registration). - **Add coverage** for every behaviour change: a `t.check(...)` in `tests/e2e/suites/.py` (API path `t.owui.chat`, browser path `t.browser().chat`, upstream request via `mock.last()`), mock behaviour in `tests/e2e/mocks/mock_.py`. Keep `tests/e2e` Ruff-clean. Evidence of a new known bug must not match unrelated failures: the CI meta-test (main's files with `E2E_MOCK_FAULT=500`) allows no KNOWN (`REQUEST_SIDE_KNOWN` in `.github/workflows/e2e.yml`, empty today, is only for bugs that can show nothing but the request sent upstream), so the evidence must include proof that the upstream answered (e.g. `HTTP 200` plus the mock's answer). - **Gotchas:** `valves/update` REPLACES all valves (use `update_valves`, which merges); refresh `/api/models?refresh=true` after model/filterIds changes; settings in the data volume override env vars; only the browser path (socket.io + saved chat) exercises event emitters, saved content, usage and sources; background tasks (`/api/v1/tasks/*`) call pipes with `__task__` set and `__event_emitter__=None`; the first Gemini install pip-installs `google-genai` (~30-60 s); in Git Bash prefix your own `docker exec`/`docker cp` with `MSYS_NO_PATHCONV=1` per command (do not export it; `run.sh` unsets it because `git -C /c/...` needs path conversion); never remove Docker images, only your containers/volumes. diff --git a/docs/testing.md b/docs/testing.md index 6bed98c..8a4ea68 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -61,13 +61,9 @@ function versions: google_gemini.py 1.18.0, azure_ai_foundry.py 3.0.0, n8n.py 2. ... === infomaniak === ... -[KNOWN] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names - -> known infomaniak-name-prefix (found by tests/e2e, no issue filed): changing the - NAME_PREFIX valve does not change the model names (the prefix is read once in - __init__); no fix yet - name='Infomaniak: Mixtral Mock' +[PASS ] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names, and changing it back restores them ... -SUMMARY: 530 PASS, 0 FAIL, 1 KNOWN in 1089s +SUMMARY: 531 PASS, 0 FAIL, 0 KNOWN in 1089s total runtime: 1123s output: tests/e2e/out/20261009-111759-owui-e2e-111759-1234 ``` @@ -210,24 +206,23 @@ query generation, errors, client `data_sources` with a key, also streamed); no l contain a key or token. Checks per suite on Open WebUI v0.11.4-slim in strict known mode (observed 2026-10-10, -`main` 4b80e24 (Azure pipeline 3.0.0) plus the Logs Ingestion API of #188 (2 new `spec` -checks, the `ingest` group with 41 and `log.no-secrets-debug`) and native tool calling -in `google_gemini.py` 1.18.0 with the #194 fix (the `tools` / `toolsapi` groups and the -19 `imgedit` checks)): +`main` d3184b9: Azure pipeline 3.0.0, Time Token Tracker 2.7.0 with the Logs Ingestion +API, native tool calling in `google_gemini.py` 1.18.0 with the #194 fix; plus the +`NAME_PREFIX` fix in `infomaniak.py` 2.2.2): | Suite | Checks | PASS / KNOWN | | --- | ---: | ---: | | `gemini` | 149 | 149 / 0 | | `azure` | 198 | 198 / 0 | | `n8n` | 51 | 51 / 0 | -| `infomaniak` | 32 | 31 / 1 | +| `infomaniak` | 32 | 32 / 0 | | `filters` | 101 | 101 / 0 | -| **all** | **531** | **530 / 1** | +| **all** | **531** | **531 / 0** | No FAIL and no obsolete marker; `v0.11.3-slim` gives the same counts for every suite. -`harness/known_*.py` registers one known bug, `infomaniak-name-prefix` -(`NAME_PREFIX` is read only once, no fix yet, `fixed_in=""`): the one KNOWN. The markers -of the 62 bugs fixed by #182-#185 were dropped after the merge; their checks stay and +No known bug is registered (`harness/known_n8n.py` has no entry): the last one, +`infomaniak-name-prefix` (the `NAME_PREFIX` valve was read only once), was fixed in +`infomaniak.py` 2.2.2, and its marker went with the fix. The markers of the 62 bugs fixed by #182-#185 were dropped after the merge; their checks stay and must pass. The `imgedit` group came with its fix and carries no marker either: with `google_gemini.py` 1.18.0 before the fix (a7fc191) 15 of its 19 checks FAIL. The four that pass check what did not change: `imgedit.temporary` (the fallback to the @@ -362,11 +357,11 @@ is asked and the second is not run) are only recorded in the detail of - `PASS` – the check held. - `FAIL` – the check did not hold. The run exits with 1. - `KNOWN` – the check did not hold because of a **known bug** registered in - `tests/e2e/harness/known_.py` (today only `known_n8n.py` for n8n + Infomaniak; - re-exported by `known.py`): key, summary, issue reference, pull request with the + `tests/e2e/harness/known_.py` (today only `known_n8n.py` for n8n + Infomaniak, + with no entry at the moment; re-exported by `known.py`): key, summary, issue reference, pull request with the pending fix, evidence, and the function `file` plus the version `fixed_in` that fixes - it. Printed with the bug, e.g. `known infomaniak-name-prefix (found by tests/e2e, no - issue filed): ...; no fix yet`, or for a bug with a pending fix `...; fix pending in + it. Printed with the bug, e.g. `known (found by tests/e2e, no issue + filed): ...; no fix yet`, or for a bug with a pending fix `...; fix pending in PR # (not merged yet), fixed in `. KNOWN does not fail the run. A tagged check only counts as KNOWN when the failure **looks like that bug**: one of the diff --git a/pipelines/infomaniak/infomaniak.py b/pipelines/infomaniak/infomaniak.py index 360b898..ceb7076 100644 --- a/pipelines/infomaniak/infomaniak.py +++ b/pipelines/infomaniak/infomaniak.py @@ -5,7 +5,7 @@ project_url: https://github.com/owndev/Open-WebUI-Functions funding_url: https://github.com/sponsors/owndev infomaniak_url: https://own.dev/infomaniak-com-en-hosting-ai-tools -version: 2.2.1 +version: 2.2.2 required_open_webui_version: 0.8.0 license: Apache License 2.0 description: A manifold pipeline for interacting with Infomaniak AI Tools. @@ -351,6 +351,9 @@ async def pipes(self) -> List[Dict[str, str]]: Returns: List of dictionaries containing pipe id and name. """ + # Open WebUI reads self.name after pipes() to prefix the model names; + # refresh it so a changed NAME_PREFIX valve applies without a restart. + self.name = self.valves.NAME_PREFIX self.validate_environment() return await self.get_infomaniak_models() diff --git a/tests/e2e/harness/known.py b/tests/e2e/harness/known.py index d04327e..f8bfeac 100644 --- a/tests/e2e/harness/known.py +++ b/tests/e2e/harness/known.py @@ -2,9 +2,9 @@ Registry of known bugs that make scenarios fail on ``main`` today. The entries live in one module per area, ``known_.py`` (today only -``known_n8n.py`` for n8n + Infomaniak), and are re-exported here, so suites write -``known.INFOMANIAK_NAME_PREFIX``. A new area module needs its own import at the -bottom of this file. +``known_n8n.py`` for n8n + Infomaniak, with no entry at the moment), and are +re-exported here, so suites write ``known.``. A new area module needs its +own import at the bottom of this file. A failing scenario that carries a ``KnownIssue`` is reported as KNOWN and does not fail the run, but only when the failure looks like that bug: diff --git a/tests/e2e/harness/known_n8n.py b/tests/e2e/harness/known_n8n.py index fe52979..5f689be 100644 --- a/tests/e2e/harness/known_n8n.py +++ b/tests/e2e/harness/known_n8n.py @@ -1,17 +1,8 @@ """Known bugs of pipelines/n8n/n8n.py and pipelines/infomaniak/infomaniak.py -(see ``harness.known``).""" +(see ``harness.known``). -from .known import FOUND_BY_E2E, KnownIssue - -INFOMANIAK_FILE = "pipelines/infomaniak/infomaniak.py" - -INFOMANIAK_NAME_PREFIX = KnownIssue( - "infomaniak-name-prefix", - "changing the NAME_PREFIX valve does not change the model names (the " - "prefix is read once in __init__)", - "", # no fix yet (follow-up issue) - ref=FOUND_BY_E2E, - evidence=(r"^name='Infomaniak: ",), - file=INFOMANIAK_FILE, - fixed_in="", -) +None is registered right now: the last one, ``infomaniak-name-prefix`` (the +NAME_PREFIX valve was read only once), was fixed in infomaniak.py 2.2.2 and its +check ``infomaniak.models.name-prefix`` must pass. A new entry imports +``FOUND_BY_E2E`` / ``KnownIssue`` from ``.known``. +""" diff --git a/tests/e2e/suites/infomaniak.py b/tests/e2e/suites/infomaniak.py index 449d6e9..11fc000 100644 --- a/tests/e2e/suites/infomaniak.py +++ b/tests/e2e/suites/infomaniak.py @@ -13,7 +13,7 @@ tasks background title task """ -from harness import Suite, known, short +from harness import Suite, short GROUPS = ("models", "api", "browser", "tasks") FID = "infomaniak" @@ -107,17 +107,19 @@ async def models(t: Suite) -> None: and str(listed.get(model("mixtral"))).startswith(PREFIX), f"listed={[(k, v) for k, v in listed.items() if k.startswith(FID + '.')]}", ) + base = str(listed.get(model("mixtral")))[len(PREFIX) :] await t.owui.update_valves(FID, NAME_PREFIX="IK> ") listed = {m["id"]: m.get("name") for m in await t.owui.models()} name = listed.get(model("mixtral")) + await t.owui.update_valves(FID, NAME_PREFIX=PREFIX) + listed = {m["id"]: m.get("name") for m in await t.owui.models()} + restored = listed.get(model("mixtral")) t.check( "models.name-prefix", - "NAME_PREFIX valve changes the model names", - str(name).startswith("IK> "), - f"name={name!r}", - known=known.INFOMANIAK_NAME_PREFIX, + "NAME_PREFIX valve changes the model names, and changing it back restores them", + bool(base) and name == f"IK> {base}" and restored == f"{PREFIX}{base}", + f"name={name!r} restored={restored!r}", ) - await t.owui.update_valves(FID, NAME_PREFIX=PREFIX) async def api(t: Suite, mock) -> None: