From d857662436d8463104a4cc567185dcd77c3a8e04 Mon Sep 17 00:00:00 2001 From: "Claude Opus 4.8" Date: Wed, 29 Jul 2026 16:44:51 +0200 Subject: [PATCH 1/5] feat(manifest): add upstream_docs to watch vendor register sources A driver decodes a device by following the vendor's own reference material - a register map, a parameter changelog PDF. When that material changes upstream, the driver can silently fall behind. upstream_docs records those documents at a semi-persistent URL so a watcher can poll them and flag the driver for review, instead of a human noticing months later. - new optional manifest field: list of {url, title?, kind?} entries - parse_upstream_docs() in manifest_parser.py, mirroring parse_tested_devices - validate_manifest.py enforces an http(s) url and a known kind - nibe_local + myuplink both declare NIBE's myUplink register changelog - nibe_local manifest author corrected to its real provenance - descriptive metadata only: never copied into index.yaml - documented in spec/manifest-v2.md (V2.3); tests in test_upstream_docs.py Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com> --- CHANGELOG.md | 2 + manifests/myuplink.yaml | 4 + manifests/nibe_local.yaml | 6 +- spec/manifest-v2.md | 30 ++++++++ tests/test_upstream_docs.py | 144 ++++++++++++++++++++++++++++++++++++ tools/manifest_parser.py | 58 +++++++++++++++ tools/validate_manifest.py | 29 +++++++- 7 files changed, 271 insertions(+), 2 deletions(-) create mode 100644 tests/test_upstream_docs.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 0362f80..3c7c25c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,6 +107,7 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html ### Changed - The 19 drivers whose implementation was replaced by FTW's move to a **major version**. The signed channel refuses to publish changed bytes under a version it already published — that immutability is what makes rollback mean anything — so the release failed until they moved. A major bump is also the honest signal: these are different implementations, not patches, and an operator pinning a version needs to know that +- **nibe_local** manifest `author` is now `HuggeK with the help of Claude Code`, matching the driver's own `DRIVER.authors` and its real provenance rather than the `Sourceful Labs AB` default the V1→V2 migration stamped on every core manifest ### Fixed - **sungrow** 1.4.0 — **The SG12RT outage.** Sungrow ships two families behind one driver: SH hybrids answer the 13xxx block, SG string inverters have no battery and answer none of it. The driver read that block regardless, and because the host fails a whole poll when any single read fails, a string inverter did not report less telemetry — it reported none. A customer's SG12RT lost everything, 12 of 19 reads failing on every poll. It now asks the device which family it is, using the classification `driver_fingerprint` already had, and reads the hybrid block only when there is reason to. Measured: **zero failed reads from the first poll**, where it was eight @@ -123,6 +124,7 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html ### Added - **`blueprint/BLUEPRINT.lua`** — a complete, working driver for an imaginary inverter, written for people and agents alike. Every rule this repository enforces appears beside the code that follows it: bounded probing, never fabricating a zero, decoding on 16-bit halves, negating at the sign boundary. `tests/test_blueprint.py` holds it to every rule a shipped driver must meet and runs it against the harness, including the test that matters most — that failed reads per poll settle to zero, because a driver that never gives up takes the site offline - **`docs/WRITING-A-DRIVER.md`** — the reasoning behind each rule, with the blueprint as the specification +- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and a known kind; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` ### Removed - `tools/canonical_debt.py` measured distance from an ideal that never existed. In its place `tools/host_api_check.py` asks the only question that predicts a crash: does a driver call a function no host provides? That is what 35 drivers were doing while passing every test here diff --git a/manifests/myuplink.yaml b/manifests/myuplink.yaml index 17090d3..b0b844a 100644 --- a/manifests/myuplink.yaml +++ b/manifests/myuplink.yaml @@ -13,6 +13,10 @@ tested_devices: firmware_versions: "" notes: "Read-only heat-pump telemetry via MyUplink Cloud REST API v2: compressor power + hot-water/indoor/outdoor temperatures. Observe-only — no control. OAuth: authorization-code + refresh-token (connect in Settings → Devices)." min_driver_version: "1.0.0" +upstream_docs: + - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" + title: "NIBE myUplink register/parameter changelog (nibe-n.pdf)" + kind: changelog min_host_version: "2.0.0" size_bytes: 15278 dkb_id: "myuplink" diff --git a/manifests/nibe_local.yaml b/manifests/nibe_local.yaml index 5e9d478..329e357 100644 --- a/manifests/nibe_local.yaml +++ b/manifests/nibe_local.yaml @@ -1,7 +1,7 @@ name: "nibe_local" version: "1.1.1" tier: core -author: "Sourceful Labs AB" +author: "HuggeK with the help of Claude Code" protocol: http ders: [heatpump] control: true @@ -13,6 +13,10 @@ tested_devices: firmware_versions: "" notes: "Read-only NIBE S-series heat-pump telemetry over the on-prem Local REST API (HTTPS + Basic auth, self-signed cert pinned via tls_pin_sha256). Emits compressor/used power, lifetime energy meters, and the full ~980-point register map. Observe-only — no control." min_driver_version: "1.0.0" +upstream_docs: + - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" + title: "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)" + kind: changelog min_host_version: "2.0.0" size_bytes: 18227 dkb_id: "nibe_local" diff --git a/spec/manifest-v2.md b/spec/manifest-v2.md index 702993c..cbf114d 100644 --- a/spec/manifest-v2.md +++ b/spec/manifest-v2.md @@ -14,6 +14,7 @@ Each driver has a YAML manifest in `manifests/` that describes its metadata, cap | `ders` | list | Yes | DER types: `pv`, `battery`, `meter`, `v2x_charger` | | `control` | bool | Yes | Whether the driver supports EMS control commands | | `tested_devices` | list | No | Devices tested against (see below) | +| `upstream_docs` | list | No | Vendor reference documents to watch for register/protocol changes (see below) | | `min_host_version` | string | No | Minimum gateway firmware version required | | `size_bytes` | int | Yes | Size of the `.lua` file in bytes | | `dkb_id` | string | No | Corresponding Hugin DKB device profile ID | @@ -40,6 +41,32 @@ The `tested_devices` list describes the device models a driver has been verified A driver may list multiple `tested_devices` entries if it supports devices from different manufacturers or distinct model families. +## Upstream Docs + +A driver decodes a device by following the vendor's own reference material: a +register map, a Modbus map, a parameter/changelog PDF, an API reference. When +that material changes upstream — a register renumbered, a parameter added — the +driver may silently fall out of date. `upstream_docs` records those documents at +a **semi-persistent URL** so a watcher can poll them and flag the driver for +review when the source moves, instead of a human noticing months later. + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `url` | string | Yes | Semi-persistent `http(s)` URL of the document to watch | +| `title` | string | No | Human-readable label (e.g., `"NIBE S-series register changelog"`) | +| `kind` | string | No | One of: `changelog`, `register_map`, `manual`, `api_docs`, `firmware_notes`, `other` | + +A driver may list multiple `upstream_docs` entries — for example a register map +and a separate changelog. The field is descriptive metadata only: it is not +copied into `index.yaml` and never affects how a driver is installed or run. + +```yaml +upstream_docs: + - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" + title: "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)" + kind: changelog +``` + ## Example ```yaml @@ -88,6 +115,7 @@ Version changes must be accompanied by a `CHANGELOG.md` entry under `[Unreleased 7. `core` tier drivers must have an `author` 8. Every manifest must have a corresponding `.lua` file in `drivers/` 9. `tested_devices` entries must have `manufacturer` and `model_family` +10. `upstream_docs` entries must have an `http(s)` `url`; `kind`, when present, must be a known kind ## Migration from V1 @@ -99,4 +127,6 @@ V2.1 extends `tested_devices` with: `model_family` (replaces `model`), `variants V2.2 adds bytecode fields: `bytecode_sha256`, `bytecode_signature`, `bytecode_size` for Lua 5.5.0 compiled bytecode. +V2.3 adds `upstream_docs`: an optional list of vendor reference documents (`url`, `title`, `kind`) to watch for register/protocol changes. + Use `tools/migrate_manifests.py` to convert V1 JSON → V2 YAML. diff --git a/tests/test_upstream_docs.py b/tests/test_upstream_docs.py new file mode 100644 index 0000000..22fc3ed --- /dev/null +++ b/tests/test_upstream_docs.py @@ -0,0 +1,144 @@ +"""upstream_docs: the vendor documents a driver is watched against. + +A driver decodes a device by following the vendor's own register map or +changelog. When that document moves upstream, the driver can fall behind +without anything noticing. `upstream_docs` records those documents at a +semi-persistent URL so a watcher can poll them and flag the driver for review. + +These tests hold the field to its contract and check that the two NIBE drivers +actually declare the changelog they were built against. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tools")) + +from manifest_parser import parse_tested_devices, parse_upstream_docs # noqa: E402 +from validate_manifest import validate_manifest # noqa: E402 + +MANIFEST_DIR = ROOT / "manifests" +NIBE_CHANGELOG = "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" +NIBE_IDS = ["nibe_local", "myuplink"] + + +# ---- Parser ------------------------------------------------------------- + + +def test_absent_block_parses_as_empty(): + assert parse_upstream_docs("name: x\nversion: 1.0.0\n") == [] + + +def test_empty_inline_list_parses_as_empty(): + assert parse_upstream_docs("upstream_docs: []\nmin_host_version: 2.0.0\n") == [] + + +def test_block_parses_url_title_kind(): + text = ( + "tested_devices:\n" + ' - manufacturer: "NIBE"\n' + ' model_family: "S"\n' + "upstream_docs:\n" + ' - url: "https://example.com/a.pdf"\n' + ' title: "A"\n' + " kind: changelog\n" + ' - url: "https://example.com/b.pdf"\n' + " kind: register_map\n" + 'min_host_version: "2.0.0"\n' + ) + assert parse_upstream_docs(text) == [ + {"url": "https://example.com/a.pdf", "title": "A", "kind": "changelog"}, + {"url": "https://example.com/b.pdf", "kind": "register_map"}, + ] + + +def test_block_does_not_bleed_into_tested_devices(): + """The two adjacent blocks must not consume each other's lines.""" + text = ( + "tested_devices:\n" + ' - manufacturer: "NIBE"\n' + ' model_family: "S"\n' + "upstream_docs:\n" + ' - url: "https://example.com/a.pdf"\n' + ' title: "A"\n' + 'min_host_version: "2.0.0"\n' + ) + assert parse_tested_devices(text) == [{"manufacturer": "NIBE", "model_family": "S"}] + assert parse_upstream_docs(text) == [ + {"url": "https://example.com/a.pdf", "title": "A"} + ] + + +# ---- Validation --------------------------------------------------------- + + +def _errors_for(tmp_path: Path, upstream_block: str) -> list[str]: + body = ( + 'name: "nibe_local"\n' + 'version: "1.0.0"\n' + "tier: community\n" + "protocol: http\n" + "ders: [meter]\n" + "size_bytes: 1\n" + f"{upstream_block}" + ) + path = tmp_path / "nibe_local.yaml" + path.write_text(body, encoding="utf-8") + return validate_manifest(path, tmp_path / "nonexistent") + + +def test_url_is_required(tmp_path): + errors = _errors_for(tmp_path, 'upstream_docs:\n - title: "no url"\n') + assert any("upstream_docs[0]: missing required field 'url'" in e for e in errors) + + +def test_non_http_url_is_rejected(tmp_path): + errors = _errors_for(tmp_path, 'upstream_docs:\n - url: "ftp://example.com/a.pdf"\n') + assert any("upstream_docs[0]: url must be an http(s) URL" in e for e in errors) + + +def test_unknown_kind_is_rejected(tmp_path): + errors = _errors_for( + tmp_path, 'upstream_docs:\n - url: "https://x/a.pdf"\n kind: bogus\n' + ) + assert any("upstream_docs[0]: kind 'bogus' is not valid" in e for e in errors) + + +def test_unknown_field_is_rejected(tmp_path): + errors = _errors_for( + tmp_path, 'upstream_docs:\n - url: "https://x/a.pdf"\n foo: bar\n' + ) + assert any("upstream_docs[0]: unknown field 'foo'" in e for e in errors) + + +def test_wellformed_entry_raises_no_upstream_error(tmp_path): + errors = _errors_for( + tmp_path, + 'upstream_docs:\n - url: "https://x/a.pdf"\n title: "A"\n kind: changelog\n', + ) + assert not [e for e in errors if e.startswith("upstream_docs")] + + +# ---- The shipped manifests --------------------------------------------- + + +@pytest.mark.parametrize("driver_id", NIBE_IDS) +def test_nibe_drivers_watch_the_changelog(driver_id): + text = (MANIFEST_DIR / f"{driver_id}.yaml").read_text(encoding="utf-8") + docs = parse_upstream_docs(text) + assert docs, f"{driver_id} declares no upstream_docs" + assert NIBE_CHANGELOG in [d.get("url") for d in docs] + assert all(d.get("kind") == "changelog" for d in docs) + + +def test_every_declared_upstream_doc_is_wellformed(): + for path in sorted(MANIFEST_DIR.glob("*.yaml")): + for i, doc in enumerate(parse_upstream_docs(path.read_text(encoding="utf-8"))): + url = doc.get("url", "") + assert url.startswith(("http://", "https://")), f"{path.name}[{i}]: bad url" + assert set(doc) <= {"url", "title", "kind"}, f"{path.name}[{i}]: unknown field" diff --git a/tools/manifest_parser.py b/tools/manifest_parser.py index d2ea011..9eeafb5 100644 --- a/tools/manifest_parser.py +++ b/tools/manifest_parser.py @@ -104,6 +104,64 @@ def parse_tested_devices(text: str) -> list[dict]: return devices +def parse_upstream_docs(text: str) -> list[dict]: + """Parse the optional upstream_docs block from manifest text. + + upstream_docs lists the vendor reference documents a driver was built + against — register maps, parameter changelogs, protocol manuals — at a + semi-persistent URL. A change to one of these is the upstream signal that + a driver's register decoding may need review. Each entry carries a `url` + and optional `title`/`kind`. + + Mirrors parse_tested_devices(): a top-level block of "- " list items. + Returns a list of doc dicts (empty when the block is absent or `[]`). + """ + docs = [] + current = None + in_block = False + + for line in text.splitlines(): + stripped = line.strip() + + # Detect start of upstream_docs block + if stripped.startswith("upstream_docs:"): + in_block = True + # Handle empty list: upstream_docs: [] + if stripped == "upstream_docs: []": + return [] + continue + + # Detect end of block (next top-level key) + if in_block and stripped and not line.startswith(" ") and not line.startswith("\t"): + break + + if not in_block: + continue + + # Skip empty lines and comments + if not stripped or stripped.startswith("#"): + continue + + # New doc entry: "- key: value" or just "-" + if stripped.startswith("- "): + if current is not None: + docs.append(current) + current = {} + rest = stripped[2:].strip() + if rest: + _parse_device_field(current, rest) + continue + + # Continuation field within current doc + if current is not None and ":" in stripped: + _parse_device_field(current, stripped) + + if current is not None: + docs.append(current) + + return docs + + def _parse_device_field(device: dict, field_str: str) -> None: """Parse a single key: value field into a device dict.""" # Handle inline lists: key: [a, b, c] diff --git a/tools/validate_manifest.py b/tools/validate_manifest.py index 31b5b28..250e96a 100644 --- a/tools/validate_manifest.py +++ b/tools/validate_manifest.py @@ -11,7 +11,7 @@ from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) -from manifest_parser import parse_yaml_simple, parse_tested_devices +from manifest_parser import parse_yaml_simple, parse_tested_devices, parse_upstream_docs REQUIRED_FIELDS = ["name", "version", "tier", "protocol", "ders", "size_bytes"] VALID_TIERS = {"core", "community", "oem"} @@ -27,6 +27,13 @@ VALID_DEVICE_FIELDS = {"manufacturer", "model_family", "model", "variants", "regions", "firmware_versions", "min_driver_version", "notes"} +# Upstream reference documents a driver was built against (register maps, +# parameter changelogs, manuals). Their URLs are semi-persistent so a watcher +# can poll them and flag a driver for review when the source moves. +VALID_UPSTREAM_DOC_FIELDS = {"url", "title", "kind"} +VALID_UPSTREAM_DOC_KINDS = {"changelog", "register_map", "manual", "api_docs", + "firmware_notes", "other"} + def validate_manifest(yaml_path: Path, drivers_dir: Path) -> list[str]: """Validate a single manifest file. Returns list of error strings.""" @@ -117,6 +124,26 @@ def validate_manifest(yaml_path: Path, drivers_dir: Path) -> list[str]: if key not in VALID_DEVICE_FIELDS: errors.append(f"{prefix}: unknown field '{key}'") + # Validate upstream_docs + docs = parse_upstream_docs(text) + for i, doc in enumerate(docs): + prefix = f"upstream_docs[{i}]" + + url = doc.get("url", "") + if not url: + errors.append(f"{prefix}: missing required field 'url'") + elif not (url.startswith("http://") or url.startswith("https://")): + errors.append(f"{prefix}: url must be an http(s) URL, got '{url}'") + + kind = doc.get("kind", "") + if kind and kind not in VALID_UPSTREAM_DOC_KINDS: + errors.append(f"{prefix}: kind '{kind}' is not valid " + f"(expected: {', '.join(sorted(VALID_UPSTREAM_DOC_KINDS))})") + + for key in doc: + if key not in VALID_UPSTREAM_DOC_FIELDS: + errors.append(f"{prefix}: unknown field '{key}'") + return errors From 4e274e5f92f3ff82c0e15ea3eccce7e4031e69d1 Mon Sep 17 00:00:00 2001 From: "Claude Opus 4.8" Date: Wed, 29 Jul 2026 17:12:43 +0200 Subject: [PATCH 2/5] feat(watch): add scheduled watcher for upstream_docs + url_stability Adds the automation half of upstream_docs and a stability signal. Watcher: - tools/check_upstream_docs.py fetches each watched URL, hashes it, and diffs against a committed baseline (upstream-docs-state.json). - .github/workflows/watch-upstream-docs.yml runs weekly, opens a tracking issue when a document changes (review the registers) or goes missing (link rotted), and commits the updated baseline. - Notifies once per event: a change updates the baseline so it cannot re-fire; an outage alerts only on crossing a 3-run failure threshold; the workflow skips a title that already has an open issue. - Shared URLs are fetched once; detection is a raw-byte hash (a prompt to look, not proof registers moved) - the issue says so. url_stability field: - new optional upstream_docs field: committed | stable | volatile | unknown - records whether the manufacturer keeps the URL put; weights how loudly a broken link is reported. - both NIBE entries set to 'stable'; validated in validate_manifest.py. Baseline seeded for the two NIBE entries. Pure diff + rendering logic is covered by tests/test_upstream_docs_watch.py; make watch-upstream-docs runs a local dry-run. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com> --- .github/workflows/watch-upstream-docs.yml | 80 +++++ CHANGELOG.md | 3 +- Makefile | 8 +- manifests/myuplink.yaml | 1 + manifests/nibe_local.yaml | 1 + spec/manifest-v2.md | 20 +- tests/test_upstream_docs.py | 20 +- tests/test_upstream_docs_watch.py | 145 ++++++++ tools/check_upstream_docs.py | 384 ++++++++++++++++++++++ tools/validate_manifest.py | 14 +- upstream-docs-state.json | 31 ++ 11 files changed, 701 insertions(+), 6 deletions(-) create mode 100644 .github/workflows/watch-upstream-docs.yml create mode 100644 tests/test_upstream_docs_watch.py create mode 100644 tools/check_upstream_docs.py create mode 100644 upstream-docs-state.json diff --git a/.github/workflows/watch-upstream-docs.yml b/.github/workflows/watch-upstream-docs.yml new file mode 100644 index 0000000..1c183ea --- /dev/null +++ b/.github/workflows/watch-upstream-docs.yml @@ -0,0 +1,80 @@ +name: watch-upstream-docs + +# Weekly, fetch every manifest `upstream_docs` URL, diff it against the committed +# baseline (upstream-docs-state.json), and open a tracking issue when a watched +# document changes or goes missing. See tools/check_upstream_docs.py. + +on: + schedule: + - cron: "17 6 * * 1" # Mondays 06:17 UTC + workflow_dispatch: + +permissions: + contents: write # commit the updated baseline + issues: write # open tracking issues + +concurrency: + group: watch-upstream-docs + cancel-in-progress: false + +jobs: + watch: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 + - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 + with: + python-version: "3.12" + - run: uv sync --frozen + + - name: Fetch and diff watched documents + run: | + uv run --no-sync python tools/check_upstream_docs.py \ + --changes-out "$RUNNER_TEMP/upstream-doc-changes.json" + + - name: Open a tracking issue for each change + env: + GH_TOKEN: ${{ github.token }} + CHANGES: ${{ runner.temp }}/upstream-doc-changes.json + run: | + set -euo pipefail + count="$(jq 'length' "$CHANGES")" + echo "alerts: $count" + if [ "$count" -eq 0 ]; then exit 0; fi + + # Ensure the labels exist (no-op if they already do). + gh label create upstream-doc-changed --color d93f0b \ + --description "A watched upstream document changed" 2>/dev/null || true + gh label create upstream-doc-unreachable --color b60205 \ + --description "A watched upstream document could not be fetched" 2>/dev/null || true + + for i in $(seq 0 $((count - 1))); do + title="$(jq -r ".[$i].issue_title" "$CHANGES")" + label="$(jq -r ".[$i].label" "$CHANGES")" + jq -r ".[$i].issue_body" "$CHANGES" > "$RUNNER_TEMP/body.md" + + # Notify once: if an open issue with this exact title already + # exists, this change (or this outage) has already been reported. + existing="$(gh issue list --state open --search "$title in:title" \ + --json title | jq --arg t "$title" '[.[] | select(.title == $t)] | length')" + if [ "$existing" -gt 0 ]; then + echo "already open, skipping: $title" + continue + fi + + echo "opening: $title" + gh issue create --title "$title" --body-file "$RUNNER_TEMP/body.md" --label "$label" + done + + - name: Commit the updated baseline + run: | + set -euo pipefail + if git diff --quiet -- upstream-docs-state.json; then + echo "baseline unchanged" + exit 0 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add upstream-docs-state.json + git commit -m "chore(upstream-docs): update watched-document baseline [skip ci]" + git push diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c7c25c..94cf61e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -124,7 +124,8 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html ### Added - **`blueprint/BLUEPRINT.lua`** — a complete, working driver for an imaginary inverter, written for people and agents alike. Every rule this repository enforces appears beside the code that follows it: bounded probing, never fabricating a zero, decoding on 16-bit halves, negating at the sign boundary. `tests/test_blueprint.py` holds it to every rule a shipped driver must meet and runs it against the harness, including the test that matters most — that failed reads per poll settle to zero, because a driver that never gives up takes the site offline - **`docs/WRITING-A-DRIVER.md`** — the reasoning behind each rule, with the blueprint as the specification -- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and a known kind; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` +- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` +- **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py` ### Removed - `tools/canonical_debt.py` measured distance from an ideal that never existed. In its place `tools/host_api_check.py` asks the only question that predicts a crash: does a driver call a function no host provides? That is what 35 drivers were doing while passing every test here diff --git a/Makefile b/Makefile index 0325b99..30e1890 100644 --- a/Makefile +++ b/Makefile @@ -9,7 +9,7 @@ LEVEL ?= patch .PHONY: bootstrap new-driver test-driver package-driver check boundary \ refused-write-report absent-register-report \ sync-manifests bump-driver history ftw-baseline ftw-baseline-report \ - host-api site + host-api site watch-upstream-docs # Build the public driver catalog page into site/, exactly as GitHub Pages # publishes it. Open site/index.html to review a change before it ships. @@ -67,6 +67,12 @@ boundary: sync-manifests: uv run --frozen --extra package --extra dev python tools/sync_manifests.py +# Fetch every manifest upstream_docs URL and report changes against the +# baseline WITHOUT rewriting it. The scheduled watch-upstream-docs workflow +# runs the same tool for real and opens a tracking issue on a change. +watch-upstream-docs: + uv run --frozen --extra package --extra dev python tools/check_upstream_docs.py --dry-run + # Raise a driver version in the manifest and the DRIVER table together. # Example: make bump-driver ID=sungrow LEVEL=patch bump-driver: diff --git a/manifests/myuplink.yaml b/manifests/myuplink.yaml index b0b844a..00c8cd9 100644 --- a/manifests/myuplink.yaml +++ b/manifests/myuplink.yaml @@ -17,6 +17,7 @@ upstream_docs: - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" title: "NIBE myUplink register/parameter changelog (nibe-n.pdf)" kind: changelog + url_stability: stable min_host_version: "2.0.0" size_bytes: 15278 dkb_id: "myuplink" diff --git a/manifests/nibe_local.yaml b/manifests/nibe_local.yaml index 329e357..b20df41 100644 --- a/manifests/nibe_local.yaml +++ b/manifests/nibe_local.yaml @@ -17,6 +17,7 @@ upstream_docs: - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" title: "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)" kind: changelog + url_stability: stable min_host_version: "2.0.0" size_bytes: 18227 dkb_id: "nibe_local" diff --git a/spec/manifest-v2.md b/spec/manifest-v2.md index cbf114d..9167013 100644 --- a/spec/manifest-v2.md +++ b/spec/manifest-v2.md @@ -55,16 +55,32 @@ review when the source moves, instead of a human noticing months later. | `url` | string | Yes | Semi-persistent `http(s)` URL of the document to watch | | `title` | string | No | Human-readable label (e.g., `"NIBE S-series register changelog"`) | | `kind` | string | No | One of: `changelog`, `register_map`, `manual`, `api_docs`, `firmware_notes`, `other` | +| `url_stability` | string | No | How durable the URL is: `committed`, `stable`, `volatile`, `unknown` (default) | + +`url_stability` records whether the URL can be trusted to stay put: + +- `committed` — the manufacturer documents or promises the URL is permanent; +- `stable` — stable in practice, with no explicit promise; +- `volatile` — known to rotate (dated or versioned links); +- `unknown` — not assessed (also the default when the field is absent). + +The watcher uses it to weigh a broken link: a `committed` or `stable` URL going +missing is a real signal that the document moved and the driver may be following +a source that no longer exists, whereas a `volatile` one breaking can be routine. A driver may list multiple `upstream_docs` entries — for example a register map and a separate changelog. The field is descriptive metadata only: it is not copied into `index.yaml` and never affects how a driver is installed or run. +It is consumed by `tools/check_upstream_docs.py` and the `watch-upstream-docs` +workflow, which fetch each URL, diff it against `upstream-docs-state.json`, and +open a tracking issue when a watched document changes or disappears. ```yaml upstream_docs: - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" title: "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)" kind: changelog + url_stability: stable ``` ## Example @@ -115,7 +131,7 @@ Version changes must be accompanied by a `CHANGELOG.md` entry under `[Unreleased 7. `core` tier drivers must have an `author` 8. Every manifest must have a corresponding `.lua` file in `drivers/` 9. `tested_devices` entries must have `manufacturer` and `model_family` -10. `upstream_docs` entries must have an `http(s)` `url`; `kind`, when present, must be a known kind +10. `upstream_docs` entries must have an `http(s)` `url`; `kind` and `url_stability`, when present, must be known values ## Migration from V1 @@ -127,6 +143,6 @@ V2.1 extends `tested_devices` with: `model_family` (replaces `model`), `variants V2.2 adds bytecode fields: `bytecode_sha256`, `bytecode_signature`, `bytecode_size` for Lua 5.5.0 compiled bytecode. -V2.3 adds `upstream_docs`: an optional list of vendor reference documents (`url`, `title`, `kind`) to watch for register/protocol changes. +V2.3 adds `upstream_docs`: an optional list of vendor reference documents (`url`, `title`, `kind`, `url_stability`) to watch for register/protocol changes. Use `tools/migrate_manifests.py` to convert V1 JSON → V2 YAML. diff --git a/tests/test_upstream_docs.py b/tests/test_upstream_docs.py index 22fc3ed..f51b353 100644 --- a/tests/test_upstream_docs.py +++ b/tests/test_upstream_docs.py @@ -116,6 +116,23 @@ def test_unknown_field_is_rejected(tmp_path): assert any("upstream_docs[0]: unknown field 'foo'" in e for e in errors) +def test_unknown_url_stability_is_rejected(tmp_path): + errors = _errors_for( + tmp_path, + 'upstream_docs:\n - url: "https://x/a.pdf"\n url_stability: rock_solid\n', + ) + assert any("upstream_docs[0]: url_stability 'rock_solid' is not valid" in e + for e in errors) + + +def test_known_url_stability_is_accepted(tmp_path): + errors = _errors_for( + tmp_path, + 'upstream_docs:\n - url: "https://x/a.pdf"\n url_stability: committed\n', + ) + assert not [e for e in errors if e.startswith("upstream_docs")] + + def test_wellformed_entry_raises_no_upstream_error(tmp_path): errors = _errors_for( tmp_path, @@ -141,4 +158,5 @@ def test_every_declared_upstream_doc_is_wellformed(): for i, doc in enumerate(parse_upstream_docs(path.read_text(encoding="utf-8"))): url = doc.get("url", "") assert url.startswith(("http://", "https://")), f"{path.name}[{i}]: bad url" - assert set(doc) <= {"url", "title", "kind"}, f"{path.name}[{i}]: unknown field" + allowed = {"url", "title", "kind", "url_stability"} + assert set(doc) <= allowed, f"{path.name}[{i}]: unknown field" diff --git a/tests/test_upstream_docs_watch.py b/tests/test_upstream_docs_watch.py new file mode 100644 index 0000000..bf533d7 --- /dev/null +++ b/tests/test_upstream_docs_watch.py @@ -0,0 +1,145 @@ +"""The watcher must notice a changed document, and pester a maintainer once. + +check_upstream_docs.py holds the folding of fresh fetches into a committed +baseline: a first sighting is a baseline (not news), a moved hash is one alert, +and a link that stays down is one alert on the run it crosses the failure +threshold — never every run after. These tests pin that behaviour without a +network, which is where the once-only guarantee actually lives. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tools")) + +from check_upstream_docs import ( # noqa: E402 + compute_updates, + load_state, + write_state, +) + +KEY = "nibe_local::https://x/a.pdf" +META = { + "driver": "nibe_local", + "url": "https://x/a.pdf", + "title": "NIBE changelog", + "kind": "changelog", + "url_stability": "stable", +} +CURRENT = {KEY: META} +SHA_A = "a" * 64 +SHA_B = "b" * 64 + + +def ok(sha: str) -> dict: + return {KEY: {"ok": True, "sha256": sha, "content_length": 100}} + + +def fail(error: str = "boom") -> dict: + return {KEY: {"ok": False, "error": error}} + + +# ---- Baseline / change -------------------------------------------------- + + +def test_first_sighting_records_baseline_without_alert(): + docs, alerts = compute_updates({}, CURRENT, ok(SHA_A), "t0") + assert alerts == [] + entry = docs[KEY] + assert entry["sha256"] == SHA_A + assert entry["first_seen"] == "t0" and entry["last_changed"] == "t0" + assert entry["consecutive_failures"] == 0 + assert entry["url_stability"] == "stable" + + +def test_unchanged_document_raises_no_alert_and_keeps_timestamps(): + base, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + docs, alerts = compute_updates(base, CURRENT, ok(SHA_A), "t1") + assert alerts == [] + assert docs[KEY]["first_seen"] == "t0" + assert docs[KEY]["last_changed"] == "t0" # unchanged, so not bumped to t1 + + +def test_changed_document_alerts_exactly_once(): + base, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + + docs, alerts = compute_updates(base, CURRENT, ok(SHA_B), "t1") + assert len(alerts) == 1 + alert = alerts[0] + assert alert["type"] == "changed" + assert alert["old_sha"] == SHA_A and alert["new_sha"] == SHA_B + assert docs[KEY]["last_changed"] == "t1" + assert docs[KEY]["first_seen"] == "t0" + + # Re-running against the now-updated baseline must not re-alert. + _, alerts_again = compute_updates(docs, CURRENT, ok(SHA_B), "t2") + assert alerts_again == [] + + +def test_document_dropped_from_manifests_is_pruned(): + base, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + docs, alerts = compute_updates(base, {}, {}, "t1") + assert docs == {} and alerts == [] + + +# ---- Broken link: once --------------------------------------------------- + + +def test_broken_link_alerts_once_on_crossing_the_threshold(): + state, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + alert_runs = [] + for run in range(1, 6): + state, alerts = compute_updates( + state, CURRENT, fail(), f"t{run}", failure_threshold=3 + ) + if alerts: + alert_runs.append((run, alerts[0]["type"])) + + assert alert_runs == [(3, "unreachable")] # exactly one, on the 3rd failure + assert state[KEY]["consecutive_failures"] == 5 + assert state[KEY]["sha256"] == SHA_A # last good hash is retained + + +def test_recovery_resets_the_failure_count(): + state, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + state, _ = compute_updates(state, CURRENT, fail(), "t1", failure_threshold=3) + state, alerts = compute_updates(state, CURRENT, ok(SHA_A), "t2") + assert alerts == [] + assert state[KEY]["consecutive_failures"] == 0 + assert state[KEY]["last_error"] is None + + +# ---- Issue rendering ---------------------------------------------------- + + +def test_changed_alert_title_and_body_carry_the_hashes(): + base, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + _, alerts = compute_updates(base, CURRENT, ok(SHA_B), "t1") + alert = alerts[0] + assert alert["issue_title"] == "[upstream-doc] nibe_local: NIBE changelog changed (bbbbbbbb)" + assert SHA_A in alert["issue_body"] and SHA_B in alert["issue_body"] + assert alert["label"] == "upstream-doc-changed" + + +def test_unreachable_alert_weighs_the_stability(): + state, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + for run in range(1, 4): + state, alerts = compute_updates(state, CURRENT, fail(), f"t{run}", failure_threshold=3) + body = alerts[0]["issue_body"] + assert "URL stability:** stable" in body + assert alerts[0]["issue_title"] == "[upstream-doc] nibe_local: NIBE changelog unreachable" + assert alerts[0]["label"] == "upstream-doc-unreachable" + + +# ---- State file --------------------------------------------------------- + + +def test_state_file_round_trips(tmp_path): + docs, _ = compute_updates({}, CURRENT, ok(SHA_A), "t0") + path = tmp_path / "upstream-docs-state.json" + write_state(path, docs) + assert load_state(path) == docs + assert load_state(tmp_path / "missing.json") == {} diff --git a/tools/check_upstream_docs.py b/tools/check_upstream_docs.py new file mode 100644 index 0000000..499df1f --- /dev/null +++ b/tools/check_upstream_docs.py @@ -0,0 +1,384 @@ +#!/usr/bin/env python3 +"""Watch the upstream_docs a driver was built against and flag changes. + +A manifest may list the vendor documents a driver decodes a device by — a +register map, a parameter changelog PDF (see spec/manifest-v2.md, `upstream_docs`). +Those documents change without telling us. This tool fetches each one, hashes it, +compares it against a committed baseline in `upstream-docs-state.json`, and +reports which drivers now sit on a changed source so a maintainer can review the +registers. + +It is the automation half of the field: the manifest records WHAT to watch, this +records WHAT IT WAS and notices when that moves. + + python tools/check_upstream_docs.py # fetch, update baseline, report + python tools/check_upstream_docs.py --dry-run # fetch + report, write nothing + python tools/check_upstream_docs.py --check # validate baseline only, no network + python tools/check_upstream_docs.py --changes-out P.json # write alerts for the workflow + +Change detection is a raw-byte SHA-256. A PDF's bytes can churn without its +content changing (rebuild timestamps in the file metadata), so a flagged change +is a prompt to look, not proof the registers moved — the tracking issue says so. +The tool only detects and records; the workflow opens the tracking issue. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +import urllib.error +import urllib.request +from datetime import datetime, timezone +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +from manifest_parser import parse_upstream_docs, parse_yaml_simple + +ROOT = Path(__file__).resolve().parents[1] +MANIFEST_DIR = ROOT / "manifests" +STATE_PATH = ROOT / "upstream-docs-state.json" + +SCHEMA_VERSION = "sourceful.upstream-docs-state/v1" +# Vendor URLs rot and rate-limit; a single failed fetch is noise, not a story. +# Surface a document as unreachable only after it has failed this many runs. +FAILURE_ALERT_THRESHOLD = 3 +CHANGED_LABEL = "upstream-doc-changed" +UNREACHABLE_LABEL = "upstream-doc-unreachable" +USER_AGENT = ("srcfl-device-drivers upstream-docs watcher " + "(+https://github.com/srcfl/device-drivers)") +FETCH_TIMEOUT_S = 30 + + +# ---- Manifest collection ------------------------------------------------ + + +def doc_key(driver: str, url: str) -> str: + """Stable identity for one watched document.""" + return f"{driver}::{url}" + + +def collect_docs(manifests_dir: Path) -> dict[str, dict]: + """Every upstream_docs entry across all manifests, keyed by doc_key.""" + docs: dict[str, dict] = {} + for path in sorted(manifests_dir.glob("*.yaml")): + text = path.read_text(encoding="utf-8") + driver = parse_yaml_simple(text).get("name", path.stem) + for entry in parse_upstream_docs(text): + url = entry.get("url", "") + if not url: + continue + docs[doc_key(driver, url)] = { + "driver": driver, + "url": url, + "title": entry.get("title", ""), + "kind": entry.get("kind", ""), + "url_stability": entry.get("url_stability", ""), + } + return docs + + +# ---- Fetching (the only network in this file) --------------------------- + + +def fetch_doc(url: str, timeout: int = FETCH_TIMEOUT_S) -> dict: + """Fetch a document. Returns an observation dict. + + {"ok": True, "sha256": ..., "content_length": ...} on success, + {"ok": False, "error": "..."} on any failure. Never raises. + """ + request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + body = response.read() + except (urllib.error.URLError, TimeoutError, ValueError, OSError) as exc: + return {"ok": False, "error": f"{type(exc).__name__}: {exc}"} + return { + "ok": True, + "sha256": hashlib.sha256(body).hexdigest(), + "content_length": len(body), + } + + +# ---- Diff (pure — this is what the tests pin) --------------------------- + + +def compute_updates( + prev_docs: dict[str, dict], + current: dict[str, dict], + observations: dict[str, dict], + now: str, + failure_threshold: int = FAILURE_ALERT_THRESHOLD, +) -> tuple[dict[str, dict], list[dict]]: + """Fold fresh observations into the baseline. Pure; no I/O. + + Returns (new baseline docs, alerts). An alert is raised when a document's + bytes change, or when it has been unreachable for `failure_threshold` runs + in a row (once, on crossing the threshold). A document seen for the first + time only records a baseline — a new watch is not a change. Documents no + longer named by any manifest are dropped from the baseline. + """ + new_docs: dict[str, dict] = {} + alerts: list[dict] = [] + + for key, meta in current.items(): + prev = prev_docs.get(key) + obs = observations.get(key, {"ok": False, "error": "not fetched"}) + entry = { + "driver": meta["driver"], + "url": meta["url"], + "title": meta.get("title", ""), + "kind": meta.get("kind", ""), + "url_stability": meta.get("url_stability", ""), + } + + if obs.get("ok"): + if prev is None: + entry.update( + sha256=obs["sha256"], + content_length=obs["content_length"], + first_seen=now, + last_changed=now, + consecutive_failures=0, + last_error=None, + ) + elif prev.get("sha256") != obs["sha256"]: + entry.update( + sha256=obs["sha256"], + content_length=obs["content_length"], + first_seen=prev.get("first_seen", now), + last_changed=now, + consecutive_failures=0, + last_error=None, + ) + alerts.append(_changed_alert(entry, prev.get("sha256"), now)) + else: + entry.update( + sha256=obs["sha256"], + content_length=obs["content_length"], + first_seen=prev.get("first_seen", now), + last_changed=prev.get("last_changed", now), + consecutive_failures=0, + last_error=None, + ) + else: # fetch failed + failures = (prev.get("consecutive_failures", 0) if prev else 0) + 1 + entry.update( + sha256=prev.get("sha256") if prev else None, + content_length=prev.get("content_length") if prev else None, + first_seen=prev.get("first_seen", now) if prev else now, + last_changed=prev.get("last_changed") if prev else None, + consecutive_failures=failures, + last_error=obs.get("error", "unknown error"), + ) + if failures == failure_threshold: + alerts.append(_unreachable_alert(entry, failures)) + + new_docs[key] = entry + + return new_docs, alerts + + +# ---- Alert / issue rendering (pure) ------------------------------------- + + +def _doc_label(entry: dict) -> str: + return entry.get("title") or entry["url"].rsplit("/", 1)[-1] or entry["url"] + + +def _changed_alert(entry: dict, old_sha: str | None, now: str) -> dict: + driver = entry["driver"] + label = _doc_label(entry) + # The short hash in the title makes each distinct change its own thread and + # lets the workflow skip a re-run of the SAME change: notify exactly once. + title = f"[upstream-doc] {driver}: {label} changed ({entry['sha256'][:8]})" + body = "\n".join([ + f"The upstream document **{driver}** is watched against has changed.", + "", + f"- **Driver:** `{driver}` — `drivers/lua/{driver}.lua`, `manifests/{driver}.yaml`", + f"- **Document:** {label}", + f"- **URL:** {entry['url']}", + f"- **Kind:** {entry.get('kind') or 'unspecified'}", + f"- **URL stability:** {entry.get('url_stability') or 'unknown'}", + f"- **Previous SHA-256:** `{old_sha or 'none'}`", + f"- **New SHA-256:** `{entry['sha256']}`", + f"- **Detected:** {now}", + "", + "### Review", + "", + "- [ ] Read the new document and note what changed (registers renumbered,", + " parameters added, scaling corrected).", + f"- [ ] Cross-check against the registers/parameters `{driver}` reads.", + "- [ ] If the driver is affected, open a fix and bump its version;", + " otherwise close this issue.", + "", + "> Detection is a raw-byte hash: the file's bytes moved. That can be a", + "> genuine content change or just rebuild churn in the PDF metadata —", + "> this issue is a prompt to look, not proof the registers changed.", + "", + f"", + ]) + return { + "type": "changed", + "key": doc_key(driver, entry["url"]), + "driver": driver, + "url": entry["url"], + "old_sha": old_sha, + "new_sha": entry["sha256"], + "label": CHANGED_LABEL, + "issue_title": title, + "issue_body": body, + } + + +def _unreachable_alert(entry: dict, failures: int) -> dict: + driver = entry["driver"] + label = _doc_label(entry) + stability = entry.get("url_stability") or "unknown" + # How much a broken link should worry a maintainer depends on whether the + # manufacturer promised the URL would stay put (the url_stability field). + weight = { + "committed": "The manufacturer commits to this URL, so a break is " + "unexpected — the document has very likely moved.", + "stable": "This URL was recorded as stable, so a break most likely " + "means the document moved.", + "volatile": "This URL was flagged as volatile, so an occasional break " + "may be routine — confirm before acting.", + }.get(stability, "This URL's stability was not assessed.") + # No short hash here: while the link stays down there is one outage to tell + # the maintainers about, and the workflow skips it if that issue is open. + title = f"[upstream-doc] {driver}: {label} unreachable" + body = "\n".join([ + f"The upstream document **{driver}** is watched against could not be", + f"fetched for {failures} runs in a row. It may have moved or been removed —", + "which is itself worth checking, since the driver would then be following", + "a source that no longer exists.", + "", + f"- **Driver:** `{driver}`", + f"- **Document:** {label}", + f"- **URL:** {entry['url']}", + f"- **URL stability:** {stability} — {weight}", + f"- **Last error:** `{entry.get('last_error')}`", + "", + "- [ ] Confirm whether the document moved to a new URL and update", + f" `manifests/{driver}.yaml`, or remove the entry if it is gone.", + "", + f"", + ]) + return { + "type": "unreachable", + "key": doc_key(driver, entry["url"]), + "driver": driver, + "url": entry["url"], + "failures": failures, + "label": UNREACHABLE_LABEL, + "issue_title": title, + "issue_body": body, + } + + +# ---- State file --------------------------------------------------------- + + +def load_state(path: Path) -> dict[str, dict]: + if not path.exists(): + return {} + data = json.loads(path.read_text(encoding="utf-8")) + return data.get("docs", {}) + + +def write_state(path: Path, docs: dict[str, dict]) -> None: + payload = { + "schema_version": SCHEMA_VERSION, + "docs": {key: docs[key] for key in sorted(docs)}, + } + path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + + +# ---- CLI ---------------------------------------------------------------- + + +def _now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat() + + +def _validate_state(state_path: Path, manifests_dir: Path) -> int: + """No-network sanity check: baseline parses and names only real docs.""" + docs = load_state(state_path) + current = collect_docs(manifests_dir) + errors = [] + for key, entry in docs.items(): + if key not in current: + errors.append(f"{key}: baseline names a doc no manifest declares") + for field in ("driver", "url"): + if not entry.get(field): + errors.append(f"{key}: missing '{field}'") + for message in errors: + print(message) + print(f"{len(docs)} watched docs, {len(errors)} problems.") + return 1 if errors else 0 + + +def _summarize(alerts: list[dict], docs: dict, checked: int, failed: int) -> str: + lines = [ + "## upstream-docs watch", + "", + f"- Watched documents: **{len(docs)}**", + f"- Fetched this run: **{checked}** ({failed} failed)", + f"- Alerts: **{len(alerts)}**", + ] + for alert in alerts: + lines.append(f" - `{alert['type']}` — {alert['driver']}: {alert['url']}") + return "\n".join(lines) + "\n" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--dry-run", action="store_true", + help="fetch and report, but do not write the baseline") + parser.add_argument("--check", action="store_true", + help="validate the baseline file only; no network") + parser.add_argument("--changes-out", type=Path, + help="write the alert list as JSON to this path") + parser.add_argument("--state", type=Path, default=STATE_PATH) + parser.add_argument("--manifests", type=Path, default=MANIFEST_DIR) + args = parser.parse_args() + + if args.check: + return _validate_state(args.state, args.manifests) + + current = collect_docs(args.manifests) + # Several drivers can watch the same document (nibe_local and myuplink share + # one PDF); fetch each distinct URL once, not once per driver. + fetched = {url: fetch_doc(url) + for url in sorted({meta["url"] for meta in current.values()})} + observations = {key: fetched[meta["url"]] for key, meta in current.items()} + failed = sum(1 for obs in fetched.values() if not obs.get("ok")) + + prev_docs = load_state(args.state) + new_docs, alerts = compute_updates(prev_docs, current, observations, _now_iso()) + + if args.changes_out: + args.changes_out.write_text(json.dumps(alerts, indent=2) + "\n", encoding="utf-8") + + summary = _summarize(alerts, new_docs, len(fetched), failed) + print(summary) + _append_step_summary(summary) + + if not args.dry_run: + write_state(args.state, new_docs) + + return 0 + + +def _append_step_summary(summary: str) -> None: + import os + path = os.environ.get("GITHUB_STEP_SUMMARY") + if path: + with open(path, "a", encoding="utf-8") as handle: + handle.write(summary) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/validate_manifest.py b/tools/validate_manifest.py index 250e96a..73ddfbf 100644 --- a/tools/validate_manifest.py +++ b/tools/validate_manifest.py @@ -30,9 +30,16 @@ # Upstream reference documents a driver was built against (register maps, # parameter changelogs, manuals). Their URLs are semi-persistent so a watcher # can poll them and flag a driver for review when the source moves. -VALID_UPSTREAM_DOC_FIELDS = {"url", "title", "kind"} +VALID_UPSTREAM_DOC_FIELDS = {"url", "title", "kind", "url_stability"} VALID_UPSTREAM_DOC_KINDS = {"changelog", "register_map", "manual", "api_docs", "firmware_notes", "other"} +# How durable the URL is — does the manufacturer keep it put? It weighs how +# loudly the watcher should complain when the link breaks. +# committed = manufacturer documents/promises the URL is permanent +# stable = stable in practice, no explicit promise +# volatile = known to rotate (dated or versioned links) +# unknown = not assessed (also the default when the field is absent) +VALID_URL_STABILITY = {"committed", "stable", "volatile", "unknown"} def validate_manifest(yaml_path: Path, drivers_dir: Path) -> list[str]: @@ -140,6 +147,11 @@ def validate_manifest(yaml_path: Path, drivers_dir: Path) -> list[str]: errors.append(f"{prefix}: kind '{kind}' is not valid " f"(expected: {', '.join(sorted(VALID_UPSTREAM_DOC_KINDS))})") + stability = doc.get("url_stability", "") + if stability and stability not in VALID_URL_STABILITY: + errors.append(f"{prefix}: url_stability '{stability}' is not valid " + f"(expected: {', '.join(sorted(VALID_URL_STABILITY))})") + for key in doc: if key not in VALID_UPSTREAM_DOC_FIELDS: errors.append(f"{prefix}: unknown field '{key}'") diff --git a/upstream-docs-state.json b/upstream-docs-state.json new file mode 100644 index 0000000..e716d43 --- /dev/null +++ b/upstream-docs-state.json @@ -0,0 +1,31 @@ +{ + "schema_version": "sourceful.upstream-docs-state/v1", + "docs": { + "myuplink::https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf": { + "driver": "myuplink", + "url": "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf", + "title": "NIBE myUplink register/parameter changelog (nibe-n.pdf)", + "kind": "changelog", + "url_stability": "stable", + "sha256": "fa1186981a020f6c262cf35abd50fa447343cfaa16a4eedb018cb2525c00f221", + "content_length": 850716, + "first_seen": "2026-07-29T15:08:58+00:00", + "last_changed": "2026-07-29T15:08:58+00:00", + "consecutive_failures": 0, + "last_error": null + }, + "nibe_local::https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf": { + "driver": "nibe_local", + "url": "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf", + "title": "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)", + "kind": "changelog", + "url_stability": "stable", + "sha256": "fa1186981a020f6c262cf35abd50fa447343cfaa16a4eedb018cb2525c00f221", + "content_length": 850716, + "first_seen": "2026-07-29T15:08:58+00:00", + "last_changed": "2026-07-29T15:08:58+00:00", + "consecutive_failures": 0, + "last_error": null + } + } +} From 591aa602967b3c5201ee3366354656c8ecb1f7fa Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Fri, 31 Jul 2026 09:17:20 +0200 Subject: [PATCH 3/5] docs: move the upstream_docs entries to the current batch Merging main carried both entries along inside the Added group they were written in, next to BLUEPRINT.lua, which now reads as though the field shipped in that batch. [Unreleased] prepends a group per merged change, so they belong at the top - the same move 6f8eab7 made for the catalog. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com> --- CHANGELOG.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94cf61e..5986e80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,10 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html - **`nibe_local` was filed as a meter.** Its manifest declared `ders: [meter]`; it reads a NIBE S-series heat pump. The driver emits nothing but metrics — `hp_power_w`, `hp_used_power_w`, the temperatures and the lifetime energy counters, all through `host.emit_metric` and never through `host.emit("meter", …)` — which its header states is deliberate, so that a pump's electrical draw cannot double-count against the site's real grid meter. The catalog was advertising a DER the driver does not provide, and the published page (#51) badges and filters on exactly that field: the pump sat under **Meter** and was absent from **Heat pump**, where an owner looking for it would start. Now `ders: [heatpump]`, matching `heishamon` — the other metric-only heat pump, which has been filed that way since it landed. The Lua source and its `sha256` are untouched, so the driver a host runs is byte-for-byte the one it ran before - **`nibe_local`** 1.1.0 → 1.1.1 — **the version has to move anyway, because the signed artifact carries this metadata.** `ders` is not catalog-only decoration: `_load_channel` copies it into the artifact's `capabilities` and the channel publishes source and metadata as one signed file, so correcting the field changed the published bytes. The channel then refused the release — `error: nibe_local: changed artifact needs a higher version than 1.1.0` — and it was right to, since that immutability is what lets an operator name the bytes a site is running. A patch rather than a minor: no register, field or emitted metric changed, only the DER the catalog states. The `DRIVER` table is deliberately left alone, keeping the driver byte-identical to its FTW baseline the way #29 established +### Added +- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` +- **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py` + ### Added - **A published driver catalog at [srcfl.github.io/device-drivers](https://srcfl.github.io/device-drivers/)** — all 80 drivers, searchable by manufacturer or model number and filterable by device type, protocol, tier and control. `tools/generate_site.py` builds it from the manifests and the Lua sources on every push to `main`, so no copy of this data is checked in and the page cannot fall behind what a host installs. It reports hardware evidence the way the repository does rather than the way a catalog usually does: 5 drivers state they have been confirmed against physical hardware and 28 state that they have not, and `tests/test_site.py` fails the build if a driver's `verification_status` is ever rendered as a stronger claim than the driver makes @@ -124,8 +128,6 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html ### Added - **`blueprint/BLUEPRINT.lua`** — a complete, working driver for an imaginary inverter, written for people and agents alike. Every rule this repository enforces appears beside the code that follows it: bounded probing, never fabricating a zero, decoding on 16-bit halves, negating at the sign boundary. `tests/test_blueprint.py` holds it to every rule a shipped driver must meet and runs it against the harness, including the test that matters most — that failed reads per poll settle to zero, because a driver that never gives up takes the site offline - **`docs/WRITING-A-DRIVER.md`** — the reasoning behind each rule, with the blueprint as the specification -- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` -- **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py` ### Removed - `tools/canonical_debt.py` measured distance from an ideal that never existed. In its place `tools/host_api_check.py` asks the only question that predicts a crash: does a driver call a function no host provides? That is what 35 drivers were doing while passing every test here From 665e5d31e4ac86a6ee20e18f04f364008c91286e Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Fri, 31 Jul 2026 09:56:50 +0200 Subject: [PATCH 4/5] docs: tell a driver author to record what they decoded it from upstream_docs was specified in spec/manifest-v2.md and nowhere else, which is where you read for field syntax after you have decided to use a field - not where you find out you should. Neither AGENTS.md nor WRITING-A-DRIVER.md mentioned it, so nothing in the repository ever asks an author for the one thing the watcher needs. A field nobody is told to populate stays at two manifests out of eighty, and a driver with no entry is one nobody will be warned about. WRITING-A-DRIVER.md already says the valuable part of a driver is the knowledge rather than the code, and to write down what you learn about the registers as you find it. Where that knowledge came from is the half it never asked for. The reasoning and the YAML now sit beside the manifest step that writes it; AGENTS.md carries the one-line rule with the source rules. Both state the limits rather than overselling it: a document behind a login cannot be fetched and belongs in a driver comment, and detection is a hash of the bytes, so a flagged document is a prompt to look rather than proof the registers moved. No code, manifest or generated file changes. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com> --- AGENTS.md | 7 +++++++ CHANGELOG.md | 2 +- docs/WRITING-A-DRIVER.md | 45 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index ea30eaa..355667f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,6 +57,13 @@ rule may be right and the driver wrong, or the check itself may be wrong. - PV generation, meter export and battery/vehicle discharge are negative. - Report stable hardware identity early. - Do not emit stale cached telemetry as fresh. +- Record the vendor documents a driver was decoded from — register map, + parameter changelog, API reference — in the manifest's `upstream_docs`, at + the most durable URL available. A weekly watcher opens a tracking issue when + one changes or disappears, so a driver that has fallen behind its source is + caught there rather than by a wrong value on a customer's site. A document + behind a login cannot be watched; reference it in a driver comment instead. + See `docs/WRITING-A-DRIVER.md`. - Keep Lua compatible with every runtime declared in the package recipe. - Package id, version, read-only state and target metadata must match the Lua `DRIVER` block. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5986e80..0825d0d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,7 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html - **`nibe_local`** 1.1.0 → 1.1.1 — **the version has to move anyway, because the signed artifact carries this metadata.** `ders` is not catalog-only decoration: `_load_channel` copies it into the artifact's `capabilities` and the channel publishes source and metadata as one signed file, so correcting the field changed the published bytes. The channel then refused the release — `error: nibe_local: changed artifact needs a higher version than 1.1.0` — and it was right to, since that immutability is what lets an operator name the bytes a site is running. A patch rather than a minor: no register, field or emitted metric changed, only the DER the catalog states. The `DRIVER` table is deliberately left alone, keeping the driver byte-identical to its FTW baseline the way #29 established ### Added -- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Documented in `spec/manifest-v2.md` (V2.3); tested in `tests/test_upstream_docs.py` +- **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Specified in `spec/manifest-v2.md` (V2.3), and `docs/WRITING-A-DRIVER.md` and `AGENTS.md` now tell a driver author to record it beside the older instruction to write down what they learned about the registers — where the knowledge came from is the half that was never asked for, and a field nobody is told to populate stays at two manifests out of 80. Both say what it cannot do: a document behind a login is not fetchable and belongs in a driver comment instead. Tested in `tests/test_upstream_docs.py` - **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py` ### Added diff --git a/docs/WRITING-A-DRIVER.md b/docs/WRITING-A-DRIVER.md index 2d6555e..e726dde 100644 --- a/docs/WRITING-A-DRIVER.md +++ b/docs/WRITING-A-DRIVER.md @@ -21,6 +21,11 @@ what, what scale it uses, which firmware lies, what the vendor documented wrongly. **Write that down in comments as you find it.** Nobody can re-derive it from the numbers later, and it is the reason this repository exists. +**Write down where you got it, too.** The register map, the parameter +changelog, the API reference you decoded the device from — record those in the +manifest so the driver can be checked against them again later. See +[Record what you decoded it from](#record-what-you-decoded-it-from). + ## The five entry points | Function | When | @@ -223,6 +228,46 @@ Then write `manifests/mydevice.yaml`. The manifest and the driver's own `DRIVER` table must agree on the version; `make bump-driver ID=mydevice` moves both. +### Record what you decoded it from + +A driver is only ever as current as the vendor document it was written from, +and that document does not hold still. A register renumbered, a parameter +added, a scale corrected in a new revision — none of it announces itself. The +driver goes on decoding the old map and reporting a plausible wrong number, +and it is a human noticing on a real site months later that ends it. + +So record the documents you worked from, at the most durable URL you can find: + +```yaml +upstream_docs: + - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf" + title: "NIBE S-series myUplink register/parameter changelog" + kind: changelog + url_stability: stable +``` + +Only `url` is required. `kind` is one of `changelog`, `register_map`, +`manual`, `api_docs`, `firmware_notes`, `other`. `url_stability` says how much +a broken link means — `committed` (the vendor promises the URL is permanent), +`stable` (stable in practice), `volatile` (dated or versioned links that +rotate), `unknown`. `spec/manifest-v2.md` has the full field reference. + +`.github/workflows/watch-upstream-docs.yml` fetches each URL weekly and opens +a tracking issue when a document changes or disappears, so the driver gets +re-read against the new material rather than quietly falling behind. It is +descriptive metadata only: it never reaches `index.yaml` and changes nothing +about how the driver installs or runs. `make watch-upstream-docs` runs the +check locally without touching the baseline. + +Two things it cannot do. A document behind a login or a vendor portal is not +fetchable — put that reference in a comment in the driver instead, where it +still tells the next reader where to look. And detection is a hash of the +bytes, so a rebuilt PDF can flag a change that isn't one; a flagged document +is a prompt to look, not proof the registers moved. + +Only drivers that declare the field are watched. A driver with no entry is one +nobody will be warned about. + ```bash make test-driver ID=mydevice make check From ede9210281c8deb81a381b78f9424116b78d8117 Mon Sep 17 00:00:00 2001 From: Claude Opus 5 Date: Fri, 31 Jul 2026 11:21:13 +0200 Subject: [PATCH 5/5] docs: keep the upstream_docs entries at the top after the rebase Rebasing onto main put main's own group above this batch's, so [Unreleased] read as though upstream_docs shipped before the nibe_local fix that main merged after it. A group per merged change is prepended, and this change merges next, so its group leads again -- the same move 892a599 made when main was merged in, and 6f8eab7 before it for the catalog. Ordering only: no entry text changed. Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com> Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> --- CHANGELOG.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0825d0d..4588543 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,14 +7,14 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html ## [Unreleased] -### Fixed -- **`nibe_local` was filed as a meter.** Its manifest declared `ders: [meter]`; it reads a NIBE S-series heat pump. The driver emits nothing but metrics — `hp_power_w`, `hp_used_power_w`, the temperatures and the lifetime energy counters, all through `host.emit_metric` and never through `host.emit("meter", …)` — which its header states is deliberate, so that a pump's electrical draw cannot double-count against the site's real grid meter. The catalog was advertising a DER the driver does not provide, and the published page (#51) badges and filters on exactly that field: the pump sat under **Meter** and was absent from **Heat pump**, where an owner looking for it would start. Now `ders: [heatpump]`, matching `heishamon` — the other metric-only heat pump, which has been filed that way since it landed. The Lua source and its `sha256` are untouched, so the driver a host runs is byte-for-byte the one it ran before -- **`nibe_local`** 1.1.0 → 1.1.1 — **the version has to move anyway, because the signed artifact carries this metadata.** `ders` is not catalog-only decoration: `_load_channel` copies it into the artifact's `capabilities` and the channel publishes source and metadata as one signed file, so correcting the field changed the published bytes. The channel then refused the release — `error: nibe_local: changed artifact needs a higher version than 1.1.0` — and it was right to, since that immutability is what lets an operator name the bytes a site is running. A patch rather than a minor: no register, field or emitted metric changed, only the DER the catalog states. The `DRIVER` table is deliberately left alone, keeping the driver byte-identical to its FTW baseline the way #29 established - ### Added - **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Specified in `spec/manifest-v2.md` (V2.3), and `docs/WRITING-A-DRIVER.md` and `AGENTS.md` now tell a driver author to record it beside the older instruction to write down what they learned about the registers — where the knowledge came from is the half that was never asked for, and a field nobody is told to populate stays at two manifests out of 80. Both say what it cannot do: a document behind a login is not fetchable and belongs in a driver comment instead. Tested in `tests/test_upstream_docs.py` - **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py` +### Fixed +- **`nibe_local` was filed as a meter.** Its manifest declared `ders: [meter]`; it reads a NIBE S-series heat pump. The driver emits nothing but metrics — `hp_power_w`, `hp_used_power_w`, the temperatures and the lifetime energy counters, all through `host.emit_metric` and never through `host.emit("meter", …)` — which its header states is deliberate, so that a pump's electrical draw cannot double-count against the site's real grid meter. The catalog was advertising a DER the driver does not provide, and the published page (#51) badges and filters on exactly that field: the pump sat under **Meter** and was absent from **Heat pump**, where an owner looking for it would start. Now `ders: [heatpump]`, matching `heishamon` — the other metric-only heat pump, which has been filed that way since it landed. The Lua source and its `sha256` are untouched, so the driver a host runs is byte-for-byte the one it ran before +- **`nibe_local`** 1.1.0 → 1.1.1 — **the version has to move anyway, because the signed artifact carries this metadata.** `ders` is not catalog-only decoration: `_load_channel` copies it into the artifact's `capabilities` and the channel publishes source and metadata as one signed file, so correcting the field changed the published bytes. The channel then refused the release — `error: nibe_local: changed artifact needs a higher version than 1.1.0` — and it was right to, since that immutability is what lets an operator name the bytes a site is running. A patch rather than a minor: no register, field or emitted metric changed, only the DER the catalog states. The `DRIVER` table is deliberately left alone, keeping the driver byte-identical to its FTW baseline the way #29 established + ### Added - **A published driver catalog at [srcfl.github.io/device-drivers](https://srcfl.github.io/device-drivers/)** — all 80 drivers, searchable by manufacturer or model number and filterable by device type, protocol, tier and control. `tools/generate_site.py` builds it from the manifests and the Lua sources on every push to `main`, so no copy of this data is checked in and the page cannot fall behind what a host installs. It reports hardware evidence the way the repository does rather than the way a catalog usually does: 5 drivers state they have been confirmed against physical hardware and 28 state that they have not, and `tests/test_site.py` fails the build if a driver's `verification_status` is ever rendered as a stronger claim than the driver makes