feat(manifest): add upstream_docs to watch vendor register sources

[HuggeK]

What

Two halves of one feature for keeping drivers honest against the vendor
documents they decode a device by:

1. upstream_docs — an optional manifest field recording the vendor
   reference documents (register maps, parameter changelogs) a driver was built
   against, at a semi-persistent URL, plus how durable that URL is.
2. watch-upstream-docs — a scheduled watcher that fetches each URL, notices
   when a document changes or disappears, and opens a tracking issue so a
   maintainer reviews whether the driver's registers are affected.

Populated for the two NIBE drivers (both watching NIBE's myUplink register
changelog), and corrects the nibe_local manifest author to its real provenance.

The author-facing guides ask for it, so the field is something a driver author is
told to fill in rather than something they have to find in the spec first.

upstream_docs:
  - url: "https://www.nibe.eu/webdav/files/myuplink_changelog/nibe-n.pdf"
    title: "NIBE S-series myUplink register/parameter changelog (nibe-n.pdf)"
    kind: changelog
    url_stability: stable

Why

A driver decodes a device by following the vendor's own reference material. When
that material changes upstream — a register renumbered, a parameter added — the
driver silently falls behind, and it's usually a human noticing a wrong value on
a real site months later. This makes that visible: record the document, watch it,
and raise a reviewable issue the moment it moves.

Implements the pilot proposed in #47.

The field

upstream_docs is an optional list, modelled on tested_devices:

Field	Required	Notes
url	yes	semi-persistent http(s) URL to watch
title	no	human-readable label
kind	no	changelog, register_map, manual, api_docs, firmware_notes, other
url_stability	no	committed, stable, volatile, unknown (default)

url_stability records whether the manufacturer keeps the URL put — committed
(vendor promises it's permanent), stable (stable in practice), volatile
(rotates), unknown. The watcher uses it to weigh a broken link: a stable one
going missing is a real signal the document moved; a volatile one breaking can
be routine.

The watcher

• tools/check_upstream_docs.py (stdlib only): fetches each distinct URL
  once, SHA-256s it, and diffs against a committed baseline
  (upstream-docs-state.json, seeded here for the two NIBE entries).
• .github/workflows/watch-upstream-docs.yml: weekly (+ manual). On a change
  it opens an issue labelled upstream-doc-changed listing old/new hash and a
  review checklist; on a document that has been unreachable for 3 runs it opens
  one labelled upstream-doc-unreachable. Then it commits the updated baseline.

Notify-once, and the raw-hash caveat

• A changed file is reported once. Detecting a change updates the baseline,
  so the next run sees the new hash as current and does not re-fire. The issue
  title carries the new hash's short prefix, and the workflow skips a title that
  already has an open issue — so even a failed baseline-commit can't double-post.
• A broken link is reported once. The unreachable alert fires only on the run
  that crosses the 3-run failure threshold, not every run after; recovery
  resets the count.
• Detection is a raw-byte SHA-256. A PDF's bytes can churn without its content
  changing (rebuild timestamps). So a flagged change is a prompt to look, not
  proof the registers moved — the issue says exactly that.
• Impact review (auto-diffing which registers changed) is deliberately left
  as a follow-up in RFC: watch manifest upstream_docs for changes and flag affected drivers #47; v1 surfaces the change and links the driver source.

Why it's safe to add and what it does not touch

• Descriptive metadata only. generate_index.py copies a fixed field set, so
  upstream_docs never reaches index.yaml and never affects how a driver
  installs or runs. devices.yaml and support-status.* are unaffected —
  regenerated and confirmed byte-identical.
• No version bump. The Lua source is untouched, so sha256/size_bytes are
  unchanged and no driver version moves. Both drivers stay byte-identical to
  their baselines/ftw sources.
• Enforced. validate_manifest.py requires an http(s) url and known
  kind/url_stability values, and rejects unknown entry fields.
• The watcher's contents: write / issues: write are scoped to that workflow.

Changes

• tools/manifest_parser.py — parse_upstream_docs()
• tools/validate_manifest.py — validates url / kind / url_stability / unknown fields
• tools/check_upstream_docs.py — the watcher (fetch, diff, render issues)
• .github/workflows/watch-upstream-docs.yml — weekly job
• upstream-docs-state.json — seeded baseline
• spec/manifest-v2.md — documents the field (V2.3)
• docs/WRITING-A-DRIVER.md, AGENTS.md — tell a driver author to record the documents they decoded the device from, and what the watcher cannot do
• manifests/nibe_local.yaml, manifests/myuplink.yaml — entries; nibe_local author corrected
• Makefile — make watch-upstream-docs (local dry-run)
• tests/test_upstream_docs.py, tests/test_upstream_docs_watch.py
• CHANGELOG.md — [Unreleased]

Testing

• python tools/validate_manifest.py → 80 manifests, 0 errors
• python tools/sync_manifests.py --check → no drift
• generate_index.py / generate_devices.py / generate_support_status.py → no content change
• pytest tests/test_upstream_docs.py tests/test_upstream_docs_watch.py → 23 passed
• python tools/check_upstream_docs.py --dry-run → fetches both NIBE URLs live, 0 failures

PR evidence

• Device / source: NIBE S-series (nibe_local) and MyUplink heat pumps
  (myuplink); watched document is NIBE's public myUplink register changelog
  (nibe-n.pdf).
• No hardware behaviour changes — manifest metadata plus tooling; no
  registers, emits, control paths or driver bytes are modified.
• Operational note: the baseline-commit step pushes to the default branch. If
  branch protection blocks the Actions bot, either allow github-actions[bot]
  for upstream-docs-state.json or switch that step to open a PR — issues are
  still filed correctly either way, and the open-issue dedup prevents duplicates.

RFC / discussion: #47

[frahlg]

Automated maintenance pass: not merging this one, flagging for a maintainer.

The change itself reads well-scoped — descriptive metadata only, generate_index.py doesn't copy upstream_docs through, no driver bytes/version touched, validate_manifest.py enforces the schema, and the PR reports 80/80 manifests valid, no sync drift, and 23 passing tests for the new tooling. On the merits of the diff I'd call this close to ready.

Two things stop it from going further autonomously:

1. It's from a non-collaborator contributor via a fork, and GitHub has held every CI run action_required (both pushes to this branch) rather than running it — this repo's CI has not actually validated this PR yet, and approving that requires a maintainer with the standing to review what external code is about to run in Actions.
2. It adds a new scheduled workflow (watch-upstream-docs.yml) with contents: write and issues: write permissions, authored externally. Granting an untrusted contribution write access to the repo and issue tracker via a recurring Action is exactly the kind of change that should get a human security read before its CI is even allowed to run, separate from whether the Python/Lua logic is correct.

Secondary note: the companion RFC (#47) still has open checklist items ("agree the surfacing mechanism," decisions to make "before the watcher is wired up") — worth confirming those are settled the way this PR settled them before merging.

Flagging for a maintainer to approve the workflow run (or review the Action file directly) and confirm the RFC is considered resolved.

---

Generated by Claude Code

[HuggeK]

It is now done.

[frahlg]

Automated maintenance pass, follow-up on the earlier flag.

What's changed since: CI now runs and is green (public-boundary, drivers, packages, dco, history-secret-scan all passed on 0bfddd0), and HuggeK holds maintain on this repo — so the "untrusted fork, CI never ran" concern from the previous pass no longer applies. I also read tools/check_upstream_docs.py and the new workflow in full: fetch is stdlib urllib only, the issue-creation step passes titles/bodies through jq and --body-file (no shell interpolation of fetched content), actions are pinned to commit SHAs, and contents/issues write permissions are scoped to just this workflow. No security concerns from the diff itself.

Why I'm still not merging it:

1. Real merge conflict against main, not just staleness — update_pull_request_branch fails with 422 merge conflict between base and head. This has to be resolved on the PR branch (a fork) before anything can merge; I can't push a resolution there myself.
2. Worth a maintainer's eyes regardless, since granting a new scheduled workflow contents: write + issues: write (auto-commits a baseline file, auto-opens issues) is a standing policy decision, not just a code-correctness one — even though the implementation looks clean.

Secondary: the companion RFC (#47) checklist is still unticked, though this PR does appear to settle "agree the surfacing mechanism" by implementing it. Not a blocker, just flagging it's still open.

Ask: @HuggeK — could you rebase/resolve the conflict against current main? Once it's mergeable I'll re-check, but the write-permissions question is one for a maintainer to explicitly sign off on either way.

---

Generated by Claude Code

[HuggeK]

Conflict resolved — the branch now merges cleanly against current main (892a599, CI green).

What conflicted: exactly one hunk, the .PHONY continuation line in Makefile. main added site (plus refused-write-report / absent-register-report) in #50/#51; this branch added watch-upstream-docs. Both edits landed on the same line. Resolution keeps every target from both sides and main's site: recipe verbatim — nothing was dropped.

Merged rather than rebased so the branch history stays append-only and the two reviewed commits keep their hashes; no force-push.

Second commit: a CHANGELOG entry moved, no content changed

Merging carried both upstream_docs entries along inside the ### Added group they were written in — next to BLUEPRINT.lua — which now reads as though the field shipped in that batch. [Unreleased] prepends a group per merged change, so they moved to the top. This is the same move 6f8eab7 made for the catalog entry after its own rebase. The entry text is byte-identical; only its position changed.

Re-verified after the merge

main gained tools/generate_site.py, which reads the manifests — so the interaction worth checking is whether the two manifests carrying upstream_docs still render:

• tools/validate_manifest.py → 80 manifests, 0 errors
• tools/sync_manifests.py --check → 80 match their source, no drift
• tools/generate_index.py → index.yaml byte-identical (upstream_docs still never reaches it)
• tools/generate_site.py → 80 drivers, 69 manufacturers, 171 model families; upstream_docs correctly absent from drivers.json, so the catalog page is unaffected
• tools/check_upstream_docs.py --dry-run → 2 watched documents, 0 failures, 0 alerts

The full pytest suite ran in CI rather than locally (no uv on the machine I resolved this on) — drivers, packages, public-boundary, dco and history-secret-scan all pass on 892a599.

On the standing question — granting a scheduled workflow contents: write + issues: write — that's yours to decide, not something I can settle by resolving a conflict, and it's a fair thing to want explicit. Two notes that may narrow it:

• The contents: write exists only to commit upstream-docs-state.json. If you'd rather no scheduled job writes to the default branch at all, that step can open a PR instead, or the baseline can move to an Actions cache / artifact — issues are still filed correctly either way, and the open-issue title dedup already prevents duplicates if the commit fails. Happy to make that change if you prefer it as the shipping shape.
• issues: write is load-bearing; the whole point is the tracking issue.

RFC #47's checklist is still unticked — agreed that's not a blocker, and this PR does implement the surfacing mechanism it proposes.

[HuggeK]

Added the missing half of this: the guidance to actually use the field.

upstream_docs was specified in spec/manifest-v2.md and nowhere else — which is where you read for field syntax after you've decided to use a field, not where you find out you should. Neither AGENTS.md nor docs/WRITING-A-DRIVER.md mentioned it, so nothing in the repository ever asked an author for the one thing the watcher needs. That's a real gap in the feature as proposed: a field nobody is told to populate stays at 2 manifests out of 80, and a driver with no entry is one nobody gets warned about.

WRITING-A-DRIVER.md already argues that the valuable part of a driver is the knowledge rather than the code, and tells you to write down what you learn about the registers as you find it. Where that knowledge came from is the half it never asked for — so the reasoning and the YAML now sit beside the manifest step that writes it, and AGENTS.md carries the one-line rule alongside the other source rules.

Both state the limits rather than overselling it: a document behind a login or a vendor portal can't be fetched and belongs in a driver comment instead, and detection is a hash of the bytes, so a flagged document is a prompt to look rather than proof the registers moved.

No code, manifest or generated file changes — three markdown files. Re-verified after the commit: validate_manifest 80/0, sync_manifests --check no drift, generate_site 80 drivers, check_upstream_docs --dry-run 0 failures. CI green on 08d1acd; still MERGEABLE / CLEAN.