The blind spot, stated before it costs anything
Saruman now has two LVM-thin pools: pve/data on the HDD mirror, holding alexander, and Large_data/Large_data on the SSD logical drive made on 2026-09-19 (#527). A thin pool is not a filesystem, so node_filesystem_* never sees either one — the host reports only /, /boot/efi and /etc/pve. HostDiskCritical and HostDiskWillFillIn24h in stacks/observability/prometheus/rules/host.rules.yaml therefore cannot fire for a pool, and a thin pool that fills makes every guest on it read-only, at once, with no warning from anything in this estate.
docs/runbooks/fit-the-saruman-ssds.md §7 records the gap and says it "deserves its own issue, not a step here". This is that issue. It is the same shape as #351: a reading the host has and the stack never asked for.
What closes it
- A textfile collector beside
scripts/collect-guest-state.sh — same unit and timer shape, same --print / --self-test contract — that emits, per pool, lvs --noheadings -o vg_name,lv_name,data_percent,metadata_percent,lv_size --select 'lv_attr=~^t' as gauges: homelab_thin_pool_data_percent{vg,pool}, homelab_thin_pool_metadata_percent{vg,pool}, and the pool size. lvs needs root, which the agent collectors already run as.
- Two rules with unit tests: one on data or metadata use above a threshold for a sustained window (
warning), one on predict_linear of data use reaching 100 % inside 24 h (critical, since the consequence is every guest on the pool going read-only). Metadata full is worse than data full and the threshold should be lower.
- A staleness check in the shape of
GuestStateStopped, or the existing one extended, so a collector that stops does not look like a pool that is fine.
- A line in
docs/observability.md naming why disk alerts on Saruman come in two kinds — filesystems from node_exporter, pools from the collector — and a row in the runbook's verdict table.
Done when
Both pools have data_percent and metadata_percent series here, the two rules pass make check-rules with fixtures for both pools, and the runbook's §7 blind-spot paragraph points at the collector rather than at this issue.
Refs #527, #351.
The blind spot, stated before it costs anything
Sarumannow has two LVM-thin pools:pve/dataon the HDD mirror, holdingalexander, andLarge_data/Large_dataon the SSD logical drive made on 2026-09-19 (#527). A thin pool is not a filesystem, sonode_filesystem_*never sees either one — the host reports only/,/boot/efiand/etc/pve.HostDiskCriticalandHostDiskWillFillIn24hinstacks/observability/prometheus/rules/host.rules.yamltherefore cannot fire for a pool, and a thin pool that fills makes every guest on it read-only, at once, with no warning from anything in this estate.docs/runbooks/fit-the-saruman-ssds.md§7 records the gap and says it "deserves its own issue, not a step here". This is that issue. It is the same shape as #351: a reading the host has and the stack never asked for.What closes it
scripts/collect-guest-state.sh— same unit and timer shape, same--print/--self-testcontract — that emits, per pool,lvs --noheadings -o vg_name,lv_name,data_percent,metadata_percent,lv_size --select 'lv_attr=~^t'as gauges:homelab_thin_pool_data_percent{vg,pool},homelab_thin_pool_metadata_percent{vg,pool}, and the pool size.lvsneeds root, which the agent collectors already run as.warning), one onpredict_linearof data use reaching 100 % inside 24 h (critical, since the consequence is every guest on the pool going read-only). Metadata full is worse than data full and the threshold should be lower.GuestStateStopped, or the existing one extended, so a collector that stops does not look like a pool that is fine.docs/observability.mdnaming why disk alerts onSarumancome in two kinds — filesystems fromnode_exporter, pools from the collector — and a row in the runbook's verdict table.Done when
Both pools have
data_percentandmetadata_percentseries here, the two rules passmake check-ruleswith fixtures for both pools, and the runbook's §7 blind-spot paragraph points at the collector rather than at this issue.Refs #527, #351.