`erebor` lost `ZVTBSDL3` at lifetime hour 27, node_exporter hung nine minutes before the fault, and `erebor/apps` has no copy anywhere else

[Gerrrt]

What happened

At 20:55 PDT on 2026-09-19 (03:55 UTC 2026-09-20) TrueNAS on smaug raised:

Pool erebor state is ONLINE: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state.
Disk ST18000NM003D-3DL103 ZVTBSDL3 is FAULTED

ZVTBSDL3 is sdb, one half of the two-disk Exos X20 mirror
build-the-nas.md §3
built on 2026-09-18. It was at about lifetime hour 27, one day after arrival, and its
extended self-test had completed without error at hour 26 that same evening
(hardware.md, Exos entry).
The TS150 has two trays and no spare, so the pool runs on ZVTBS4NL alone until a
replacement is fitted.

What the monitoring host saw, read the same night:

When (PDT, 2026-09-19)	What
20:46	up{job="node",instance="smaug"} went 1 → 0 and stayed there. TCP 9100 still accepts a connection; GET /metrics never returns (curl gives up at 30 s, exit 28). A collector is blocked on the faulted device.
20:50	InstanceDown firing, severity=critical, routed to urgent. Still firing.
20:55	The TrueNAS alert above.

Three things this exposed

1. No rule reads node_zfs_zpool_state. smaug has exported it since the scrape went live (feat(nas): the drives are in, erebor exists, the stack runs and the scrape is on #522) — one 0/1 series per pool per state, for erebor and boot-pool — and nothing in stacks/observability/prometheus/rules/ reads any node_zfs_* series. The only page was InstanceDown, and only because the exporter hung. The rule that should have fired is in the PR on this issue; it can only catch the case where the exporter survives the fault, and that limit is written into it.
2. erebor/apps has zero off-host copies. feat(backup): pull Jellyfin's state off smaug from a ZFS snapshot, encrypt it here and copy it to oracle (#484) #554 merged scripts/backup-nas.sh today and closed erebor/apps is declared backed up in two files and backed up by nothing #484, but on the monitoring host backups/nas does not exist, homelab-backup-nas.timer is not installed, ssh frodo@10.0.40.30 is refused, and build-the-nas.md §6.2 is marked Not yet done. ADR-0008 calls that dataset the irreplaceable half of the media tier, and it is now on one spinning disk and nowhere else.
3. No runbook covered a NAS disk swap, and build-the-nas.md §7 and hardware.md both still read that the pool is ONLINE with no errors.

The clock

The pair is eBay item 237056026029, delivered 2026-09-18. The eBay Money Back Guarantee runs
30 days from delivery, so a "not as described / defective" return must be opened by
2026-10-18. Whether Seagate's own warranty applies to an "0HR" lot is unknown until the
serial is checked.

What has to happen, in this order

• Triage at the console: zpool status -v erebor, dmesg for sdb, smartctl -a, -l selftest and -l farm on sdb; readings into the runbook's status block; media-node-exporter restarted, or the hang recorded if it does not come back
• erebor/apps copied off and verified — §6.2 and make backup-nas, or the console tar in the runbook — before the tray is touched — done 2026-09-20: set 20260920T060234Z, jellyfin.db present, hashed on oracle (docs(nas): snapshot task and backup pull done; port 22 stops being inert everywhere it was said (#484) #563, erebor/apps is declared backed up in two files and backed up by nothing #484 closed on it)
• ZpoolNotOnline rule and its tests merged (feat(nas): erebor lost a disk one day in — a rule that reads pool state, the runbook for the swap, and the record of what fired first #559) — and, read at the console, blind to this fault: the pool kstat says online with a faulted leaf. A vdev-level signal is now a line of its own below
• A signal that sees a faulted leaf under an ONLINE pool: TrueNAS alert service to a phone channel, or a textfile metric on smaug (smaug has no SMART and no patch state, and the mechanism every other host uses is a push it cannot make #483)
• Return opened under the eBay guarantee before 2026-10-18; Seagate warranty check by serial recorded
• Replacement decided: the seller's replacement, or a purchase and a row in the roadmap's buy list
• Disk offlined, swapped, replaced in the UI; resilver complete; scrub clean
• hardware.md carries the new serial and its §2 readings; build-the-nas.md §7's zpool status line is true again
• The faulted drive wiped, or the decision recorded, before it ships back
• homelab-backup-nas.timer installed (§6.2 step 7), so the copy stops being a one-off — done 2026-09-20, next run Sat 2026-09-26 03:32 UTC

Runbook: docs/runbooks/replace-the-nas-disk.md. Refs #483 (why nothing on smaug can
push more than node_exporter offers), #413 (the build this happened one day after).

---

Corrected 2026-09-20

Point 2 under Three things this exposed is answered: erebor/apps has an off-host copy as of 20260920T060234Z, frodo and SSH exist on smaug, §6.2 reads Done 2026-09-20, and the timer is installed — #484 closed on it. Two boxes above are ticked with the evidence. The tray can be touched.

Milestone NAS, seq/1: this swap is now what that milestone closes on, beside #523. The vdev-level signal (a faulted leaf under an ONLINE pool) stays an open box here; #483, which owns how anything host-level reaches smaug, is re-sequenced to seq/1 in its own milestone so the two are worked together rather than decided twice.

[Gerrrt]

Step 1 read at the console, 2026-09-19 23:19 PDT. It is the drive.

• zpool status -v erebor: pool ONLINE, mirror-0 ONLINE, leaf 24c4970d… FAULTED, 3 read / 99 write / 0 cksum, "too many errors", no known data errors.
• dmesg from 20:47:12: Sense Key: Not Ready / Logical unit not ready, cause not reportable, 60 s command timeouts, task aborts, one target reset that succeeded and changed nothing; reads and writes failing at sector 0, 2080 and the far end of the disk alike. No link resets, no SError, no CRC errors. The drive goes away; the path does not.
• SMART at lifetime hour 32: PASSED; 850 Current_Pending_Sector, 850 Offline_Uncorrectable (both 0 on 2026-09-18); Command_Timeout normalised 1; 0 reallocated; error log empty; extended self-test still logged clean at hour 26.
• FARM: all 850 reallocation candidates on head 5; CTO count 124; hardware reset count 179; 12 V 12.150, 5 V 4.960; 28 °C.

So the fault was at about lifetime hour 27, an hour after a clean extended self-test, on one head, and the eBay return is the answer. ZVTBS4NL is unaffected.

The exporter came back on docker restart media-node-exporter at 23:26 PDT: /metrics in 46 ms, all collectors succeeding, up = 1, InstanceDown resolved.

Which settles the runbook's first open question the wrong way. node_zfs_zpool_state{zpool="erebor",state="online"} reads 1 — the kstat is the pool state, and the pool state is ONLINE with a faulted leaf. ZpoolNotOnline (#559) cannot see this fault. From 23:27 PDT no alert in the estate is firing for a mirror running on one disk; only TrueNAS's own alert knows, and it reaches the web UI and nothing else. Two ways out, for this issue to pick: a TrueNAS alert service pointed at a channel that reaches a phone (System → Alert Settings; the estate's ntfy receiver if the type list offers it), or a periodic task on smaug writing vdev states to a textfile the exporter serves — #483's argument.

One thing the readings do not match the docs on. readlink /sys/block/sdb resolves under host0 at PCI 0000:01:00.0, handled with SCSI task aborts and target resets, while the board's ata2 and ata3 read SATA link down at boot. build-the-nas.md §1 says the trays are on SATA2/SATA3. Either there is a controller in the PCIe slot that hardware.md does not list, or the port names are wrong. lspci -nn and readlink /sys/block/sda settle it.

Readings recorded in the runbook's status block and the Exos entry in hardware.md.

[Gerrrt]

The controller question is settled, and it changes the hardware record. lspci -nn at the console, 2026-09-19:

00:17.0 SATA controller: Intel Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] [8086:a102]
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] [1000:005f]

and readlink /sys/block/sda resolves under host0 at 01:00.0, target 8, beside sdb at target 9. Both Exos are behind a MegaRAID SAS3008 in the PCIe slot — 1000:005f is the SAS3008's MegaRAID personality (the ThinkServer RAID 520i option for the TS150); 1000:0097 would be the same silicon in IT mode. The chipset AHCI carries the boot SSD alone; ata1–ata5 read link down at boot. No document recorded the card, build-the-nas.md §1 says the trays are on SATA2/SATA3, and hardware.md says AHCI is what keeps a controller out from between ZFS and the disks. All three are corrected on #565.

Two things follow:

• The failure was handled by the MegaRAID firmware, not the chipset — the 60 s command timeouts, task aborts and target reset in dmesg are its error path, and they are what held the exporter for forty minutes.
• Step 4 of the runbook has a new check. A RAID card not in JBOD mode holds a replacement disk as Unconfigured Good and the OS never sees it; the runbook now says to confirm the new device in lsblk and where to go if it is missing.

Still unread, and worth two commands at the console before the swap: cat /sys/class/scsi_host/host0/proc_name (the driver) and dmesg -T | grep -iE 'megaraid|megasas' (firmware version, and whether the disks are JBOD). smartctl reaching the drives by their own model without -d megaraid points to JBOD pass-through. Whether to leave the card, flash the 3008 to IT firmware, or cable the bays to the chipset's free ports and remove the card is a decision for this issue after the swap.

[Gerrrt]

Mode read: JBOD pass-through on a cacheless iMR. cat /sys/class/scsi_host/host0/proc_name → megaraid_sas; dmesg at boot: driver 07.727.03.00-rc1, controller type : iMR(0MB), pci id 1000:005f / subsystem 1000:9340 (the 9340-8i family the ThinkServer RAID 520i is built on), Secure JBOD support : Yes, JBOD sequence map : enabled. So the disks are JBOD devices, not virtual drives — which is why smartctl reaches them by their own model and serial — behind a controller with no write cache and no battery. The better of the two MegaRAID arrangements, and still not an IT-mode HBA: the 60 s timeouts, task aborts and the target reset in the fault's dmesg are its firmware's error path, and the driver logged a disable/enable of its interrupts at 21:03:51, the same second as that reset — the Online Controller Reset it advertises.

Recorded in hardware.md's smaug entry and the runbook's open list on #565. One reading still owed, when convenient:

cat /sys/class/scsi_host/host0/fw_ver

Nothing here changes the swap: a JBOD card presents a fresh disk as JBOD only if it is configured to, so step 4's lsblk check stays.

[Gerrrt]

Correction to the comment above: /sys/class/scsi_host/host0/fw_ver does not exist — that path was wrong. megaraid_sas exposes crash-dump and queue attributes in sysfs and nothing about its firmware, and TrueNAS ships no storcli. The version is read off the card's POST banner, or from Ctrl-R → controller properties during boot, so it gets read at step 4 of the runbook when the machine is at POST for the swap anyway. hardware.md and the runbook now say so, on #565.

[Gerrrt]

Two things this issue parked now have their own tracker, filed 2026-09-20 so they survive this one closing on the swap: #571 is the controller decision ("leave the card, flash to IT, or cable to the chipset — after the swap"), and #570 is the exporter that reported healthy for 2 h 40 min while /metrics hung. The firmware version read at POST in step 4 goes to #571. Nothing in this checklist moves.