docs(nas): the console readings — it is the drive, head 5, and the pool-state rule cannot see it

docs/hardware.md

@@ -261,7 +261,44 @@ revisions of this repository treated `shiva` as the hypervisor itself.
   [#418](https://github.com/Gerrrt/HomeLab/issues/418) is the cautionary tale
   for, and `CSM [Disabled]`, so it boots UEFI as TrueNAS wants.
   Two 3.5" trays, filled by the Exos pair on 2026-09-18 — exactly the mirror
-  and no spare.
+  and no spare. **The trays are not on those six ports.** Read with `lspci`
+  and `readlink` at the console on 2026-09-19, while triaging the faulted
+  disk: both Exos enumerate under `host0` at PCI `01:00.0`, a **Broadcom /
+  LSI MegaRAID SAS-3 3008 "Fury"**, PCI ID `1000:005f` — the SAS3008 in its
+  MegaRAID personality, which is the ThinkServer RAID 520i option for this
+  chassis, sitting in the PCIe slot and cabled to the bays. The chipset AHCI
+  at `00:17.0` carries only the boot SSD on `ata6`; `ata1`–`ata5` read *SATA
+  link down* at boot. So `Configure SATA as [AHCI]` protects the boot disk
+  and nothing else, and the pool has had a RAID controller between ZFS and
+  its disks since the day it was built — the arrangement the line above
+  calls #418's cautionary tale. It is the MegaRAID firmware, not the
+  chipset, that handled `sdb`'s failure with 60-second command timeouts,
+  task aborts and a target reset. `smartctl` reaches the drives without a
+  `-d megaraid` option and reports them by their own model and serial, which
+  is what a JBOD pass-through looks like — and `dmesg` confirms it, read the
+  same night: driver `megaraid_sas` 07.727.03.00-rc1, controller type
+  **`iMR(0MB)`** — the cacheless entry-level MegaRAID, no write cache and
+  no battery to worry about — subsystem `1000:9340`, which is the 9340-8i
+  family the ThinkServer RAID 520i is built on, *Secure JBOD support: Yes*,
+  and **`JBOD sequence map : enabled`**, which is the driver's way of
+  saying the disks are JBOD devices rather than virtual drives. So ZFS sees
+  the drives themselves through a RAID firmware's error handling, which is
+  the better of the two arrangements a MegaRAID offers and still not an
+  IT-mode HBA. The firmware version is not in `dmesg` and not in sysfs
+  either — `/sys/class/scsi_host/host0/fw_ver` does not exist, read
+  2026-09-19, and `megaraid_sas` exposes crash-dump and queue attributes
+  there and nothing about its firmware. TrueNAS ships no `storcli`. So the
+  version is read off the card's own POST banner, or from *Ctrl-R* →
+  controller properties during boot, and it is owed here from the next
+  time the machine is at POST — which the swap will be. The driver
+  logged a disable/enable of its interrupts at 21:03:51 on 2026-09-19, the
+  same second as the target reset in the fault's `dmesg` — the controller
+  resetting itself around a disk that had stopped answering, which is the
+  Online Controller Reset it advertises as enabled. `1000:005f` is the ID to watch:
+  `1000:0097` is the same silicon in IT mode, and the card was never
+  recorded here, like the optical drive was not. Cabling below, in
+  [`build-the-nas.md`](runbooks/build-the-nas.md) §1, says `SATA2` and
+  `SATA3`; that is now known to be wrong, and the same reading corrects it.
   **The 5.25" bay was not empty**: a PLDS `DVD-RW DU8AESH` answered on SATA5.
   A photograph of the open case had been read here as an empty cage and was
   wrong; the BIOS summary is what caught it. The optical drive came out on
@@ -318,7 +355,16 @@ revisions of this repository treated `shiva` as the hypervisor itself.
   said at the console, and the outcome, belong in
   [`replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)'s status
   block and then here; [#558](https://github.com/Gerrrt/HomeLab/issues/558)
-  carries it.
+  carries it. **Read at the console on 2026-09-19 at 23:19 PDT, at lifetime
+  hour 32: it is the drive.** `zpool status` counts 3 read and 99 write
+  errors on the leaf; `dmesg` is *Logical unit not ready* and 60-second
+  command timeouts from 20:47 on, with no link resets and no CRC errors;
+  SMART reads **850 pending and 850 offline-uncorrectable sectors** against
+  the 0 and 0 of the day before, `Command_Timeout` normalised to 1, overall
+  health still `PASSED`; and FARM puts **all 850 reallocation candidates on
+  head 5**, with 124 command timeouts and 179 hardware resets, both rails
+  in spec, 28 °C. `ZVTBS4NL` is unaffected. The extended self-test that
+  completed at hour 26 was true when it was read and is not evidence now.
 - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss
   protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot
   disk, carrying TrueNAS and the media stack it launches

docs/runbooks/build-the-nas.md

@@ -309,6 +309,18 @@ anything in this runbook.
 > alone. **The power lead had four wires and no orange one**, so the pin-3
 > trap below did not fire on this supply; both drives spun up and appeared in
 > `lsblk` first time.
+>
+> **Corrected 2026-09-19: the trays are not on `SATA2` and `SATA3`.** Read
+> with `lspci` and `readlink /sys/block/sd?` while triaging the faulted disk
+> ([`replace-the-nas-disk.md`](replace-the-nas-disk.md)): both Exos sit
+> behind a **Broadcom / LSI MegaRAID SAS-3 3008** at PCI `01:00.0`, the RAID
+> card in the PCIe slot that no document had recorded, and the chipset's
+> `ata1`–`ata5` read *SATA link down* at boot. The board's AHCI carries the
+> boot SSD alone. The cabling instruction below is kept as written, because
+> it is what was believed when the drives went in; what is actually cabled is
+> in [`hardware.md`](../hardware.md)'s `smaug` entry, and the consequence —
+> a controller between ZFS and its disks, and what that means for the swap —
+> is in the disk runbook's step 4 and open list.
 
 Power down, unplug, hold the power button five seconds, ground yourself.
 

docs/runbooks/replace-the-nas-disk.md

@@ -3,8 +3,9 @@
 **One faulted Exos, two trays and no spare, and a dataset with no copy
 anywhere else — so the copy comes first and the tray comes last.**
 
-> **Status — 2026-09-20: nothing below is done. The pool runs on one disk,
-> and `erebor/apps` exists only on it.**
+> **Status — 2026-09-20: step 1 is read and it is the drive; steps 2–6 are
+> not done. The pool runs on one disk, `erebor/apps` exists only on it, and
+> since the exporter came back nothing is paging for either.**
 >
 > | When (PDT, 2026-09-19) | What |
 > | --- | --- |
@@ -25,6 +26,34 @@ anywhere else — so the copy comes first and the tray comes last.**
 > refused, and §6.2 is marked *Not yet done*. That is step 2, and it is why
 > the tray waits.
 > [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries this.
+>
+> **Step 1 read at the console, 2026-09-19 23:19 PDT. It is the drive, and
+> the return is the answer.** `zpool status -v erebor`: pool `ONLINE`,
+> `mirror-0` `ONLINE`, the `24c4970d…` leaf `FAULTED` with **3 read, 99
+> write, 0 checksum** errors, "too many errors", `errors: No known data
+> errors`. `dmesg` from 20:47:12 onward is one shape only: `Sense Key: Not
+> Ready`, *Logical unit not ready, cause not reportable*, commands timing
+> out at 60 s and aborted, a target reset that succeeded and changed
+> nothing, reads and writes failing at sector 0, at 2080, and at the far end
+> of the disk alike — the drive going away, not the path. No link resets, no
+> `SError`, no CRC. SMART at lifetime hour 32: overall `PASSED`, **850
+> pending and 850 offline-uncorrectable sectors** where both read 0 on
+> 2026-09-18, `Command_Timeout` normalised to **1**, no reallocations, error
+> log empty, the extended self-test still logged as completed clean at hour
+> 26. FARM says which head: **all 850 reallocation candidates are on head
+> 5**, 124 command timeouts, 179 hardware resets, 12 V and 5 V rails inside
+> spec, 28 °C. The fault was at about **lifetime hour 27**, one hour after
+> the self-test that passed.
+>
+> **The exporter came back on `docker restart media-node-exporter`** at
+> 23:26 PDT (06:26 UTC 2026-09-20): `/metrics` in 46 ms, every collector
+> reporting success, `up` back to 1 on the next scrape, `InstanceDown`
+> resolved. **And that is the problem.** The kstat behind
+> `node_zfs_zpool_state` reads `online` for `erebor` — the pool state, which
+> `zpool status` also prints as `ONLINE` — so `ZpoolNotOnline` sees nothing,
+> and from 23:27 PDT **no alert in the estate is firing for a mirror running
+> on one disk.** The only thing that noticed is TrueNAS's own alert, which
+> reaches the web UI and nothing else. See *What is still open*.
 
 `smaug` is the TrueNAS host at `10.0.40.30` on CasaBonita, which is terminal
 outward ([ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)):
@@ -191,7 +220,15 @@ leads, so a cable that was the fault is found by the next reading rather
 than hidden by a fresh one. Leave the `AUX1_FAN` cage fan alone; it is the
 airflow over both trays and §1 says why it is not optional.
 
-Power on. **Read the new drive before trusting it**, §2-style:
+Power on, and **check that the new drive is there at all**: `lsblk` must
+show a third 18 TB device. The bays are behind a MegaRAID SAS3008, not the
+chipset (see *What is still open*), and a RAID card that is not in JBOD
+mode holds a fresh disk as *Unconfigured Good* and shows the operating
+system nothing. If `lsblk` has no new device, the card's own boot-time
+utility (`Ctrl-R` during POST on a MegaRAID) is where the disk is made a
+JBOD, and that reading — the card's firmware, its mode — goes in
+[`hardware.md`](../hardware.md) the same evening. Then **read the new drive
+before trusting it**, §2-style:
 
 ```bash
 smartctl -a /dev/sdX
@@ -251,16 +288,46 @@ refund fight — is the operator's, and it is recorded here.
 
 ## What is still open
 
-- **Whether `zpool status` says `DEGRADED` or `ONLINE` for a faulted mirror
-  leaf.** TrueNAS's alert said `ONLINE`. Step 1 settles it, and it decides
-  whether `ZpoolNotOnline` can see this class of fault at all.
-- **No vdev-level metric.** node_exporter exports pool state and nothing per
-  device; [#483](https://github.com/Gerrrt/HomeLab/issues/483) is why
-  nothing on `smaug` can push more. A `FAULTED` leaf under an `ONLINE` pool
-  is invisible from here until that changes.
-- **Whether the exporter hangs on every device fault.** If step 1's restart
-  does not bring it back, then `InstanceDown` is the NAS disk alert in
-  practice and this runbook should say so at the top.
+- **Settled 2026-09-19: `zpool status` says `ONLINE` for a faulted mirror
+  leaf, and so does the kstat.** `ZpoolNotOnline` cannot see this class of
+  fault. It still catches a pool that is genuinely degraded, suspended or
+  unavailable — a resilver, a second disk gone — but the one fault that has
+  actually happened is below its resolution.
+- **No vdev-level metric, and now nothing fires.** node_exporter exports
+  pool state and nothing per device; [#483](https://github.com/Gerrrt/HomeLab/issues/483)
+  is why nothing on `smaug` can push more. With the exporter restarted,
+  `InstanceDown` has resolved and no alert covers the degraded mirror. Two
+  ways out, neither built: point TrueNAS's own alert service at a channel
+  that reaches a phone (System → Alert Settings → Add; the estate's ntfy
+  receiver is the obvious candidate if the type list offers it), or have a
+  periodic task on `smaug` write `zpool status` vdev states to a textfile
+  the exporter serves — the mechanism #483 argues about.
+- **The exporter's hang was the fault, not a habit.** It came back on a
+  restart within seven minutes of the console session, so `InstanceDown` is
+  the NAS disk alert only while the device is still timing out I/O. The
+  runbook now says so at the top.
+- **Settled 2026-09-19: the pair is behind a RAID card.** `lspci -nn` reads
+  a **Broadcom / LSI MegaRAID SAS-3 3008 "Fury"**, `1000:005f`, at
+  `01:00.0`, and both `sda` and `sdb` resolve under its `host0`; the
+  chipset AHCI carries the boot SSD alone. [`build-the-nas.md`](build-the-nas.md)
+  §1's `SATA2`/`SATA3` was wrong and now says so;
+  [`hardware.md`](../hardware.md) carries the card. **Read the same
+  night:** driver `megaraid_sas`, controller type `iMR(0MB)` — cacheless,
+  no battery — subsystem `9340`, and `JBOD sequence map : enabled`, so the
+  disks are **JBOD pass-through**, not virtual drives, which is why
+  `smartctl` reaches them by their own model without `-d megaraid`. That is
+  the better of the two answers a MegaRAID offers and still not the IT-mode
+  HBA ZFS is designed for: the sixty-second timeouts, task aborts and the
+  21:03:51 controller reset in the fault's `dmesg` are its firmware's error
+  path, and ZFS waited on them. The firmware version is the one reading
+  still owed, and it is not in sysfs — `/sys/class/scsi_host/host0/fw_ver`
+  does not exist, and TrueNAS ships no `storcli` — so it is read off the
+  card's POST banner or its *Ctrl-R* controller properties, at step 4,
+  when the machine is at POST anyway. Whether to leave the
+  card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the
+  bays to the chipset's free `SATA0`–`SATA3` and take the card out is a
+  decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after
+  the swap, not before it.
 - **§6.2**, until step 2 path A has run — and `oracle`'s copy of the NAS
   set, which `backup-nas.sh` makes and nothing has yet made.
 - **The replacement decision**, and whether a third drive follows.

stacks/observability/prometheus/rules/host.rules.yaml

@@ -427,6 +427,17 @@ groups:
       # dead, InstanceDown is the signal, and the runbook's first step is at
       # the console either way.
       #
+      # AND THE RESOLUTION IT HAS. Read at the console on 2026-09-19 23:19
+      # PDT with the exporter restarted: `zpool status` printed the pool
+      # ONLINE, mirror-0 ONLINE and one leaf FAULTED (3 read, 99 write
+      # errors), and this kstat read online=1 for erebor. The kstat is the
+      # POOL state, and a two-disk mirror with one faulted leaf is, to the
+      # pool, online. So this rule cannot see the fault that produced it; it
+      # sees a pool that is degraded, suspended or unavailable as a whole —
+      # a resilver in progress, or the second disk gone. A per-vdev signal
+      # is #558's open line, and until it exists a one-disk mirror pages
+      # nothing here once the exporter is back.
+      #
       # `state!="online"` == 1 rather than `state="online"` == 0, so the label
       # carries WHICH state the pool is in and a state name the collector
       # learns later is caught without editing this line. boot-pool is covered