diff --git a/docs/hardware.md b/docs/hardware.md index c75a27ad..7a608f1e 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -261,7 +261,44 @@ revisions of this repository treated `shiva` as the hypervisor itself. [#418](https://github.com/Gerrrt/HomeLab/issues/418) is the cautionary tale for, and `CSM [Disabled]`, so it boots UEFI as TrueNAS wants. Two 3.5" trays, filled by the Exos pair on 2026-09-18 — exactly the mirror - and no spare. + and no spare. **The trays are not on those six ports.** Read with `lspci` + and `readlink` at the console on 2026-09-19, while triaging the faulted + disk: both Exos enumerate under `host0` at PCI `01:00.0`, a **Broadcom / + LSI MegaRAID SAS-3 3008 "Fury"**, PCI ID `1000:005f` — the SAS3008 in its + MegaRAID personality, which is the ThinkServer RAID 520i option for this + chassis, sitting in the PCIe slot and cabled to the bays. The chipset AHCI + at `00:17.0` carries only the boot SSD on `ata6`; `ata1`–`ata5` read *SATA + link down* at boot. So `Configure SATA as [AHCI]` protects the boot disk + and nothing else, and the pool has had a RAID controller between ZFS and + its disks since the day it was built — the arrangement the line above + calls #418's cautionary tale. It is the MegaRAID firmware, not the + chipset, that handled `sdb`'s failure with 60-second command timeouts, + task aborts and a target reset. `smartctl` reaches the drives without a + `-d megaraid` option and reports them by their own model and serial, which + is what a JBOD pass-through looks like — and `dmesg` confirms it, read the + same night: driver `megaraid_sas` 07.727.03.00-rc1, controller type + **`iMR(0MB)`** — the cacheless entry-level MegaRAID, no write cache and + no battery to worry about — subsystem `1000:9340`, which is the 9340-8i + family the ThinkServer RAID 520i is built on, *Secure JBOD support: Yes*, + and **`JBOD sequence map : enabled`**, which is the driver's way of + saying the disks are JBOD devices rather than virtual drives. So ZFS sees + the drives themselves through a RAID firmware's error handling, which is + the better of the two arrangements a MegaRAID offers and still not an + IT-mode HBA. The firmware version is not in `dmesg` and not in sysfs + either — `/sys/class/scsi_host/host0/fw_ver` does not exist, read + 2026-09-19, and `megaraid_sas` exposes crash-dump and queue attributes + there and nothing about its firmware. TrueNAS ships no `storcli`. So the + version is read off the card's own POST banner, or from *Ctrl-R* → + controller properties during boot, and it is owed here from the next + time the machine is at POST — which the swap will be. The driver + logged a disable/enable of its interrupts at 21:03:51 on 2026-09-19, the + same second as the target reset in the fault's `dmesg` — the controller + resetting itself around a disk that had stopped answering, which is the + Online Controller Reset it advertises as enabled. `1000:005f` is the ID to watch: + `1000:0097` is the same silicon in IT mode, and the card was never + recorded here, like the optical drive was not. Cabling below, in + [`build-the-nas.md`](runbooks/build-the-nas.md) §1, says `SATA2` and + `SATA3`; that is now known to be wrong, and the same reading corrects it. **The 5.25" bay was not empty**: a PLDS `DVD-RW DU8AESH` answered on SATA5. A photograph of the open case had been read here as an empty cage and was wrong; the BIOS summary is what caught it. The optical drive came out on @@ -318,7 +355,16 @@ revisions of this repository treated `shiva` as the hypervisor itself. said at the console, and the outcome, belong in [`replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)'s status block and then here; [#558](https://github.com/Gerrrt/HomeLab/issues/558) - carries it. + carries it. **Read at the console on 2026-09-19 at 23:19 PDT, at lifetime + hour 32: it is the drive.** `zpool status` counts 3 read and 99 write + errors on the leaf; `dmesg` is *Logical unit not ready* and 60-second + command timeouts from 20:47 on, with no link resets and no CRC errors; + SMART reads **850 pending and 850 offline-uncorrectable sectors** against + the 0 and 0 of the day before, `Command_Timeout` normalised to 1, overall + health still `PASSED`; and FARM puts **all 850 reallocation candidates on + head 5**, with 124 command timeouts and 179 hardware resets, both rails + in spec, 28 °C. `ZVTBS4NL` is unaffected. The extended self-test that + completed at hour 26 was true when it was read and is not evidence now. - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot disk, carrying TrueNAS and the media stack it launches diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 475186d0..90354695 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -309,6 +309,18 @@ anything in this runbook. > alone. **The power lead had four wires and no orange one**, so the pin-3 > trap below did not fire on this supply; both drives spun up and appeared in > `lsblk` first time. +> +> **Corrected 2026-09-19: the trays are not on `SATA2` and `SATA3`.** Read +> with `lspci` and `readlink /sys/block/sd?` while triaging the faulted disk +> ([`replace-the-nas-disk.md`](replace-the-nas-disk.md)): both Exos sit +> behind a **Broadcom / LSI MegaRAID SAS-3 3008** at PCI `01:00.0`, the RAID +> card in the PCIe slot that no document had recorded, and the chipset's +> `ata1`–`ata5` read *SATA link down* at boot. The board's AHCI carries the +> boot SSD alone. The cabling instruction below is kept as written, because +> it is what was believed when the drives went in; what is actually cabled is +> in [`hardware.md`](../hardware.md)'s `smaug` entry, and the consequence — +> a controller between ZFS and its disks, and what that means for the swap — +> is in the disk runbook's step 4 and open list. Power down, unplug, hold the power button five seconds, ground yourself. diff --git a/docs/runbooks/replace-the-nas-disk.md b/docs/runbooks/replace-the-nas-disk.md index 2f9b51d6..091f204b 100644 --- a/docs/runbooks/replace-the-nas-disk.md +++ b/docs/runbooks/replace-the-nas-disk.md @@ -3,8 +3,9 @@ **One faulted Exos, two trays and no spare, and a dataset with no copy anywhere else — so the copy comes first and the tray comes last.** -> **Status — 2026-09-20: nothing below is done. The pool runs on one disk, -> and `erebor/apps` exists only on it.** +> **Status — 2026-09-20: step 1 is read and it is the drive; steps 2–6 are +> not done. The pool runs on one disk, `erebor/apps` exists only on it, and +> since the exporter came back nothing is paging for either.** > > | When (PDT, 2026-09-19) | What | > | --- | --- | @@ -25,6 +26,34 @@ anywhere else — so the copy comes first and the tray comes last.** > refused, and §6.2 is marked *Not yet done*. That is step 2, and it is why > the tray waits. > [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries this. +> +> **Step 1 read at the console, 2026-09-19 23:19 PDT. It is the drive, and +> the return is the answer.** `zpool status -v erebor`: pool `ONLINE`, +> `mirror-0` `ONLINE`, the `24c4970d…` leaf `FAULTED` with **3 read, 99 +> write, 0 checksum** errors, "too many errors", `errors: No known data +> errors`. `dmesg` from 20:47:12 onward is one shape only: `Sense Key: Not +> Ready`, *Logical unit not ready, cause not reportable*, commands timing +> out at 60 s and aborted, a target reset that succeeded and changed +> nothing, reads and writes failing at sector 0, at 2080, and at the far end +> of the disk alike — the drive going away, not the path. No link resets, no +> `SError`, no CRC. SMART at lifetime hour 32: overall `PASSED`, **850 +> pending and 850 offline-uncorrectable sectors** where both read 0 on +> 2026-09-18, `Command_Timeout` normalised to **1**, no reallocations, error +> log empty, the extended self-test still logged as completed clean at hour +> 26. FARM says which head: **all 850 reallocation candidates are on head +> 5**, 124 command timeouts, 179 hardware resets, 12 V and 5 V rails inside +> spec, 28 °C. The fault was at about **lifetime hour 27**, one hour after +> the self-test that passed. +> +> **The exporter came back on `docker restart media-node-exporter`** at +> 23:26 PDT (06:26 UTC 2026-09-20): `/metrics` in 46 ms, every collector +> reporting success, `up` back to 1 on the next scrape, `InstanceDown` +> resolved. **And that is the problem.** The kstat behind +> `node_zfs_zpool_state` reads `online` for `erebor` — the pool state, which +> `zpool status` also prints as `ONLINE` — so `ZpoolNotOnline` sees nothing, +> and from 23:27 PDT **no alert in the estate is firing for a mirror running +> on one disk.** The only thing that noticed is TrueNAS's own alert, which +> reaches the web UI and nothing else. See *What is still open*. `smaug` is the TrueNAS host at `10.0.40.30` on CasaBonita, which is terminal outward ([ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)): @@ -191,7 +220,15 @@ leads, so a cable that was the fault is found by the next reading rather than hidden by a fresh one. Leave the `AUX1_FAN` cage fan alone; it is the airflow over both trays and §1 says why it is not optional. -Power on. **Read the new drive before trusting it**, §2-style: +Power on, and **check that the new drive is there at all**: `lsblk` must +show a third 18 TB device. The bays are behind a MegaRAID SAS3008, not the +chipset (see *What is still open*), and a RAID card that is not in JBOD +mode holds a fresh disk as *Unconfigured Good* and shows the operating +system nothing. If `lsblk` has no new device, the card's own boot-time +utility (`Ctrl-R` during POST on a MegaRAID) is where the disk is made a +JBOD, and that reading — the card's firmware, its mode — goes in +[`hardware.md`](../hardware.md) the same evening. Then **read the new drive +before trusting it**, §2-style: ```bash smartctl -a /dev/sdX @@ -251,16 +288,46 @@ refund fight — is the operator's, and it is recorded here. ## What is still open -- **Whether `zpool status` says `DEGRADED` or `ONLINE` for a faulted mirror - leaf.** TrueNAS's alert said `ONLINE`. Step 1 settles it, and it decides - whether `ZpoolNotOnline` can see this class of fault at all. -- **No vdev-level metric.** node_exporter exports pool state and nothing per - device; [#483](https://github.com/Gerrrt/HomeLab/issues/483) is why - nothing on `smaug` can push more. A `FAULTED` leaf under an `ONLINE` pool - is invisible from here until that changes. -- **Whether the exporter hangs on every device fault.** If step 1's restart - does not bring it back, then `InstanceDown` is the NAS disk alert in - practice and this runbook should say so at the top. +- **Settled 2026-09-19: `zpool status` says `ONLINE` for a faulted mirror + leaf, and so does the kstat.** `ZpoolNotOnline` cannot see this class of + fault. It still catches a pool that is genuinely degraded, suspended or + unavailable — a resilver, a second disk gone — but the one fault that has + actually happened is below its resolution. +- **No vdev-level metric, and now nothing fires.** node_exporter exports + pool state and nothing per device; [#483](https://github.com/Gerrrt/HomeLab/issues/483) + is why nothing on `smaug` can push more. With the exporter restarted, + `InstanceDown` has resolved and no alert covers the degraded mirror. Two + ways out, neither built: point TrueNAS's own alert service at a channel + that reaches a phone (System → Alert Settings → Add; the estate's ntfy + receiver is the obvious candidate if the type list offers it), or have a + periodic task on `smaug` write `zpool status` vdev states to a textfile + the exporter serves — the mechanism #483 argues about. +- **The exporter's hang was the fault, not a habit.** It came back on a + restart within seven minutes of the console session, so `InstanceDown` is + the NAS disk alert only while the device is still timing out I/O. The + runbook now says so at the top. +- **Settled 2026-09-19: the pair is behind a RAID card.** `lspci -nn` reads + a **Broadcom / LSI MegaRAID SAS-3 3008 "Fury"**, `1000:005f`, at + `01:00.0`, and both `sda` and `sdb` resolve under its `host0`; the + chipset AHCI carries the boot SSD alone. [`build-the-nas.md`](build-the-nas.md) + §1's `SATA2`/`SATA3` was wrong and now says so; + [`hardware.md`](../hardware.md) carries the card. **Read the same + night:** driver `megaraid_sas`, controller type `iMR(0MB)` — cacheless, + no battery — subsystem `9340`, and `JBOD sequence map : enabled`, so the + disks are **JBOD pass-through**, not virtual drives, which is why + `smartctl` reaches them by their own model without `-d megaraid`. That is + the better of the two answers a MegaRAID offers and still not the IT-mode + HBA ZFS is designed for: the sixty-second timeouts, task aborts and the + 21:03:51 controller reset in the fault's `dmesg` are its firmware's error + path, and ZFS waited on them. The firmware version is the one reading + still owed, and it is not in sysfs — `/sys/class/scsi_host/host0/fw_ver` + does not exist, and TrueNAS ships no `storcli` — so it is read off the + card's POST banner or its *Ctrl-R* controller properties, at step 4, + when the machine is at POST anyway. Whether to leave the + card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the + bays to the chipset's free `SATA0`–`SATA3` and take the card out is a + decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after + the swap, not before it. - **§6.2**, until step 2 path A has run — and `oracle`'s copy of the NAS set, which `backup-nas.sh` makes and nothing has yet made. - **The replacement decision**, and whether a third drive follows. diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index 010b0222..6546ebe0 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -427,6 +427,17 @@ groups: # dead, InstanceDown is the signal, and the runbook's first step is at # the console either way. # + # AND THE RESOLUTION IT HAS. Read at the console on 2026-09-19 23:19 + # PDT with the exporter restarted: `zpool status` printed the pool + # ONLINE, mirror-0 ONLINE and one leaf FAULTED (3 read, 99 write + # errors), and this kstat read online=1 for erebor. The kstat is the + # POOL state, and a two-disk mirror with one faulted leaf is, to the + # pool, online. So this rule cannot see the fault that produced it; it + # sees a pool that is degraded, suspended or unavailable as a whole — + # a resilver in progress, or the second disk gone. A per-vdev signal + # is #558's open line, and until it exists a one-disk mirror pages + # nothing here once the exporter is back. + # # `state!="online"` == 1 rather than `state="online"` == 0, so the label # carries WHICH state the pool is in and a state name the collector # learns later is caught without editing this line. boot-pool is covered