From c70d93753c16b146654e201282a80eaa84e47180 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 06:35:27 +0000 Subject: [PATCH 1/4] =?UTF-8?q?docs(nas):=20the=20console=20readings=20?= =?UTF-8?q?=E2=80=94=20it=20is=20the=20drive,=20head=205,=20and=20the=20po?= =?UTF-8?q?ol-state=20rule=20cannot=20see=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 1 of replace-the-nas-disk.md read at the console on 2026-09-19 23:19 PDT: zpool status counts 3 read and 99 write errors on the leaf; dmesg is "Logical unit not ready" and 60-second timeouts from 20:47 with no link resets and no CRC; SMART reads 850 pending and 850 offline-uncorrectable sectors against 0 and 0 the day before, Command_Timeout normalised to 1; FARM puts all 850 on head 5. The drive, not the path — the return is the answer. The exporter came back on a restart, InstanceDown resolved, and the kstat behind node_zfs_zpool_state reads online for erebor with a FAULTED leaf: ZpoolNotOnline (#559) cannot see the fault that produced it, and nothing in the estate now pages for a mirror on one disk. The rule's comment says so; the runbook's open list carries the two ways out and one thing the readings do not match the docs on — sdb enumerates under a SCSI host on the PCIe slot, not the board's SATA2/SATA3. Refs #558, #483. Co-Authored-By: Claude Fable 5.1 --- docs/hardware.md | 11 ++- docs/runbooks/replace-the-nas-disk.md | 70 +++++++++++++++---- .../prometheus/rules/host.rules.yaml | 11 +++ 3 files changed, 79 insertions(+), 13 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index c75a27ad..5fd314b5 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -318,7 +318,16 @@ revisions of this repository treated `shiva` as the hypervisor itself. said at the console, and the outcome, belong in [`replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)'s status block and then here; [#558](https://github.com/Gerrrt/HomeLab/issues/558) - carries it. + carries it. **Read at the console on 2026-09-19 at 23:19 PDT, at lifetime + hour 32: it is the drive.** `zpool status` counts 3 read and 99 write + errors on the leaf; `dmesg` is *Logical unit not ready* and 60-second + command timeouts from 20:47 on, with no link resets and no CRC errors; + SMART reads **850 pending and 850 offline-uncorrectable sectors** against + the 0 and 0 of the day before, `Command_Timeout` normalised to 1, overall + health still `PASSED`; and FARM puts **all 850 reallocation candidates on + head 5**, with 124 command timeouts and 179 hardware resets, both rails + in spec, 28 °C. `ZVTBS4NL` is unaffected. The extended self-test that + completed at hour 26 was true when it was read and is not evidence now. - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot disk, carrying TrueNAS and the media stack it launches diff --git a/docs/runbooks/replace-the-nas-disk.md b/docs/runbooks/replace-the-nas-disk.md index 2f9b51d6..2d49e1cd 100644 --- a/docs/runbooks/replace-the-nas-disk.md +++ b/docs/runbooks/replace-the-nas-disk.md @@ -3,8 +3,9 @@ **One faulted Exos, two trays and no spare, and a dataset with no copy anywhere else — so the copy comes first and the tray comes last.** -> **Status — 2026-09-20: nothing below is done. The pool runs on one disk, -> and `erebor/apps` exists only on it.** +> **Status — 2026-09-20: step 1 is read and it is the drive; steps 2–6 are +> not done. The pool runs on one disk, `erebor/apps` exists only on it, and +> since the exporter came back nothing is paging for either.** > > | When (PDT, 2026-09-19) | What | > | --- | --- | @@ -25,6 +26,34 @@ anywhere else — so the copy comes first and the tray comes last.** > refused, and §6.2 is marked *Not yet done*. That is step 2, and it is why > the tray waits. > [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries this. +> +> **Step 1 read at the console, 2026-09-19 23:19 PDT. It is the drive, and +> the return is the answer.** `zpool status -v erebor`: pool `ONLINE`, +> `mirror-0` `ONLINE`, the `24c4970d…` leaf `FAULTED` with **3 read, 99 +> write, 0 checksum** errors, "too many errors", `errors: No known data +> errors`. `dmesg` from 20:47:12 onward is one shape only: `Sense Key: Not +> Ready`, *Logical unit not ready, cause not reportable*, commands timing +> out at 60 s and aborted, a target reset that succeeded and changed +> nothing, reads and writes failing at sector 0, at 2080, and at the far end +> of the disk alike — the drive going away, not the path. No link resets, no +> `SError`, no CRC. SMART at lifetime hour 32: overall `PASSED`, **850 +> pending and 850 offline-uncorrectable sectors** where both read 0 on +> 2026-09-18, `Command_Timeout` normalised to **1**, no reallocations, error +> log empty, the extended self-test still logged as completed clean at hour +> 26. FARM says which head: **all 850 reallocation candidates are on head +> 5**, 124 command timeouts, 179 hardware resets, 12 V and 5 V rails inside +> spec, 28 °C. The fault was at about **lifetime hour 27**, one hour after +> the self-test that passed. +> +> **The exporter came back on `docker restart media-node-exporter`** at +> 23:26 PDT (06:26 UTC 2026-09-20): `/metrics` in 46 ms, every collector +> reporting success, `up` back to 1 on the next scrape, `InstanceDown` +> resolved. **And that is the problem.** The kstat behind +> `node_zfs_zpool_state` reads `online` for `erebor` — the pool state, which +> `zpool status` also prints as `ONLINE` — so `ZpoolNotOnline` sees nothing, +> and from 23:27 PDT **no alert in the estate is firing for a mirror running +> on one disk.** The only thing that noticed is TrueNAS's own alert, which +> reaches the web UI and nothing else. See *What is still open*. `smaug` is the TrueNAS host at `10.0.40.30` on CasaBonita, which is terminal outward ([ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)): @@ -251,16 +280,33 @@ refund fight — is the operator's, and it is recorded here. ## What is still open -- **Whether `zpool status` says `DEGRADED` or `ONLINE` for a faulted mirror - leaf.** TrueNAS's alert said `ONLINE`. Step 1 settles it, and it decides - whether `ZpoolNotOnline` can see this class of fault at all. -- **No vdev-level metric.** node_exporter exports pool state and nothing per - device; [#483](https://github.com/Gerrrt/HomeLab/issues/483) is why - nothing on `smaug` can push more. A `FAULTED` leaf under an `ONLINE` pool - is invisible from here until that changes. -- **Whether the exporter hangs on every device fault.** If step 1's restart - does not bring it back, then `InstanceDown` is the NAS disk alert in - practice and this runbook should say so at the top. +- **Settled 2026-09-19: `zpool status` says `ONLINE` for a faulted mirror + leaf, and so does the kstat.** `ZpoolNotOnline` cannot see this class of + fault. It still catches a pool that is genuinely degraded, suspended or + unavailable — a resilver, a second disk gone — but the one fault that has + actually happened is below its resolution. +- **No vdev-level metric, and now nothing fires.** node_exporter exports + pool state and nothing per device; [#483](https://github.com/Gerrrt/HomeLab/issues/483) + is why nothing on `smaug` can push more. With the exporter restarted, + `InstanceDown` has resolved and no alert covers the degraded mirror. Two + ways out, neither built: point TrueNAS's own alert service at a channel + that reaches a phone (System → Alert Settings → Add; the estate's ntfy + receiver is the obvious candidate if the type list offers it), or have a + periodic task on `smaug` write `zpool status` vdev states to a textfile + the exporter serves — the mechanism #483 argues about. +- **The exporter's hang was the fault, not a habit.** It came back on a + restart within seven minutes of the console session, so `InstanceDown` is + the NAS disk alert only while the device is still timing out I/O. The + runbook now says so at the top. +- **Which controller the pair is on.** `readlink /sys/block/sdb` resolves + under `host0` at PCI `0000:01:00.0`, and `dmesg` handles it with task + aborts and target resets — a SCSI host on the PCIe slot, not the board's + AHCI, whose `ata2` and `ata3` read *SATA link down* at boot while + [`build-the-nas.md`](build-the-nas.md) §1 says the trays are cabled to + `SATA2` and `SATA3`. Either there is a controller in the slot that + [`hardware.md`](../hardware.md) does not list, or §1's port names are + wrong. `lspci -nn` and `readlink /sys/block/sda` at the console settle + it, and the answer belongs in the hardware entry. - **§6.2**, until step 2 path A has run — and `oracle`'s copy of the NAS set, which `backup-nas.sh` makes and nothing has yet made. - **The replacement decision**, and whether a third drive follows. diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index 010b0222..6546ebe0 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -427,6 +427,17 @@ groups: # dead, InstanceDown is the signal, and the runbook's first step is at # the console either way. # + # AND THE RESOLUTION IT HAS. Read at the console on 2026-09-19 23:19 + # PDT with the exporter restarted: `zpool status` printed the pool + # ONLINE, mirror-0 ONLINE and one leaf FAULTED (3 read, 99 write + # errors), and this kstat read online=1 for erebor. The kstat is the + # POOL state, and a two-disk mirror with one faulted leaf is, to the + # pool, online. So this rule cannot see the fault that produced it; it + # sees a pool that is degraded, suspended or unavailable as a whole — + # a resilver in progress, or the second disk gone. A per-vdev signal + # is #558's open line, and until it exists a one-disk mirror pages + # nothing here once the exporter is back. + # # `state!="online"` == 1 rather than `state="online"` == 0, so the label # carries WHICH state the pool is in and a state name the collector # learns later is caught without editing this line. boot-pool is covered From 0bacb7c815c70f62c76b5415345c720c8d2efb7b Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 06:52:08 +0000 Subject: [PATCH 2/4] docs(nas): the Exos pair is behind a MegaRAID SAS3008 the documents never recorded, not the board's SATA2 and SATA3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lspci at the console on 2026-09-19: a Broadcom / LSI MegaRAID SAS-3 3008 "Fury" (1000:005f, the SAS3008's MegaRAID personality) at 01:00.0, with both sda and sdb under its host0 and the chipset's ata1-ata5 reading link down at boot. The AHCI setting hardware.md credits with keeping a controller out from between ZFS and its disks protects the boot SSD and nothing else, and build-the-nas.md §1's cabling was what was believed, not what was cabled. Both say so now; the card has an entry; the disk runbook's step 4 checks that a replacement appears in lsblk at all, because a RAID card not in JBOD mode holds a fresh disk as Unconfigured Good. Firmware and mode are the two readings still owed. Refs #558. Co-Authored-By: Claude Fable 5.1 --- docs/hardware.md | 22 +++++++++++++++- docs/runbooks/build-the-nas.md | 12 +++++++++ docs/runbooks/replace-the-nas-disk.md | 36 +++++++++++++++++++-------- 3 files changed, 59 insertions(+), 11 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index 5fd314b5..4bbfe0f9 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -261,7 +261,27 @@ revisions of this repository treated `shiva` as the hypervisor itself. [#418](https://github.com/Gerrrt/HomeLab/issues/418) is the cautionary tale for, and `CSM [Disabled]`, so it boots UEFI as TrueNAS wants. Two 3.5" trays, filled by the Exos pair on 2026-09-18 — exactly the mirror - and no spare. + and no spare. **The trays are not on those six ports.** Read with `lspci` + and `readlink` at the console on 2026-09-19, while triaging the faulted + disk: both Exos enumerate under `host0` at PCI `01:00.0`, a **Broadcom / + LSI MegaRAID SAS-3 3008 "Fury"**, PCI ID `1000:005f` — the SAS3008 in its + MegaRAID personality, which is the ThinkServer RAID 520i option for this + chassis, sitting in the PCIe slot and cabled to the bays. The chipset AHCI + at `00:17.0` carries only the boot SSD on `ata6`; `ata1`–`ata5` read *SATA + link down* at boot. So `Configure SATA as [AHCI]` protects the boot disk + and nothing else, and the pool has had a RAID controller between ZFS and + its disks since the day it was built — the arrangement the line above + calls #418's cautionary tale. It is the MegaRAID firmware, not the + chipset, that handled `sdb`'s failure with 60-second command timeouts, + task aborts and a target reset. `smartctl` reaches the drives without a + `-d megaraid` option and reports them by their own model and serial, which + is what a JBOD pass-through looks like; whether the card is in JBOD mode + or presenting two single-disk virtual drives, and which firmware it runs, + are unread and belong here when they are. `1000:005f` is the ID to watch: + `1000:0097` is the same silicon in IT mode, and the card was never + recorded here, like the optical drive was not. Cabling below, in + [`build-the-nas.md`](runbooks/build-the-nas.md) §1, says `SATA2` and + `SATA3`; that is now known to be wrong, and the same reading corrects it. **The 5.25" bay was not empty**: a PLDS `DVD-RW DU8AESH` answered on SATA5. A photograph of the open case had been read here as an empty cage and was wrong; the BIOS summary is what caught it. The optical drive came out on diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 475186d0..90354695 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -309,6 +309,18 @@ anything in this runbook. > alone. **The power lead had four wires and no orange one**, so the pin-3 > trap below did not fire on this supply; both drives spun up and appeared in > `lsblk` first time. +> +> **Corrected 2026-09-19: the trays are not on `SATA2` and `SATA3`.** Read +> with `lspci` and `readlink /sys/block/sd?` while triaging the faulted disk +> ([`replace-the-nas-disk.md`](replace-the-nas-disk.md)): both Exos sit +> behind a **Broadcom / LSI MegaRAID SAS-3 3008** at PCI `01:00.0`, the RAID +> card in the PCIe slot that no document had recorded, and the chipset's +> `ata1`–`ata5` read *SATA link down* at boot. The board's AHCI carries the +> boot SSD alone. The cabling instruction below is kept as written, because +> it is what was believed when the drives went in; what is actually cabled is +> in [`hardware.md`](../hardware.md)'s `smaug` entry, and the consequence — +> a controller between ZFS and its disks, and what that means for the swap — +> is in the disk runbook's step 4 and open list. Power down, unplug, hold the power button five seconds, ground yourself. diff --git a/docs/runbooks/replace-the-nas-disk.md b/docs/runbooks/replace-the-nas-disk.md index 2d49e1cd..573c9614 100644 --- a/docs/runbooks/replace-the-nas-disk.md +++ b/docs/runbooks/replace-the-nas-disk.md @@ -220,7 +220,15 @@ leads, so a cable that was the fault is found by the next reading rather than hidden by a fresh one. Leave the `AUX1_FAN` cage fan alone; it is the airflow over both trays and §1 says why it is not optional. -Power on. **Read the new drive before trusting it**, §2-style: +Power on, and **check that the new drive is there at all**: `lsblk` must +show a third 18 TB device. The bays are behind a MegaRAID SAS3008, not the +chipset (see *What is still open*), and a RAID card that is not in JBOD +mode holds a fresh disk as *Unconfigured Good* and shows the operating +system nothing. If `lsblk` has no new device, the card's own boot-time +utility (`Ctrl-R` during POST on a MegaRAID) is where the disk is made a +JBOD, and that reading — the card's firmware, its mode — goes in +[`hardware.md`](../hardware.md) the same evening. Then **read the new drive +before trusting it**, §2-style: ```bash smartctl -a /dev/sdX @@ -298,15 +306,23 @@ refund fight — is the operator's, and it is recorded here. restart within seven minutes of the console session, so `InstanceDown` is the NAS disk alert only while the device is still timing out I/O. The runbook now says so at the top. -- **Which controller the pair is on.** `readlink /sys/block/sdb` resolves - under `host0` at PCI `0000:01:00.0`, and `dmesg` handles it with task - aborts and target resets — a SCSI host on the PCIe slot, not the board's - AHCI, whose `ata2` and `ata3` read *SATA link down* at boot while - [`build-the-nas.md`](build-the-nas.md) §1 says the trays are cabled to - `SATA2` and `SATA3`. Either there is a controller in the slot that - [`hardware.md`](../hardware.md) does not list, or §1's port names are - wrong. `lspci -nn` and `readlink /sys/block/sda` at the console settle - it, and the answer belongs in the hardware entry. +- **Settled 2026-09-19: the pair is behind a RAID card.** `lspci -nn` reads + a **Broadcom / LSI MegaRAID SAS-3 3008 "Fury"**, `1000:005f`, at + `01:00.0`, and both `sda` and `sdb` resolve under its `host0`; the + chipset AHCI carries the boot SSD alone. [`build-the-nas.md`](build-the-nas.md) + §1's `SATA2`/`SATA3` was wrong and now says so; + [`hardware.md`](../hardware.md) carries the card. What is **not** read + yet, and belongs in the hardware entry: the card's firmware and whether + the disks are JBOD pass-through or single-disk virtual drives. + `cat /sys/class/scsi_host/host0/proc_name` names the driver, and + `dmesg -T | grep -iE 'megaraid|megasas'` prints the firmware and, on a + JBOD, says so. `smartctl` reaching the drives by their own model without + `-d megaraid` says pass-through, which is the better of the two answers + and still not the IT-mode HBA ZFS is designed for. Whether to leave the + card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the + bays to the chipset's free `SATA0`–`SATA3` and take the card out is a + decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after + the swap, not before it. - **§6.2**, until step 2 path A has run — and `oracle`'s copy of the NAS set, which `backup-nas.sh` makes and nothing has yet made. - **The replacement decision**, and whether a third drive follows. From 2b765f9fe5ab2152c942092dabacc47e65b95341 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 07:03:48 +0000 Subject: [PATCH 3/4] docs(nas): the MegaRAID is a cacheless iMR in JBOD mode, read from the driver's boot log MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dmesg at the console: megaraid_sas 07.727.03.00-rc1, controller type iMR(0MB), subsystem 1000:9340, Secure JBOD supported, JBOD sequence map enabled — the disks are JBOD pass-through, not virtual drives, which is why smartctl reaches them by their own model. No write cache and no battery. The driver logged an interrupt disable/enable at 21:03:51 on 2026-09-19, the same second as the target reset in the fault's dmesg: the controller resetting itself around a disk that had stopped answering. The firmware version is the one reading still owed, from /sys/class/scsi_host/host0/fw_ver. Refs #558. Co-Authored-By: Claude Fable 5.1 --- docs/hardware.md | 18 +++++++++++++++--- docs/runbooks/replace-the-nas-disk.md | 18 ++++++++++-------- 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index 4bbfe0f9..4bf550b7 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -275,9 +275,21 @@ revisions of this repository treated `shiva` as the hypervisor itself. chipset, that handled `sdb`'s failure with 60-second command timeouts, task aborts and a target reset. `smartctl` reaches the drives without a `-d megaraid` option and reports them by their own model and serial, which - is what a JBOD pass-through looks like; whether the card is in JBOD mode - or presenting two single-disk virtual drives, and which firmware it runs, - are unread and belong here when they are. `1000:005f` is the ID to watch: + is what a JBOD pass-through looks like — and `dmesg` confirms it, read the + same night: driver `megaraid_sas` 07.727.03.00-rc1, controller type + **`iMR(0MB)`** — the cacheless entry-level MegaRAID, no write cache and + no battery to worry about — subsystem `1000:9340`, which is the 9340-8i + family the ThinkServer RAID 520i is built on, *Secure JBOD support: Yes*, + and **`JBOD sequence map : enabled`**, which is the driver's way of + saying the disks are JBOD devices rather than virtual drives. So ZFS sees + the drives themselves through a RAID firmware's error handling, which is + the better of the two arrangements a MegaRAID offers and still not an + IT-mode HBA. The firmware version is not in `dmesg`; it is in + `/sys/class/scsi_host/host0/fw_ver` and is still owed here. The driver + logged a disable/enable of its interrupts at 21:03:51 on 2026-09-19, the + same second as the target reset in the fault's `dmesg` — the controller + resetting itself around a disk that had stopped answering, which is the + Online Controller Reset it advertises as enabled. `1000:005f` is the ID to watch: `1000:0097` is the same silicon in IT mode, and the card was never recorded here, like the optical drive was not. Cabling below, in [`build-the-nas.md`](runbooks/build-the-nas.md) §1, says `SATA2` and diff --git a/docs/runbooks/replace-the-nas-disk.md b/docs/runbooks/replace-the-nas-disk.md index 573c9614..39b14c7b 100644 --- a/docs/runbooks/replace-the-nas-disk.md +++ b/docs/runbooks/replace-the-nas-disk.md @@ -311,14 +311,16 @@ refund fight — is the operator's, and it is recorded here. `01:00.0`, and both `sda` and `sdb` resolve under its `host0`; the chipset AHCI carries the boot SSD alone. [`build-the-nas.md`](build-the-nas.md) §1's `SATA2`/`SATA3` was wrong and now says so; - [`hardware.md`](../hardware.md) carries the card. What is **not** read - yet, and belongs in the hardware entry: the card's firmware and whether - the disks are JBOD pass-through or single-disk virtual drives. - `cat /sys/class/scsi_host/host0/proc_name` names the driver, and - `dmesg -T | grep -iE 'megaraid|megasas'` prints the firmware and, on a - JBOD, says so. `smartctl` reaching the drives by their own model without - `-d megaraid` says pass-through, which is the better of the two answers - and still not the IT-mode HBA ZFS is designed for. Whether to leave the + [`hardware.md`](../hardware.md) carries the card. **Read the same + night:** driver `megaraid_sas`, controller type `iMR(0MB)` — cacheless, + no battery — subsystem `9340`, and `JBOD sequence map : enabled`, so the + disks are **JBOD pass-through**, not virtual drives, which is why + `smartctl` reaches them by their own model without `-d megaraid`. That is + the better of the two answers a MegaRAID offers and still not the IT-mode + HBA ZFS is designed for: the sixty-second timeouts, task aborts and the + 21:03:51 controller reset in the fault's `dmesg` are its firmware's error + path, and ZFS waited on them. The firmware version is the one reading + still owed, from `/sys/class/scsi_host/host0/fw_ver`. Whether to leave the card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the bays to the chipset's free `SATA0`–`SATA3` and take the card out is a decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after From c4ce1cf33f22ebd894efebf9a3ec5cdf435388c7 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 07:13:38 +0000 Subject: [PATCH 4/4] =?UTF-8?q?docs(nas):=20the=20MegaRAID's=20firmware=20?= =?UTF-8?q?version=20is=20not=20in=20sysfs=20=E2=80=94=20read=20it=20off?= =?UTF-8?q?=20the=20POST=20banner=20at=20the=20swap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /sys/class/scsi_host/host0/fw_ver does not exist, read at the console on 2026-09-19; megaraid_sas exposes crash-dump and queue attributes there and nothing about its firmware, and TrueNAS ships no storcli. The version comes off the card's POST banner or its Ctrl-R controller properties, so the hardware entry and the runbook now say it is owed from step 4, when the machine is at POST for the swap anyway. Refs #558. Co-Authored-By: Claude Fable 5.1 --- docs/hardware.md | 9 +++++++-- docs/runbooks/replace-the-nas-disk.md | 5 ++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index 4bf550b7..7a608f1e 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -284,8 +284,13 @@ revisions of this repository treated `shiva` as the hypervisor itself. saying the disks are JBOD devices rather than virtual drives. So ZFS sees the drives themselves through a RAID firmware's error handling, which is the better of the two arrangements a MegaRAID offers and still not an - IT-mode HBA. The firmware version is not in `dmesg`; it is in - `/sys/class/scsi_host/host0/fw_ver` and is still owed here. The driver + IT-mode HBA. The firmware version is not in `dmesg` and not in sysfs + either — `/sys/class/scsi_host/host0/fw_ver` does not exist, read + 2026-09-19, and `megaraid_sas` exposes crash-dump and queue attributes + there and nothing about its firmware. TrueNAS ships no `storcli`. So the + version is read off the card's own POST banner, or from *Ctrl-R* → + controller properties during boot, and it is owed here from the next + time the machine is at POST — which the swap will be. The driver logged a disable/enable of its interrupts at 21:03:51 on 2026-09-19, the same second as the target reset in the fault's `dmesg` — the controller resetting itself around a disk that had stopped answering, which is the diff --git a/docs/runbooks/replace-the-nas-disk.md b/docs/runbooks/replace-the-nas-disk.md index 39b14c7b..091f204b 100644 --- a/docs/runbooks/replace-the-nas-disk.md +++ b/docs/runbooks/replace-the-nas-disk.md @@ -320,7 +320,10 @@ refund fight — is the operator's, and it is recorded here. HBA ZFS is designed for: the sixty-second timeouts, task aborts and the 21:03:51 controller reset in the fault's `dmesg` are its firmware's error path, and ZFS waited on them. The firmware version is the one reading - still owed, from `/sys/class/scsi_host/host0/fw_ver`. Whether to leave the + still owed, and it is not in sysfs — `/sys/class/scsi_host/host0/fw_ver` + does not exist, and TrueNAS ships no `storcli` — so it is read off the + card's POST banner or its *Ctrl-R* controller properties, at step 4, + when the machine is at POST anyway. Whether to leave the card as it is, flash the 3008 to IT firmware (`1000:0097`), or cable the bays to the chipset's free `SATA0`–`SATA3` and take the card out is a decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after