docs(nas): the console readings — it is the drive, head 5, and the pool-state rule cannot see it - #565
Merged
Conversation
…ol-state rule cannot see it Step 1 of replace-the-nas-disk.md read at the console on 2026-09-19 23:19 PDT: zpool status counts 3 read and 99 write errors on the leaf; dmesg is "Logical unit not ready" and 60-second timeouts from 20:47 with no link resets and no CRC; SMART reads 850 pending and 850 offline-uncorrectable sectors against 0 and 0 the day before, Command_Timeout normalised to 1; FARM puts all 850 on head 5. The drive, not the path — the return is the answer. The exporter came back on a restart, InstanceDown resolved, and the kstat behind node_zfs_zpool_state reads online for erebor with a FAULTED leaf: ZpoolNotOnline (#559) cannot see the fault that produced it, and nothing in the estate now pages for a mirror on one disk. The rule's comment says so; the runbook's open list carries the two ways out and one thing the readings do not match the docs on — sdb enumerates under a SCSI host on the PCIe slot, not the board's SATA2/SATA3. Refs #558, #483. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
10 tasks
…ever recorded, not the board's SATA2 and SATA3 lspci at the console on 2026-09-19: a Broadcom / LSI MegaRAID SAS-3 3008 "Fury" (1000:005f, the SAS3008's MegaRAID personality) at 01:00.0, with both sda and sdb under its host0 and the chipset's ata1-ata5 reading link down at boot. The AHCI setting hardware.md credits with keeping a controller out from between ZFS and its disks protects the boot SSD and nothing else, and build-the-nas.md §1's cabling was what was believed, not what was cabled. Both say so now; the card has an entry; the disk runbook's step 4 checks that a replacement appears in lsblk at all, because a RAID card not in JBOD mode holds a fresh disk as Unconfigured Good. Firmware and mode are the two readings still owed. Refs #558. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e driver's boot log dmesg at the console: megaraid_sas 07.727.03.00-rc1, controller type iMR(0MB), subsystem 1000:9340, Secure JBOD supported, JBOD sequence map enabled — the disks are JBOD pass-through, not virtual drives, which is why smartctl reaches them by their own model. No write cache and no battery. The driver logged an interrupt disable/enable at 21:03:51 on 2026-09-19, the same second as the target reset in the fault's dmesg: the controller resetting itself around a disk that had stopped answering. The firmware version is the one reading still owed, from /sys/class/scsi_host/host0/fw_ver. Refs #558. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…off the POST banner at the swap /sys/class/scsi_host/host0/fw_ver does not exist, read at the console on 2026-09-19; megaraid_sas exposes crash-dump and queue attributes there and nothing about its firmware, and TrueNAS ships no storcli. The version comes off the card's POST banner or its Ctrl-R controller properties, so the hardware entry and the runbook now say it is owed from step 4, when the machine is at POST for the swap anyway. Refs #558. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of
replace-the-nas-disk.md, read at the console on 2026-09-19 at 23:19 PDT, recorded where the runbook says readings go.What the console said
zpool statuscounts 3 read / 99 write errors on the leaf;dmesgfrom 20:47 is Logical unit not ready and 60 s command timeouts, task aborts, a target reset that changed nothing, with no link resets and no CRC; SMART reads 850 pending / 850 offline-uncorrectable against 0 / 0 the day before,Command_Timeoutnormalised to 1, health stillPASSED; FARM puts all 850 reallocation candidates on head 5. The eBay return is the answer.docker restart media-node-exporterin 46 ms, andInstanceDownresolved.onlinewith a FAULTED leaf. SoZpoolNotOnline(feat(nas): erebor lost a disk one day in — a rule that reads pool state, the runbook for the swap, and the record of what fired first #559) cannot see this fault, and since 23:27 PDT nothing in the estate pages for a mirror on one disk. The rule's comment now says exactly what it can and cannot see; the runbook's open list carries the two ways out (a TrueNAS alert service to a phone channel, or a textfile metric onsmaug, smaug has no SMART and no patch state, and the mechanism every other host uses is a push it cannot make #483's argument), andereborlostZVTBSDL3at lifetime hour 27, node_exporter hung nine minutes before the fault, anderebor/appshas no copy anywhere else #558 has the line.lspci -nnreads a Broadcom / LSI MegaRAID SAS-3 3008 "Fury" (1000:005f, the SAS3008's MegaRAID personality) at01:00.0, with both Exos under itshost0and the chipset'sata1–ata5link down at boot.hardware.mdgets the card and stops crediting AHCI with keeping a controller out from between ZFS and its disks;build-the-nas.md§1'sSATA2/SATA3is corrected in place; the disk runbook's step 4 now checks that a replacement disk appears inlsblkat all, since a RAID card not in JBOD mode holds a fresh disk as Unconfigured Good. The driver's boot log then settled the mode:megaraid_sas, controller typeiMR(0MB)(cacheless, no battery), subsystem9340, JBOD sequence map enabled — the disks are JBOD pass-through, not virtual drives. The firmware version is the one reading still owed.Files
docs/runbooks/replace-the-nas-disk.md(status block, step 4, open list),docs/hardware.md(smaugand Exos entries),docs/runbooks/build-the-nas.md(§1 correction),stacks/observability/prometheus/rules/host.rules.yaml(comment only; no rule change).Refs #558, #483.
🤖 Generated with Claude Code