Skip to content

docs(nas): the copy off smaug exists, and the two runbooks that said otherwise say so #578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Gerrrt merged 1 commit into from
Sep 20, 2026
+35 −21
Merged

docs(nas): the copy off smaug exists, and the two runbooks that said otherwise say so #578

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/runbooks/build-the-nas.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -937,9 +937,12 @@ as `65534` across two hosts that spell it differently.
> clean, and node_exporter stopped answering nine minutes earlier, so
> `up{job="node",instance="smaug"}` reads 0 as well. The pool runs on
> `ZVTBS4NL` alone. [`replace-the-nas-disk.md`](replace-the-nas-disk.md) is
> the procedure, and §6.2 — still not done — is its step 2, because
> `erebor/apps` has no copy off this host.
> [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries it.
> the procedure. Its step 2 — the copy off this host — was §6.2, and §6.2
> ran on 2026-09-20: set `20260920T060234Z`, `jellyfin.db` present, hashed
> on `oracle`, the timer installed;
> [#484](https://github.com/Gerrrt/HomeLab/issues/484) closed on it. The
> tray can be touched.
> [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries the swap.

- A television on CasaBonita finds Jellyfin and plays something **without** any
firewall rule being involved
Expand Down Expand Up @@ -967,7 +970,8 @@ as `65534` across two hosts that spell it differently.
pool is a mirror of one until it is replaced;
[`replace-the-nas-disk.md`](replace-the-nas-disk.md) is the procedure and
[#558](https://github.com/Gerrrt/HomeLab/issues/558) carries it. Its step
2 is §6.2.
2, the copy off this host, ran on 2026-09-20 (§6.2); what is left is the
return, the replacement and the swap.
- **[#255](https://github.com/Gerrrt/HomeLab/issues/255)**, the residual saying
this host ships no logs, which is true the day it exists.
- **[ADR-0027](../adr/0027-defer-proxmox-backup-server-until-there-is-somewhere-to-send-it.md)'s
Expand Down
44 changes: 27 additions & 17 deletions docs/runbooks/replace-the-nas-disk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# Runbook: Replace a disk in `erebor`

**One faulted Exos, two trays and no spare, and a dataset with no copy
anywhere else — so the copy comes first and the tray comes last.**

> **Status — 2026-09-20: step 1 is read and it is the drive; steps 2–6 are
> not done. The pool runs on one disk, `erebor/apps` exists only on it, and
> since the exporter came back nothing is paging for either.**
**One faulted Exos, two trays and no spare, and a dataset that had no copy
anywhere else — so the copy came first, on 2026-09-20, and the tray comes
last.**

> **Status — 2026-09-20: step 1 is read and it is the drive; step 2 is done
> by path A; steps 3–6 are not. The pool runs on one disk, `erebor/apps`
> has a copy off it since 2026-09-20, and since the exporter came back
> nothing is paging for the pool.**
>
> | When (PDT, 2026-09-19) | What |
> | --- | --- |
Expand All @@ -19,13 +21,19 @@ anywhere else — so the copy comes first and the tray comes last.**
> error at hour 26. The other half of the mirror, `ZVTBS4NL` (`sda`), carries
> everything.
>
> **`erebor/apps` has zero copies off `smaug`.** `scripts/backup-nas.sh`
> merged on 2026-09-19 ([#554](https://github.com/Gerrrt/HomeLab/pull/554)),
> but on the monitoring host `backups/nas` does not exist,
> `homelab-backup-nas.timer` is not installed, `ssh frodo@10.0.40.30` is
> refused, and §6.2 is marked *Not yet done*. That is step 2, and it is why
> the tray waits.
> [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries this.
> **`erebor/apps` had zero copies off `smaug` when this was written.**
> `scripts/backup-nas.sh` had merged on 2026-09-19
> ([#554](https://github.com/Gerrrt/HomeLab/pull/554)), but on the
> monitoring host `backups/nas` did not exist, `homelab-backup-nas.timer`
> was not installed, `ssh frodo@10.0.40.30` was refused, and §6.2 was marked
> *Not yet done*. That was step 2, and it was why the tray waited.
> **Done 2026-09-20, path A:** §6.2 steps 1–8 ran; the first set is
> `20260920T060234Z` — 76 entries, `./data/jellyfin.db` present, Jellyfin
> never stopped — copied to `oracle` and hashed there in the same run,
> `make verify-backups` re-read it, and the timer is installed.
> [#484](https://github.com/Gerrrt/HomeLab/issues/484) closed on it, so the
> tray no longer waits on this.
> [#558](https://github.com/Gerrrt/HomeLab/issues/558) carries the swap.
>
> **Step 1 read at the console, 2026-09-19 23:19 PDT. It is the drive, and
> the return is the answer.** `zpool status -v erebor`: pool `ONLINE`,
Expand Down Expand Up @@ -57,8 +65,8 @@ anywhere else — so the copy comes first and the tray comes last.**

`smaug` is the TrueNAS host at `10.0.40.30` on CasaBonita, which is terminal
outward ([ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)):
the monitoring host reaches `9100` and, once §6.2 is done, `22`, and nothing
else. So the console steps below are done **at the machine** (option 8, *Open
the monitoring host reaches `9100` and, since §6.2 was done on 2026-09-20,
`22`, and nothing else. So the console steps below are done **at the machine** (option 8, *Open
Linux Shell*) or in the TrueNAS UI at `https://10.0.40.30` from a Hicks
workstation, and the monitoring host's part is reading the result. The pool
is a mirror of two Seagate Exos X20 18 TB in the TS150's two 3.5" trays;
Expand Down Expand Up @@ -165,6 +173,10 @@ make backup-nas && make backup-nas ARGS=--list && make verify-backups
and §6.2 step 7, `make install-timers`, once the first pull has passed, so
this copy is the first of a series rather than the only one.

> **Done 2026-09-20 for this fault, by path A.** The set is
> `20260920T060234Z`, on the monitoring host and on `oracle`; the timer's
> next run is Sat 2026-09-26 03:30 UTC.

Path B is for the evening §6.2 cannot be done. A USB stick at the console,
reading from the **newest snapshot** (§4.1) so that Jellyfin never stops and
the SQLite files are captured at one instant:
Expand Down Expand Up @@ -328,6 +340,4 @@ refund fight — is the operator's, and it is recorded here.
bays to the chipset's free `SATA0`–`SATA3` and take the card out is a
decision for [#558](https://github.com/Gerrrt/HomeLab/issues/558) after
the swap, not before it.
- **§6.2**, until step 2 path A has run — and `oracle`'s copy of the NAS
set, which `backup-nas.sh` makes and nothing has yet made.
- **The replacement decision**, and whether a third drive follows.
Loading