From 7a75dcdd56f917408683c7f6171f41dcf8f66a89 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 19 Sep 2026 22:27:18 +0000 Subject: [PATCH 1/7] docs: phoenix is built at 10.0.30.70 on Saruman, the lab has its first off-host client, and Saruman admits it to 8006 (#436) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The build-day commit build-the-jumpbox.md §8 describes, landed as one change because check_docs.py makes the "Not built yet" marker and a network.md row mutually exclusive and ties hardware.md's Alloy agent count to the marker: - network.md gains the phoenix row in the ImaginationLAN table; the planned guest note becomes a description, keeping the sentence that this is the one address on the segment with a path to 8006, which firewall-claims.yaml cannot say; Saruman's guest count goes to two; the WAN note stops saying the jumpbox does not exist and keeps saying the endpoint question is open. - architecture.md drops the marker from the phoenix row and recounts Saruman's guests: two running, seven planned. - hardware.md: five Alloy agents. - observability.md names phoenix as the lab stores' first client from off alexander, and says it too reaches nothing on VLAN 99. - roadmap.md and fit-the-saruman-ssds.md §9 (two guests to move, not one). - build-the-jumpbox.md gains a status block in the shape of the lab guest's. The build date and the §1, §4 and §7 results are placeholders until the runbook has been run; this must not merge before then. Refs #436 Co-Authored-By: Claude Fable 5.1 --- docs/architecture.md | 4 ++-- docs/hardware.md | 2 +- docs/network.md | 22 +++++++++++++--------- docs/observability.md | 7 ++++++- docs/roadmap.md | 2 +- docs/runbooks/build-the-jumpbox.md | 13 ++++++++++++- docs/runbooks/fit-the-saruman-ssds.md | 6 +++++- 7 files changed, 40 insertions(+), 16 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 0520f94e..8ccabc4b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -185,10 +185,10 @@ hole from the monitoring VLAN into the monitored one. | Host | VLAN | Stack | Contents | | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, docker-socket-proxy, Alloy | -| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running one guest — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)) — and eight more planned below: `odin` for the security tooling (ADR-0030), `phoenix` for the deployment toolchain (ADR-0043) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | +| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running two guests — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built 2026-09-DD ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — and seven more planned below: `odin` for the security tooling (ADR-0030) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | -| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | **Not built yet** — the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-DD: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | diff --git a/docs/hardware.md b/docs/hardware.md index dffb9173..68edf739 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -47,7 +47,7 @@ quietly swapped. | `smaug` | Lenovo ThinkServer TS150 | Xeon E3-1225 v6 (4 cores) | 8 GB ECC | 240 GB SATA SSD (boot) + 2× 18 TB ZFS mirror `erebor` | TrueNAS 25.10 | The observability stack runs on a thirteen-year-old MacBook. It handles four -SNMP devices at a 60-second interval, four Alloy agents, and 30 days of metric +SNMP devices at a 60-second interval, five Alloy agents, and 30 days of metric retention without complaint — which is a useful thing to know before spending money on a monitoring host. Its RAM is soldered at 8 GB and it has no built-in Ethernet, so it reaches the network over a USB NIC. diff --git a/docs/network.md b/docs/network.md index 416122df..e0ad6afc 100644 --- a/docs/network.md +++ b/docs/network.md @@ -91,9 +91,10 @@ truth for what a box actually does. address ([`security.md`](security.md#what-this-repository-deliberately-does-not-publish)). Before it, `morpheus` carried no `rdr` and no inbound WAN pass beyond DHCP client replies — the state ADR-0011 measured in 2026-08. **Not built:** the - jumpbox ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) does not exist - and the endpoint question is unanswered, so this describes a decision rather - than a rule on the box. + jumpbox exists — `phoenix`, built 2026-09-DD + ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — but the endpoint + question is unanswered and no `rdr` has been written, so this still + describes a decision rather than a rule on the box. [^modem]: [Xfinity Gateway (XB7)](https://www.xfinity.com/support/articles/broadband-gateways-userguides) [^ProDesk]: [HP ProDesk 600 G4 Mini](https://www.microcenter.com/product/692358/) @@ -348,6 +349,7 @@ Where things get broken on purpose. | shiva | `10.0.30.10` | `94:57:a5:xx:xx:xx` | HPE iLO 4 (DL360 Gen9 BMC)[^Shiva] | iLO 2.82 | Rack U3 | Out-of-band management | | Saruman | `10.0.30.110` | `14:02:ec:xx:xx:xx` | HPE ProLiant DL360 Gen9[^Shiva] | Proxmox VE 9 | Rack U3 | Hypervisor | | alexander | `10.0.30.40` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Lab observability | +| phoenix | `10.0.30.70` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Deployment host | ### Notes @@ -356,8 +358,10 @@ Where things get broken on purpose. dedicated port, `Saruman` is the Proxmox install. They are separate addresses and separate names, and conflating them is a mistake this document previously made. -- `Saruman` runs one guest, `alexander`, built 2026-09-05 - ([#262](https://github.com/Gerrrt/HomeLab/issues/262)). It runs +- `Saruman` runs two guests: `alexander`, built 2026-09-05 + ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built + 2026-09-DD ([#436](https://github.com/Gerrrt/HomeLab/issues/436)), described + below. `alexander` runs [`stacks/lab`](../stacks/lab) — the lab's own Prometheus, Loki, Grafana and Alloy. **It is a guest and not the hypervisor for a reason**: a compose stack is Docker, and Docker would rewrite the iptables of the box whose own @@ -378,13 +382,13 @@ Where things get broken on purpose. pushes to `alexander` and not to Winterfell — "guests get no such rule" covers it — and ADR-0029's six machines report to it as agents. Every path it needs is intra-segment, so it adds no firewall rule. -- A third guest, `phoenix`, is planned at `10.0.30.70` — the next decade — as +- A third guest, `phoenix`, is at `10.0.30.70` — the next decade — as the deployment host: the Proxmox API token, the SSH key and the checkout that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) runs from, placed by - [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md). - **Not built**; it enters the table above when it is - ([`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md)). It runs no stack + [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) + and built 2026-09-DD by + [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). It runs no stack and holds no key that signs anything — the estate's CA stays on `prometheus`, and that ADR says why. It is also where [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s diff --git a/docs/observability.md b/docs/observability.md index b91e02c2..4453c09c 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -11,7 +11,12 @@ ADR-0007's decision — lab telemetry stays in the lab, so that deliberately hostile data never lands in the store the estate is actually run from — and [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) settles its shape. It runs on `alexander`, the guest -[#262](https://github.com/Gerrrt/HomeLab/issues/262) built on 2026-09-05. +[#262](https://github.com/Gerrrt/HomeLab/issues/262) built on 2026-09-05. Its +first client from off that guest is `phoenix`, the deployment host +([ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), +built 2026-09-DD): its native Alloy pushes over the two ports the lab stack +published for it ([#436](https://github.com/Gerrrt/HomeLab/issues/436)), and +it too reaches nothing here. There is a third, and it reports to the second. [`stacks/soc`](../stacks/soc) is Wazuh and Velociraptor on `odin`, another guest on `Saruman` diff --git a/docs/roadmap.md b/docs/roadmap.md index 8d3a5454..acaa0d0a 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1125,7 +1125,7 @@ what left this one unfireable for months. - **[#436](https://github.com/Gerrrt/HomeLab/issues/436) Build a deployment jumpbox on ImaginationLAN, and decide where the CA lives.** Decided by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md); - the guest is not built. The host is `phoenix`, `10.0.30.70`, a guest on + the guest was built 2026-09-DD. The host is `phoenix`, `10.0.30.70`, a guest on `Saruman` that holds the estate's first Proxmox API credential and the SSH key the toolchain will inject into what it builds — the prerequisite for the Packer, OpenTofu and Ansible issues, which today have nowhere to run diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 771eea5f..af9cb01b 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -4,7 +4,7 @@ **Time:** about an hour, most of it the OS installer **You will need:** the Proxmox web UI on `Saruman` (or a shell on it through the KVM or `shiva`), an Ubuntu Server ISO, the pfSense UI on `morpheus` for one -reservation, a shell on `alexander` for §5 if the lab's ports are still shut, +reservation, a shell on `alexander` for §5, and the Mac for §6 — it is the only machine that reaches this segment with a checkout in hand **After this:** [`open-the-remote-path.md`](open-the-remote-path.md), if the @@ -13,6 +13,17 @@ remote path is wanted — terminates the WireGuard tunnel on this host, and its runbook begins where this one ends +> **Status — 2026-09-DD: `phoenix` is built and its agent is pushing to +> `alexander`.** +> +> Ubuntu 26.04 LTS, `10.0.30.70`, VMID 170, `bc:24:11` OUI, ISO +> `ubuntu-26.04-live-server-amd64.iso` *(ISO and OUI pending the §1 report)*. +> §4's `curl`: *(pending the §4 report)*. §7: *(pending the §7 report — both +> `up{instance="phoenix"}` jobs and the Loki job count go here)*. +> The lab's ports were opened ahead of the guest by #543 and applied on +> `alexander` in §5, the first time `deploy-agent.sh`'s `--monitoring-host` +> flag was used. + This builds the host [ADR-0043](../adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) decided: the one machine whose purpose is to hold credentials for other machines — a Proxmox API token, an SSH key, a checkout — so that the Packer, diff --git a/docs/runbooks/fit-the-saruman-ssds.md b/docs/runbooks/fit-the-saruman-ssds.md index 81c9625c..5d8ea790 100644 --- a/docs/runbooks/fit-the-saruman-ssds.md +++ b/docs/runbooks/fit-the-saruman-ssds.md @@ -619,7 +619,11 @@ lvremove /dev/pve/fiotest ## 9. Move the guest -There is exactly one guest, so this is one command rather than a campaign. +There are two guests — `alexander` (140) and, since 2026-09-DD, `phoenix` +(170, [#436](https://github.com/Gerrrt/HomeLab/issues/436)), created on +`local-lvm` like the first — so this is one command run twice rather than a +campaign. The commands below name 140; repeat them for 170 once it has +pivoted. ```bash qm config 140 From f48e740777b97606652a19c88eae80fd0a415dc1 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 06:38:07 +0000 Subject: [PATCH 2/7] docs: phoenix was created on Saruman on 2026-09-20 from the 26.04.1 server ISO (#436) Co-Authored-By: Claude Fable 5.1 --- docs/architecture.md | 4 ++-- docs/network.md | 6 +++--- docs/observability.md | 2 +- docs/roadmap.md | 2 +- docs/runbooks/build-the-jumpbox.md | 4 ++-- docs/runbooks/fit-the-saruman-ssds.md | 2 +- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 731a2df8..1c4189a4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -185,10 +185,10 @@ hole from the monitoring VLAN into the monitored one. | Host | VLAN | Stack | Contents | | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, docker-socket-proxy, Alloy | -| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running two guests — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built 2026-09-DD ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — and seven more planned below: `odin` for the security tooling (ADR-0030) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | +| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running two guests — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — and seven more planned below: `odin` for the security tooling (ADR-0030) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | -| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-DD: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | diff --git a/docs/network.md b/docs/network.md index 58bff38f..a7e44dac 100644 --- a/docs/network.md +++ b/docs/network.md @@ -91,7 +91,7 @@ truth for what a box actually does. address ([`security.md`](security.md#what-this-repository-deliberately-does-not-publish)). Before it, `morpheus` carried no `rdr` and no inbound WAN pass beyond DHCP client replies — the state ADR-0011 measured in 2026-08. **Not built:** the - jumpbox exists — `phoenix`, built 2026-09-DD + jumpbox exists — `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — and the endpoint question is answered — [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md): @@ -372,7 +372,7 @@ Where things get broken on purpose. made. - `Saruman` runs two guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built - 2026-09-DD ([#436](https://github.com/Gerrrt/HomeLab/issues/436)), described + 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)), described below. `alexander` runs [`stacks/lab`](../stacks/lab) — the lab's own Prometheus, Loki, Grafana and Alloy. **It is a guest and not the hypervisor for a reason**: a compose stack @@ -399,7 +399,7 @@ Where things get broken on purpose. that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) runs from, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) - and built 2026-09-DD by + and built 2026-09-20 by [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). It runs no stack and holds no key that signs anything — the estate's CA stays on `prometheus`, and that ADR says why. It is also where diff --git a/docs/observability.md b/docs/observability.md index 5023d580..61b8c807 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -14,7 +14,7 @@ settles its shape. It runs on `alexander`, the guest [#262](https://github.com/Gerrrt/HomeLab/issues/262) built on 2026-09-05. Its first client from off that guest is `phoenix`, the deployment host ([ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), -built 2026-09-DD): its native Alloy pushes over the two ports the lab stack +built 2026-09-20): its native Alloy pushes over the two ports the lab stack published for it ([#436](https://github.com/Gerrrt/HomeLab/issues/436)), and it too reaches nothing here. diff --git a/docs/roadmap.md b/docs/roadmap.md index 7fd821d4..a10d6ee6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1179,7 +1179,7 @@ what left this one unfireable for months. - **[#436](https://github.com/Gerrrt/HomeLab/issues/436) Build a deployment jumpbox on ImaginationLAN, and decide where the CA lives.** Decided by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md); - the guest was built 2026-09-DD. The host is `phoenix`, `10.0.30.70`, a guest on + the guest was built 2026-09-20. The host is `phoenix`, `10.0.30.70`, a guest on `Saruman` that holds the estate's first Proxmox API credential and the SSH key the toolchain will inject into what it builds — the prerequisite for the Packer, OpenTofu and Ansible issues, which today have nowhere to run diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 5fd38806..2095ab3d 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -13,11 +13,11 @@ remote path is wanted — terminates the WireGuard tunnel on this host, and its runbook begins where this one ends -> **Status — 2026-09-DD: `phoenix` is built and its agent is pushing to +> **Status — 2026-09-20: `phoenix` is built and its agent is pushing to > `alexander`.** > > Ubuntu 26.04 LTS, `10.0.30.70`, VMID 170, `bc:24:11` OUI, ISO -> `ubuntu-26.04-live-server-amd64.iso` *(ISO and OUI pending the §1 report)*. +> `ubuntu-26.04.1-live-server-amd64.iso` *(OUI pending the §1 report)*. > §4's `curl`: *(pending the §4 report)*. §7: *(pending the §7 report — both > `up{instance="phoenix"}` jobs and the Loki job count go here)*. > The lab's ports were opened ahead of the guest by #543 and applied on diff --git a/docs/runbooks/fit-the-saruman-ssds.md b/docs/runbooks/fit-the-saruman-ssds.md index b793f8bd..15709ef0 100644 --- a/docs/runbooks/fit-the-saruman-ssds.md +++ b/docs/runbooks/fit-the-saruman-ssds.md @@ -789,7 +789,7 @@ lvremove /dev/pve/fiotest ## 9. Move the guest -There are two guests — `alexander` (140) and, since 2026-09-DD, `phoenix` +There are two guests — `alexander` (140) and, since 2026-09-20, `phoenix` (170, [#436](https://github.com/Gerrrt/HomeLab/issues/436)), created on `local-lvm` like the first — so this is one command run twice rather than a campaign. The commands below name 140; repeat them for 170 once it has From 07a130e6849577535c47039cee416f19acc5fb6a Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 07:05:31 +0000 Subject: [PATCH 3/7] docs(runbook): phoenix's NIC carries the bc:24:11 OUI the network.md row assumed (#436) Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-jumpbox.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 2095ab3d..7cf02da0 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -17,7 +17,7 @@ this one ends > `alexander`.** > > Ubuntu 26.04 LTS, `10.0.30.70`, VMID 170, `bc:24:11` OUI, ISO -> `ubuntu-26.04.1-live-server-amd64.iso` *(OUI pending the §1 report)*. +> `ubuntu-26.04.1-live-server-amd64.iso`. > §4's `curl`: *(pending the §4 report)*. §7: *(pending the §7 report — both > `up{instance="phoenix"}` jobs and the Loki job count go here)*. > The lab's ports were opened ahead of the guest by #543 and applied on From 8de0af996530fa63e0b400286675e63c51f96273 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 07:16:25 +0000 Subject: [PATCH 4/7] docs(runbook): PhoenixBuilder drops VM.Monitor, which Proxmox VE 8 removed and 9 rejects, and the token secret goes into the file and nowhere else (#436) Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-jumpbox.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 7cf02da0..b4c4d6df 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -136,7 +136,7 @@ the role is its own so that what it can do is a list rather than pveum role add PhoenixBuilder --privs \ "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit \ VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network \ - VM.Config.Options VM.Console VM.Monitor VM.PowerMgmt VM.Audit VM.Snapshot \ + VM.Config.Options VM.Console VM.PowerMgmt VM.Audit VM.Snapshot \ Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use" pveum user add phoenix@pve --comment "deployment host, ADR-0043" pveum acl modify /vms --users phoenix@pve --roles PhoenixBuilder @@ -147,8 +147,15 @@ pveum acl modify /nodes/saruman --users phoenix@pve --roles PVEAuditor pveum user token add phoenix@pve builder --privsep 0 ``` -Four of those are worth knowing rather than copying: +Five of those are worth knowing rather than copying: +- **No `VM.Monitor`.** Packer's own permission list still names it, and the + first run of this section copied it in; Proxmox VE 8 dropped the privilege, + so on 9 `pveum role add` rejects the whole list and every `acl modify` + after it fails with "role does not exist". The boot command Packer types + goes through `sendkey`, which is `VM.Console`, already granted. A `pveum` + error here means the role was never made — fix the list and rerun the + role and the four ACLs; the `PVEAuditor` line and the token are unaffected. - **`/vms`, the two storages and the bridge, and nothing under `/nodes` but audit.** The token that can create guests must not be able to touch the host firewall ADR-0014 depends on, and `PVEAuditor` on the node is @@ -161,7 +168,10 @@ Four of those are worth knowing rather than copying: permissions and there is no second set to keep in step. The trade is that it is exactly as powerful as the user, which is why the user is this narrow. - **The secret prints once.** Copy it now; it cannot be shown again, only - regenerated. + regenerated. Copy it into the file in the next block and nowhere else — a + session transcript or a chat is a log, and a secret pasted into one is + rotated, not kept. `pveum user token remove phoenix@pve builder` and the + `token add` line again is the whole rotation. **Then the door.** ADR-0014 closes `8006` on `Saruman` to `10.0.50.0/24` and this guest is not on it. ADR-0043 admits one address, on this port and no From 31155e0ca98f6225df195a094df275e78d79e089 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 07:24:28 +0000 Subject: [PATCH 5/7] =?UTF-8?q?docs:=20the=20Proxmox=20node=20is=20Saruman?= =?UTF-8?q?,=20case=20and=20all,=20and=20its=20firewall=20was=20never=20en?= =?UTF-8?q?abled=20=E2=80=94=20phoenix's=208006=20line=20is=20decided=20an?= =?UTF-8?q?d=20unwritten=20(#436,=20#566)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings from running build-the-jumpbox.md §4. The node name is the hostname with its capital, so the runbook's /nodes/saruman ACL, host.fw path and API URL were wrong; all three now read Saruman, with a note that pveum does not check the path. And pve-firewall on Saruman is disabled with neither cluster.fw nor host.fw present: ADR-0014's rules are applied by the playground runbook, which has not run. The door section says so, the line was not written, and #566 carries enabling the firewall with all four rules. The sentences in network.md and architecture.md that said the rule stands now say decided and unwritten, and ADR-0043 gets a marked note on the premise. Co-Authored-By: Claude Fable 5.1 --- ...nd-build-phoenix-as-the-deployment-host.md | 14 +++++++ docs/architecture.md | 2 +- docs/network.md | 9 ++-- docs/runbooks/build-the-jumpbox.md | 41 ++++++++++++------- 4 files changed, 47 insertions(+), 19 deletions(-) diff --git a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md index ce31f757..d3260875 100644 --- a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md +++ b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md @@ -5,6 +5,20 @@ and amends the first consequence of [ADR-0039](0039-decline-proxmox-clustering-while-ifrit-is-the-range.md) +> [!NOTE] +> **One premise was wrong, 2026-09-20.** The Context below says that today +> nothing on VLAN 30 is admitted to `8006` on `Saruman`, and the Decision +> widens ADR-0014's rule by one line. Building `phoenix` found the Proxmox +> firewall on `Saruman` disabled, with neither `cluster.fw` nor `host.fw` +> present: ADR-0014's rule was decided, and the runbook that applies it — +> `build-the-playground.md` §4 — is gated on #101 and has not run. So every +> address on the segment reaches `8006`, and the line this ADR adds has no +> wall to go in. The decision stands as written — the line is what to write +> when the wall goes up, and the runbook's CAUTION not to raise it as a side +> effect of a guest build held. Enabling the firewall with all four rules is +> [#566](https://github.com/Gerrrt/HomeLab/issues/566). The text below is +> left as written, per ADR-0001. + ## Context [#436](https://github.com/Gerrrt/HomeLab/issues/436) is the prerequisite for diff --git a/docs/architecture.md b/docs/architecture.md index 1c4189a4..4b73670a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -188,7 +188,7 @@ hole from the monitoring VLAN into the monitored one. | `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running two guests — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)), and `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)) — and seven more planned below: `odin` for the security tooling (ADR-0030) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | -| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — decided, and unwritten while [#566](https://github.com/Gerrrt/HomeLab/issues/566) is open, because the build found that firewall disabled. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | diff --git a/docs/network.md b/docs/network.md index a7e44dac..925f5cb0 100644 --- a/docs/network.md +++ b/docs/network.md @@ -410,10 +410,13 @@ Where things get broken on purpose. not to Winterfell, and it gets no pass out of this segment: a rule from it into 99 would make it the bastion ADR-0002 and ADR-0012 declined. It adds nothing on `morpheus`. What it adds is on `Saruman`: one line in the - hypervisor's own firewall admitting `10.0.30.70` to `8006`, the first - address on this segment with a path to the Proxmox API, which + hypervisor's own firewall admitting `10.0.30.70` to `8006`, which `firewall-claims.yaml` cannot see because it lives in `/etc/pve` and not in - pf. + pf. That line is decided and not written: on 2026-09-20 the build found + `Saruman`'s firewall disabled and ADR-0014's rules never applied, so today + every address on this segment reaches `8006`, and + [#566](https://github.com/Gerrrt/HomeLab/issues/566) carries closing it + with all four rules. - `Saruman` runs an Alloy agent and is the one host on this segment with a path into Winterfell: a single pass, `10.0.30.110 → 10.0.99.20` on 9090 and 3100 TCP, unlogged and above the ADR-0014 tripwire. The hypervisor's own telemetry diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index b4c4d6df..7b1edf50 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -143,7 +143,7 @@ pveum acl modify /vms --users phoenix@pve --roles PhoenixBuilder pveum acl modify /storage/local --users phoenix@pve --roles PhoenixBuilder pveum acl modify /storage/local-lvm --users phoenix@pve --roles PhoenixBuilder pveum acl modify /sdn/zones/localnetwork/vmbr0 --users phoenix@pve --roles PhoenixBuilder -pveum acl modify /nodes/saruman --users phoenix@pve --roles PVEAuditor +pveum acl modify /nodes/Saruman --users phoenix@pve --roles PVEAuditor pveum user token add phoenix@pve builder --privsep 0 ``` @@ -162,8 +162,11 @@ Five of those are worth knowing rather than copying: read-only. When Packer or OpenTofu fail with `Permission check failed`, the fix is a privilege added to `PhoenixBuilder`, not the role granted at `/`. -- **`/nodes/saruman`** is the node name as `pvesh get /nodes` prints it, which - is lower-case whatever the documents call the box. +- **`/nodes/Saruman`** is the node name as `pvesh get /nodes` prints it — the + hostname, case and all, and the same string names the directory under + `/etc/pve/nodes/` and the path in every API URL. This runbook first said + lower-case and the ACL went onto a path that does not exist; `pveum acl + modify` does not check. `pveum acl list | grep phoenix` is the check. - **`--privsep 0`** is the issue's choice: the token carries the user's permissions and there is no second set to keep in step. The trade is that it is exactly as powerful as the user, which is why the user is this narrow. @@ -173,21 +176,27 @@ Five of those are worth knowing rather than copying: rotated, not kept. `pveum user token remove phoenix@pve builder` and the `token add` line again is the whole rotation. -**Then the door.** ADR-0014 closes `8006` on `Saruman` to `10.0.50.0/24` and -this guest is not on it. ADR-0043 admits one address, on this port and no -other. In `/etc/pve/nodes/saruman/host.fw`, beneath the three rules ADR-0014 -wrote: +**Then the door — which, on the day, had no wall.** ADR-0014 closes `8006` +on `Saruman` to `10.0.50.0/24` and this guest is not on it. ADR-0043 admits +one address, on this port and no other. The line is, in +`/etc/pve/nodes/Saruman/host.fw`, beneath the three rules ADR-0014 wrote: ```ini IN ACCEPT -source 10.0.30.70 -p tcp -dport 8006 -log nolog ``` > [!CAUTION] -> If that file does not exist yet, ADR-0014's rules were never applied on -> `Saruman` and the firewall is off. Do not turn it on as a side effect of -> this line — read [`build-the-playground.md`](build-the-playground.md) §4 -> first, with the KVM console to hand, because a `DROP` input policy with the -> rules unrendered locks you out of a machine whose console is a switch away. +> **Check before writing it:** `pve-firewall status` and +> `ls /etc/pve/firewall/cluster.fw /etc/pve/nodes/Saruman/host.fw`. On +> 2026-09-20 the answer was `disabled/running` and neither file — ADR-0014's +> rules had never been applied on `Saruman`, because the runbook that applies +> them, [`build-the-playground.md`](build-the-playground.md) §4, is gated on +> #101 and had not run. Do not turn the firewall on as a side effect of this +> line: a `DROP` input policy with the rules unrendered locks you out of a +> machine whose console is a KVM switch away. **The line above was not +> written**; [#566](https://github.com/Gerrrt/HomeLab/issues/566) carries +> enabling the firewall with all four rules, with the console to hand, and +> until it is done every address on this segment reaches `8006`. **On `phoenix`**, the credential and the key: @@ -219,13 +228,15 @@ Prove the door and the token together, from `phoenix`: ```bash set -a; . ~/.config/proxmox/phoenix.env; set +a curl -sk -H "Authorization: PVEAPIToken=${PROXMOX_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" \ - "${PROXMOX_URL}/nodes/saruman/qemu" | python3 -m json.tool | grep '"name"' + "${PROXMOX_URL}/nodes/Saruman/qemu" | python3 -m json.tool | grep '"name"' ``` `alexander` and `phoenix` at minimum. A connection timeout is the door — the `host.fw` line is missing or the firewall was not reloaded (`pve-firewall -compile` shows what it thinks the rules are). A `401` is the token. An empty -list with a `200` is the ACL: the user can reach the node and see no guests. +compile` shows what it thinks the rules are) — or, while #566 is open, the +network, because there is no door to be shut. A `401` is the token. An empty +list with a `200` is the ACL: the user can reach the node and see no guests, +which is also what a `PVEAuditor` grant on a mis-cased node path looks like. ## 5. Open the lab's doors From 099ad498a4bd08903ba196f3c22b7a3eba83ede3 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 08:25:34 +0000 Subject: [PATCH 6/7] =?UTF-8?q?docs(runbook):=20phoenix's=20=C2=A77=20pass?= =?UTF-8?q?ed=20=E2=80=94=20validate=20green,=20both=20jobs=20up,=20four?= =?UTF-8?q?=20Loki=20jobs=20(#436)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-jumpbox.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index 7b1edf50..e09b4b3e 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -17,12 +17,17 @@ this one ends > `alexander`.** > > Ubuntu 26.04 LTS, `10.0.30.70`, VMID 170, `bc:24:11` OUI, ISO -> `ubuntu-26.04.1-live-server-amd64.iso`. -> §4's `curl`: *(pending the §4 report)*. §7: *(pending the §7 report — both -> `up{instance="phoenix"}` jobs and the Loki job count go here)*. -> The lab's ports were opened ahead of the guest by #543 and applied on -> `alexander` in §5, the first time `deploy-agent.sh`'s `--monitoring-host` -> flag was used. +> `ubuntu-26.04.1-live-server-amd64.iso`. §4's `curl`: *(pending the §4 +> report)*. §7 passed: `make validate` green in the guest's own checkout (16 +> host-specific skips — no Docker, no age key, by design), both +> `up{instance="phoenix"}` jobs at `1`, and four Loki jobs — `auth.log`, +> `syslog`, `/var/log/*.log` and the journal — so 26.04.1 ships rsyslog as +> 26.04 did. The lab's ports were opened ahead of the guest by #543 and +> applied on `alexander` in §5, the first time `deploy-agent.sh`'s +> `--monitoring-host` flag was used. Two things the first run found and this +> runbook now carries: `VM.Monitor` is not a privilege on PVE 9, and the node +> is `Saruman`, capitalised — and one it could not fix, #566: the hypervisor's +> firewall was never on, so the `8006` line was not written. This builds the host [ADR-0043](../adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) decided: the one machine whose purpose is to hold credentials for other From 8bc22f39d243983a7792e4cf5c7a34c01582a2ef Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 20 Sep 2026 08:28:30 +0000 Subject: [PATCH 7/7] =?UTF-8?q?docs(runbook):=20phoenix's=20token=20lists?= =?UTF-8?q?=20both=20guests=20=E2=80=94=20=C2=A74=20proven,=20no=20placeho?= =?UTF-8?q?lder=20left=20(#436)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-jumpbox.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md index e09b4b3e..2395d697 100644 --- a/docs/runbooks/build-the-jumpbox.md +++ b/docs/runbooks/build-the-jumpbox.md @@ -17,8 +17,8 @@ this one ends > `alexander`.** > > Ubuntu 26.04 LTS, `10.0.30.70`, VMID 170, `bc:24:11` OUI, ISO -> `ubuntu-26.04.1-live-server-amd64.iso`. §4's `curl`: *(pending the §4 -> report)*. §7 passed: `make validate` green in the guest's own checkout (16 +> `ubuntu-26.04.1-live-server-amd64.iso`. §4's `curl` returned `alexander` +> and `phoenix` — token, ACLs and node name all right. §7 passed: `make validate` green in the guest's own checkout (16 > host-specific skips — no Docker, no age key, by design), both > `up{instance="phoenix"}` jobs at `1`, and four Loki jobs — `auth.log`, > `syslog`, `/var/log/*.log` and the journal — so 26.04.1 ships rsyslog as