diff --git a/README.md b/README.md index 5a9a3021..4ad2096c 100644 --- a/README.md +++ b/README.md @@ -368,9 +368,10 @@ self-test, and the card is set to test itself every fortnight ([#93](https://github.com/Gerrrt/HomeLab/issues/93)) โ€” and since 2026-09-08 the switch between the monitoring host and the network draws from it too, racked in U4 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and the config export -leaves the monitoring host nightly and the volume backup sets weekly -([#535](https://github.com/Gerrrt/HomeLab/issues/535)), so the rehearsal is -what is left ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). +leaves the monitoring host nightly, the volume backup sets weekly +([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and Jellyfin's state +weekly by way of it ([#484](https://github.com/Gerrrt/HomeLab/issues/484)), +so the rehearsal is what is left ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). ## License diff --git a/docs/architecture.md b/docs/architecture.md index 90f8f24e..8855fe98 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -189,7 +189,7 @@ hole from the monitoring VLAN into the monitored one. | `alexander` (10.0.30.40) | ๐ŸŸข 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy โ€” the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | ๐ŸŸข 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** โ€” Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `phoenix` (10.0.30.70) | ๐ŸŸข 30 | *(none โ€” a toolchain host, no Docker)* | **Not built yet** โ€” the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | -| `oracle` (10.0.99.30) | ๐Ÿ”ด 99 | *(none โ€” hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`) and of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)). The estate's host for small off-host jobs โ€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | +| `oracle` (10.0.99.30) | ๐Ÿ”ด 99 | *(none โ€” hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs โ€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | ๐Ÿ”ด 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** โ€” ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich โ€” four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) โ€” Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | ๐ŸŸก 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 โ€” from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) ยง6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy โ€” the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 โ†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | | `bahamut` (10.0.30.50) | ๐ŸŸข 30 | *(none โ€” Windows)* | **Not built yet** โ€” Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` โ€” Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/network.md b/docs/network.md index 45b039d8..53651c6c 100644 --- a/docs/network.md +++ b/docs/network.md @@ -184,9 +184,11 @@ listed under [Hicks](#hicks--vlan-50--trusted), and nothing else. - `oracle` runs the Lemmiwinks wiki and its Postgres โ€” it has since 2025-11-12, and [ADR-0011](adr/0011-keep-the-wiki-internal.md) depends on it โ€” and holds the off-host copies of the firewall export that `make backup-firewall` pushes - to it and of the weekly volume sets that `make backup` pushes - ([#535](https://github.com/Gerrrt/HomeLab/issues/535)), as ciphertext with - no key. Its role is the estate's small off-host + to it, of the weekly volume sets that `make backup` pushes + ([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's + state that `make backup-nas` pulls off `smaug` and pushes on + ([ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)), as + ciphertext with no key. Its role is the estate's small off-host jobs: [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md). Its NIC supports 10/100 only, so that link runs at 100 Mb/s โ€” measured 2026-09-03 โ€” and no cable will lift it. `prometheus` links at a gigabit through the same diff --git a/docs/observability.md b/docs/observability.md index fb29404d..abd88edb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -675,12 +675,16 @@ per recipient off `homelab_key_recipient_last_proof_timestamp_seconds` rather than off the job. One timestamp for every copy would mean proving either one vouched for the other, which is backwards when the whole point of the second copy is that it fails independently. With a single recipient it behaves exactly -as it always has. Two outputs leave: `backup-firewall` copies each export +as it always has. Three outputs leave: `backup-firewall` copies each export to `oracle` and fails if it cannot, so its failure alert doubles as "the config -has stopped leaving this host", and since +has stopped leaving this host"; since [#535](https://github.com/Gerrrt/HomeLab/issues/535) `backup-volumes` does the -same with each weekly set, with `verify-backups` hashing the far side every -morning โ€” no rule names either job; the generic pair carries both. +same with each weekly set; and since +[#484](https://github.com/Gerrrt/HomeLab/issues/484) `backup-nas` โ€” the one +job that first *fetches* from another host, Jellyfin's state off `smaug` โ€” +copies its set the same way. `verify-backups` hashes the far side of both +set directories every morning. No rule names any of the three; the generic +pair carries them all. That series has to exist for the nag to mean anything, and for four days it did not ([#400](https://github.com/Gerrrt/HomeLab/issues/400)): it was written only diff --git a/stacks/media/README.md b/stacks/media/README.md index e5e3991a..af80d2e2 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -60,6 +60,7 @@ every television would have to trust, and a second thing to be down. | Televisions on CasaBonita | Natively, same broadcast domain โ€” the firewall never sees the packet | | A Hicks workstation | Two of the four rules in [`build-the-nas.md`] ยง0.5 โ€” `50 โ†’ 10.0.40.30:443` and `50 โ†’ 10.0.40.30:8096`, one per port | | Prometheus, on `9100` | A third โ€” `10.0.99.20 โ†’ 10.0.40.30:9100` | +| Prometheus, on `22` | The fourth โ€” `10.0.99.20 โ†’ 10.0.40.30:22`, inert until [`build-the-nas.md`] ยง6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | | Everything else on the estate | Not at all โ€” default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and