Skip to content

security.md

Latest commit

 

History

History
764 lines (680 loc) · 51.2 KB

security.md

File metadata and controls

764 lines (680 loc) · 51.2 KB
Raw

Security

The lab is a security project, so the interesting question is not "is it secure" but "what is it defending against, and what is it knowingly not."

Threat model

What this network is actually built to survive:

Threat Control
A compromised IoT device pivoting to a workstation VLAN 20 is terminal — no route to any other segment
A guest on the Wi-Fi enumerating the LAN VLAN 10 is terminal, client isolation on
A smart TV's firmware phoning somewhere unexpected VLAN 40 is terminal outward — egress only, and nothing on it initiates into another segment. Since 2026-09-16 two more-trusted segments reach one host on it on named ports (ADR-0016), which changes what reaches in and not what gets out; the igc0.40 tripwire still reads zero packets
A corporate laptop carrying something in from outside Sits on VLAN 50 but has no management access
A lab VM escaping into the house VLAN 30 reachable only from trusted, never to it; the lab tripwire logs anything that gets past the blocks (#234)
An attacker on the lab segment reaching the hypervisor's BMC Accepted. shiva stays on VLAN 30 by decision (ADR-0033), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates
A range target with a path out It has none — ifrit's targets sit on a bridge with no physical port, on 172.30.30.0/24, which the firewall does not route and on which nothing has a default route at all (ADR-0014, ADR-0017)
Someone with the trusted Wi-Fi key quietly joining Kea's lease log reaches Loki; UnknownDeviceOnTrustedSegment fires the first time a MAC appears on VLAN 50 in seven days (ADR-0019)
Losing visibility of a failure 101 alert rules, 30 days of metrics and logs
Someone on a reachable VLAN silencing an alert to hide a failure Alertmanager binds to 127.0.0.1; silences go through authenticated Grafana
Mains power loss The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12. A pack fitted to mjolnir on 2026-08-28 passed its self-test; the TP-Link carrying prometheus and oracle has been on UPS power since 2026-09-08 (#110); the laptops ride a cut out on their own batteries, which HostBatteryHealthLow in host.rules.yaml now reads — prometheus's cell was replaced on 2026-09-18 and reads 101 % of design, oracle's is the original at 72 % and unbought (#454) — and how long either laptop runs on its cell has never been measured, so this row is answered as far as the cells being healthy and no further — see below
The estate being down while the person who runs it is unavailable Documentation, yes; data, not yet. ADR-0011 puts the emergency tier on paper; ADR-0023 extends the same reasoning to the sensitive tier's data before that tier exists — see below

What it explicitly does not defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container image, or a vulnerability in pfSense itself. There is no egress filtering by domain or by port (ADR-0014 says why not for the lab, and the reason generalises), and no MFA on the internal services.

The MFA gap has an expiry now, and it is not one gap. ADR-0022 ends ADR-0008's SSO deferral on a state rather than leaving it open: the first real secret, photo or document in the sensitive tier, any reachability from outside the house, or a third account holder — whichever comes first. The second of those has now fired and the deferral was re-accepted, not ended: ADR-0042 opens a WireGuard path terminating on the lab, which takes ADR-0008's no external exposure premise with it. Nothing in the tier became reachable — it is unbuilt, and on Winterfell when it is built — but the lab's own Grafana on alexander did, and that is one of the three below that cannot carry a factor at all. The other two triggers keep their full force. Until then the floor is per-application TOTP, and it does not reach everything. Vaultwarden, Paperless-ngx and Home Assistant can each carry a second factor; Grafana, Immich and AdGuard Home cannot — Grafana OSS has no MFA in any edition, Immich's upstream has declined it and points at OAuth, and AdGuard has one password-only admin account. For those three an identity provider is the only route to a second factor rather than a heavier alternative to one. Grafana is the only one of the six deployed today, which makes "no MFA" a standing property of the estate rather than a pending piece of work — it closes when an identity provider exists and not before.

Availability of the data is a separate question from access to it, and it now has an answer too. ADR-0008's tier will hold the household's credentials, photographs and documents on one mini PC behind a private CA, so if that box is down and the person who runs it is unreachable, the credentials for recovering everything else sit behind the thing that broke — and step-ca means the browser refuses the handshake rather than degrading. ADR-0023 declines to make any of it highly available and constrains the path instead: nothing the household needs in an emergency may have the estate on its only route. The household's own credentials are recoverable without Vaultwarden; Immich and Paperless-ngx get an encrypted copy outside the estate whose staleness is visible, off-estate rather than off-host, because oracle shares the rack and the power feed; nothing on the break-glass card depends on a certificate this estate issues; and nothing physical may be operable only through Home Assistant. Those fall due on ADR-0022's triggers — the first real credential, photo or document — and none of them is built. The copy leaving the house is a new residual: it is the first household data to sit in someone else's building, reduced to an availability problem by encryption at rest with a key that never leaves here, and accepted on that basis.

Intrusion detection has been running on Skids (VLAN 20) since 2026-08-21 and on Degens (VLAN 10) since 2026-09-02, one Suricata process per interface. Suricata sits on morpheus rather than the hypervisor because it is the only device that sees the IoT and guest segments, per ADR-0006. Alerts reach Loki through the firewall's syslog pipe, with classification, priority and interface parsed into labels; SuricataHighPriorityAlert and SuricataAlertStorm are armed against them per interface, and the homelab-security dashboard charts them next to the firewall's own block decisions. runbooks/enable-suricata.md covers the setup and the tuning.

Three limits, stated rather than implied:

  • It is alert-only. Block Offenders is off on both interfaces and stays off until a fortnight of understood alerts on each, and probably not on VLAN 20 even then — an auto-block there can take out a camera or the alarm hub. On VLAN 10 it would hit a guest's device whose owner cannot be told why.
  • It watches two segments. Skids and Degens; the rest are unwatched. WAN deliberately never will be.
  • It sees plaintext only. Suricata cannot inspect inside TLS, so the useful signal is DNS, SNI, JA3 and the diminishing share of traffic still in the clear.

Suricata dying is detected as of 2026-09-03, and Suricata going mute as of 2026-09-12. A quiet IDS and a stopped one produce identical log output for an hour at a time, so SuricataStopped in prometheus/rules/ids.rules.yaml reads the firewall's process table over SNMP and fires per declared interface within ten minutes (#90). It proves the process is alive, not that it is inspecting anything. SuricataLogsStopped in loki/rules/security.rules.yaml covers the rest of that gap slowly: it fires once both interfaces have been silent for nine hours, a window read from 22 days of the stream whose longest silence was 75 minutes, and it cannot see one interface going quiet on its own — the guest segment is silent for days at a time (#441). Re-measured on 2026-09-17 over 27.7 days, the worst aggregate silence was 80 minutes and the guest segment alone crossed nine hours seven times, so the window holds and the decision to aggregate is what keeps the rule quiet. The runbook's test alert is still the only proof that it detects.

A limit none of the three names, and no rule here closes: Suricata is not a protocol logger, and the lab is out of reach. SNI, JA3 and certificate metadata — the ground the plaintext limit gives up — are Zeek's, and east-west traffic between the lab's domain guests crosses no router, so morpheus never sees a packet of it (#437, ADR-0006). When that sensor is built its logs stay on alexander and never reach 10.0.99.20, which ADR-0007 requires and ADR-0020 enforces by giving the lab no Alertmanager for a ruler to deliver to. So the absence rule this section describes has no Zeek equivalent and will not get one: whether that sensor is still running is answered on the hypervisor, as guest state crossing under ADR-0028, by the homelab_zeek_mirror_active gauge #437 builds alongside the mirror.

Device joins are detected as of 2026-09-04, from the DHCP server rather than from the wireless. morpheus ships Kea's lease log to Loki, and the first lease a MAC takes on Hicks or Winterfell in seven days raises an alert — warning on the trusted segment, critical on management, where nothing has joined in the 13 days of logs the rules were written against. The eero cloud was rejected as the source: it is a two-minute poll of Amazon for an event the firewall logs in the same second, and it is unavailable exactly when the WAN is (ADR-0019).

Three limits here too. It sees leases, not associations — a device with a static address never asks, and never appears. It sees the segment, not the radio: on Hicks it cannot tell a new laptop on the cable from a new phone on the Wi-Fi, and it never knows which access point. And the noise floor is private MAC addresses: iOS and Android rotate them per network, so a rotation reads as a new device, which is accepted rather than filtered out because filtering it would blind the rule to the addresses an intruder would present.

Segmentation

Default deny holds for Switch LAN, Winterfell (99), ImaginationLAN (30), CasaBonita (40), Skids (20) and Degens (10). Each blocks every other segment explicitly before its egress rule, and the narrow exceptions that exist — SNMP to the iLO and to the switch, SSH to the firewall, the enumerated passes from Hicks into management — are listed in network.md and in ADR-0013.

It does not hold for Hicks (50), and since 2026-09-08 that is a decision rather than an omission. Hicks reaches management on a named list above a logged block, and reaches all of ImaginationLAN by a rule that says so — the lab is meant to be reachable from trusted workstations, and cannot be narrowed until the lab exists and its list does. ADR-0031 records both, and the two deviations it deliberately left in place: the lab rule is TCP-only, with the catch-all carrying the rest, and the DNS and NTP passes to 10.0.99.1 carry nothing because Hicks resolves at its own gateway.

The switch LAN was the other exception and is not any more. Since 2026-09-02 that interface carries six logged blocks — one per VLAN — above an egress rule renamed Allow internet, with DNS and NTP to the gateway the only passes above them (#229); since 2026-09-06 nothing on Winterfell can reach it either (ADR-0025).

IPv6 terminates at the WAN and is not carried inside the estate. That was never written down and is now measured rather than assumed (#353, 2026-09-07): the WAN interface holds one global v6 address and a working v6 default route — the firewall itself reaches the v6 internet in about 11 ms — while no VLAN interface has a global v6 address, and neither radvd nor dhcpd6 is running. So nothing inside is offered IPv6 and nothing inside can prefer a route that does not work, which is the failure a half-configured v6 stack usually causes. The estate is addressed in RFC1918 IPv4 by decision, and this paragraph is that decision.

The WAN_DHCP6 gateway has reported 100% loss for days and the link is fine. dpinger monitors the ISP gateway's link-local address, which does not answer ICMPv6 echo; traffic traverses that same gateway to Comcast's Seattle router, and its NDP entry is live. The "outage" is a property of the monitor target. make gateway-state now collects both the reported status and whether each family actually leaves the building, and GatewayMonitorUnreliable fires on precisely that disagreement — so this is visible instead of being something somebody had to go and ask the firewall about. Fixing it is a firewall change (point the gateway's Monitor IP at an address that answers, or set it to not-monitored) and belongs on the Lemmiwinks side.

IPv6 is where the segmentation pattern is not finished, and both halves of the gap are worth naming because the documents recorded only one of them for a while. The switch LAN interface still carries pfSense's stock Default allow LAN IPv6 to any with no IPv6 blocks above it, so on paper it reaches every VLAN. Going the other way, CasaBonita, ImaginationLAN, Skids and Degens each carry paired inet/inet6 blocks toward the six VLAN macros and neither toward the switch LAN, so their IPv6 catch-all reaches it. Both are latent rather than live for the same reason — igc0 has only a link-local address, which does not route — and both are divergences from the pattern rather than paths anything can use today. #353 owns whether IPv6 is carried at all.

Every claim in the four paragraphs above is checked against the running firewall rather than reviewed. docs/firewall-claims.yaml states the posture in the form pfctl can answer and make check-firewall diffs it against the live ruleset; make check-docs asserts this section names the same segments the claims file does. Neither can run in CI's sandbox — the ruleset is not in this repository and deliberately never will be — so the first runs at deploy time and on a timer. ADR-0026 records why that is the shape, and #363 is what built it.

Winterfell is the half that has since been narrowed. On 2026-09-02 the Hicks interface gained ten host- and port-scoped passes into 99 and a logged Block access to Winterfell under them, so Hicks now reaches management on an enumerated list — SSH and ping to the segment, the firewall's admin UI, resolver and NTP, the wiki, Grafana, and the UPS card — and nothing else. network.md holds the list and is the document to read for it. ADR-0013 read the ruleset the day before that landed and describes the wider state; it is left as written, per ADR-0001, and this section is where the current posture lives.

This section previously said "three exceptions", ADR-0002 said two, and ADR-0008 said five. All three were counts, and a count cannot express "reachable because a catch-all was reached". ADR-0013 supersedes ADR-0002 and replaces the count with a list; the audit behind it is recorded there. A document that overstates a control is worse than one that admits the exception, and this one overstated it for as long as it was a number.

Everything else — IoT, media, guest — gets internet and nothing more.

That sentence is about what those segments initiate, and two decisions now reach into them without touching it. ADR-0016 writes passes into CasaBonita for the NAS and ADR-0035 one into Skids, and they are at different stages — this paragraph said neither was created, and half of that stopped being true on 2026-09-16. CasaBonita's exist. Hicks reaches 10.0.40.30 on 443 and 8096, and 10.0.99.20 reaches it on 9100 and 22: four host- and port-scoped passes above Block access to CasaBonita, verified in position with pfctl rather than read off the web UI, where an appended rule looks present while matching nothing. ADR-0016 wrote three and four exist, because the Hicks pass is split rather than carrying a port list, and the 22 is inert — TrueNAS ships SSH disabled (ADR-0040). network.md holds the current list. Skids' does not exist. 10.0.99.40 → 10.0.20.104:80,443/tcp — Home Assistant to the Hue bridge, the one device on that segment with a local API — still waits above the block that has stood between 99 and 20 since the segments existed, on the host that would use it (#134).

One of CasaBonita's four is a monitoring pass, and it carries a residual worth naming. 10.0.99.20 → 10.0.40.30:9100 lets Prometheus scrape node_exporter on the NAS (#256), and that endpoint is unauthenticated. The firewall rule is what stops other segments reading it; nothing stops CasaBonita itself, because the televisions and the game consoles share that broadcast domain and the firewall never sees those packets — the same property that lets them reach Jellyfin, working the other way. node_exporter has no write API, so the exposure is disclosure of the host's shape: filesystems, uptime, load. Accepted, and the same class as the unauthenticated ports the observability stack publishes.

The row ADR-0008wrote as 99 → 20 is narrower than it read: one host to one device on two ports, with the twenty other devices on Skids still unreachable from anywhere, and both segments still initiating nothing. The tripwire below is the check that the second half holds — waiting on Skids, and since 2026-09-16 actually checking on CasaBonita.

That sentence is now checked rather than asserted. Four tripwire rules sit below the block rules that stop each cross-segment path and above the → any egress rule: three on the terminal interfaces (#223), pass + log for <terminal net> → Internal_Segments, and a fourth on ImaginationLAN (#234, decided by ADR-0014), pass + log for <lab net> → House_Segments. The lab's rule points at a different alias on purpose: Internal_Segments names the lab itself, so against it every DNS query from the lab to its own gateway logs as a crossing — 1,239 such lines in the rule's first three days after 2026-09-06, none of them real, which is why #234 is not closed by the rule existing. They log nothing while the design holds, and cost nothing; if one ever logs a line, TerminalSegmentReachedInternalNetwork or LabSegmentReachedInternalNetwork fires on it. Before them the first of those alerts matched action="pass" against a firewall that logged only blocks, so it could not fire for any input — the control was described here and not actually watched. Note that a firewall restore from a backup older than 2026-09-01 drops them silently; the restore runbook checks for them.

On CasaBonita that tripwire is now the only control watching the segment, and since 2026-09-16 there is something on it worth watching. smaug at 10.0.40.30 is scraped rather than pushing, by the same ADR-0016 decision that keeps the segment terminal outward: it runs no Alloy agent, and with the agent go its logs, its SMART attributes and its patch state, because every one of those rides a push this host does not make. Loki has no pull — ingest is a push or it is nothing — so the only rule that would centralise the logs is the 40 → 99:3100 the ADR refused, and refusing it is the residual recorded in SECURITY.md (#255 for the logs, #483 for the rest). Suricata does not watch igc0.40 either; it runs on Skids and Degens only. So the tripwire is what is left, and it is worth being exact about what that buys: TerminalSegmentReachedInternalNetwork fires on a pass from 10, 20 or 40 toward 30, 50, 99 or the switch LAN, which is a compromise leaving the segment. Anything smaug does that stays on CasaBonita, or goes straight out to the internet, is not merely unalerted — it is unrecorded.

There is no alert for that and there cannot be one, which is a decision and not an omission. A rule reading "no logs from smaug" is an absence rule, and an absence rule needs a declared expectation to measure the silence against — homelab_suricata_expected_interface is that pattern, and SuricataLogsStopped only became writable once somebody measured what a normal silence looks like: 22 days of stream, a worst gap of 75 minutes, nine hours chosen as seven times it. Here the expected volume is zero and always was, so silence is the design working and no threshold separates it from a compromise. The two collection-completeness rules are no help for the same reason. JournalSourceStopped and LogEntriesDropped are both keyed on an agent that exists, and check_loki_coverage.py takes its denominator from the hosts Loki is already hearing from — so a host that has never shipped a line is absent from the question rather than failing it. The firewall and the tripwire are the control. This paragraph is the record that there is nothing else.

Segmentation is doing more work here than it should have to. Prometheus and Loki publish unauthenticated ingest ports for oracle's agent to use, so anything that can route to 10.0.99.20:9090 or 10.0.99.20:3100 can write to the metric and log stores without a credential — which is exactly the failure ADR-0002 predicted when it recorded that "a compromised workstation reaches Winterfell". That is an accepted residual, recorded in SECURITY.md, not a solved problem.

What has changed is who "anything" is. A workstation on Hicks was in that set for as long as the catch-all was the only rule in the way; since 2026-09-02 it reaches 10.0.99.20 on 3000 only and Block access to Winterfell drops the ingest ports. What remains in the set is a host already on Winterfell, and 10.0.30.110 on ImaginationLAN, which has an explicit pass to both ports for Saruman's Alloy agent. The residual narrowed by a firewall change nobody recorded; #182 still owns closing it properly, because a control that depends on one un-reviewed rule ordering is not authentication.

What has been taken off the firewall's shoulders is Alertmanager. It had no off-host client, so it now binds to 127.0.0.1 and reaching VLAN 99 no longer lets anyone silence an alert; see ADR-0012.

The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya white-noise machine. Every one of those is a network-connected computer running firmware nobody outside its vendor has audited, several with no update mechanism at all. Treating them as untrusted is not paranoia; it is the only assumption consistent with what they are.

Secrets

  • Credentials are encrypted with SOPS + age and committed in encrypted form. See secrets/README.md.
  • The private key lives at ~/.config/sops/age/keys.txt on the deployment host and is never in the repository. That host's disk is not encrypted — see below.
  • That key is the single point of failure for every encrypted secret here, so it is copied off the host and the copy is proven to decrypt with make secrets-verify-backup KEY=<copy> — which refuses to run against the live key and blanks the environment first, because the obvious hand-typed equivalent passes even for an unrelated keypair. See runbooks/back-up-the-age-key.md.
  • scripts/render-config.sh decrypts at deploy time into gitignored files. Nothing writes a plaintext secret into a tracked path.
  • make secrets-edit hardens $EDITOR before handing it the decrypted file, so the editor cannot persist the plaintext in an undo file, swap file or backup that sops does not shred. See secrets/README.md.
  • CI runs gitleaks with rules specifically for SNMP communities, inline Grafana passwords, PEM private keys and age secret keys, and separately asserts that every secrets/*.sops.yaml is genuinely encrypted.
  • One service on the sensitive tier keeps its credentials outside SOPS, by necessity and on the record. Home Assistant obtains device credentials through its own pairing flows — the Hue application key, the Ring token — and writes them to its store inside the home-assistant-config volume; no environment variable or rendered file is a way to hand them in. So the rule above covers what that stack takes from outside, and the tier's most numerous credentials are protected instead by the disk-encryption decision #404 makes and by the encrypted volume archive — which therefore carries live credentials, as grafana-data already does. ADR-0035 records the deviation and what would retire it.

Known historical exposure

This repository previously committed real credentials. Removing them from HEAD does not remove them from history, and anything ever pushed to a public repository must be treated as compromised:

What Where Status
SNMP community shared across all four devices snmp.yaml, from commit ee3d443 (now rewritten) Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. morpheus, mjolnir and shiva verified answering the new community and refusing the old. neo answers the new one; whether it still holds the previous one is unverified, not confirmed. Its agent serves GETBULK to any community of sixteen characters or fewer without consulting the table (measured 2026-09-12), which is what the 2026-09-06 and 2026-09-09 sightings of the old string and the stock public and private actually were; those measurements are withdrawn. Over GET, which it does check, both stock strings and a junk string are refused — the previous community's row is the one thing still unmeasured, because that string was purged from history and is not to hand. Accepted risk, see SECURITY.md and the runbook
Grafana admin / admin with anonymous Admin access compose file Fixed: password from SOPS, anonymous auth disabled
Decrypted secrets in editor undo files ~/.local/state/nvim/undodir/, written by make secrets-edit Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. make secrets-edit now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis
Alertmanager webhook URL and the MokerLink SNMP community a local Claude Code session transcript under ~/.claude/projects/ Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the neo residual below all over again. That reasoning is spent — the overwrite it was afraid of was done on 2026-09-12 and persisted — and the value retires with the MokerLink hardware, which is replaced by a switch commissioned with v2c off entirely; no value the MokerLink held is carried across
Passphrase-encrypted TLS private keys certificates/, added in efb2632, deleted in 647d90a Purged from history, and the CA replaced — see runbook. Anything that trusted the old CA must be re-pointed at the new one

CI scans both the working tree and the full history, with no ignore file. Both must be clean unconditionally.

There was a .gitleaksignore listing nine historical findings, each annotated with what it was and why it was still there. It was an acknowledgement, not a fix, and it existed because a CI job that is permanently red for a known reason gets ignored — and then a genuinely new leak goes unnoticed alongside it. The purge removed what it acknowledged, so the file was deleted. A history scan that passes with no exceptions is the evidence the purge worked.

Everything above sits on an unencrypted disk

Every entry in that table is something that happened once. This one is a standing property of the host, which is why it is stated separately rather than added as a sixth row.

Measured on prometheus (10.0.99.20):

  • /dev/mapper holds control and ubuntu--vg-ubuntu--lv and nothing else — no LUKS anywhere. The root filesystem is plain ext4 on LVM.
  • /boot and the EFI partition are likewise plain.
  • /swap.img is 4 GiB, unencrypted, on that same root filesystem, and in use. Anything the stack has held in memory can have been paged into it.

Since #114 the containers no longer contribute to that last line. Every service in stacks/observability/compose.yaml sets memswap_limit equal to its mem_limit, which is memory.swap.max = 0 — the stack cannot page out at all. Left unset, memswap_limit defaults to twice mem_limit, so setting a memory limit without one would have increased what reaches this file: a constrained container spills to swap rather than being killed, and Grafana's admin password, Alertmanager's receiver URLs and the SNMP community string are all resident in those processes. The measured cost of switching it off is small — peak container_memory_swap over the fourteen days that sized the limits was 70 MiB for Prometheus, 17 for Alloy, 12 for Loki and under 5 for the rest.

This narrows the exposure; it does not close it. Swap is only one of the ways memory reaches the disk, the host's own processes still swap freely, and anything already paged out before this change was deployed is still in /swap.img. The files below remain the substance of this section.

So the age private key at ~/.config/sops/age/keys.txt, the rendered artefacts under snmp-exporter/.rendered/ and alertmanager/.rendered/, and stacks/observability/.env — the last three hold plaintext by design, because something has to hand the containers a usable credential — are protected by nothing but file permissions. They are all mode 600 and owned by robo, which is the right setting and is also the entire control. Permissions are enforced by the running kernel; they mean nothing to a disk read on another machine.

This is accepted, not scheduled. The threat model above already excludes an attacker with physical access to the rack, and this is that exclusion restated where it actually bites. Full-disk encryption on a headless host has its own failure mode — either a passphrase nobody is present to type after a power cut, or a key stored on the same machine, which is most of the way back to where this started.

It is recorded because it changes the severity of things that would otherwise look minor. The undo-file leak above is the worked example: three community strings at mode 664 in ~/.local/state/nvim/undodir/ were a real finding because the disk beneath them is readable. On an encrypted disk that is a much smaller problem. Neither fact is interesting alone.

The UPS reported a battery it did not have

Until 2026-08-28 mjolnir had no battery installed. Its Network Management Card nonetheless reported 100% state of charge, 48.0 VDC, a battery temperature, an hour of runtime, a 2030 replacement date, and upsAlarmsPresent = 0. Every one of those values was derived rather than measured.

The single honest signal it emits is the self-test result. The management card rendered it as Refused — internal fault; over SNMP it is upsTestResultsSummary = 4 (aborted), from the standard UPS-MIB the apc_ups module already walks. No extra OIDs were needed to see it.

Any alert rule keyed on charge, runtime or alarm count therefore could not fire, no matter how bad things got. UpsSelfTestFailed and UpsBatteryUnproven in ups.rules.yaml key on the self-test instead, and are the only two rules in that file that can detect this condition.

This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it.

A pack was fitted on 2026-08-28, and that closed this. A card that cannot see a pack which is present — badly seated, or faulty out of the box — emits the same five fabricated values as one sitting over an empty bay, so a healthy-looking dashboard distinguished nothing. Only a passing self-test and readings that have left the pre-fit baseline do, and both now hold: upsTestResultsSummary went 4 (aborted) to 1 (donePass), upsBatteryVoltage left 480 for a float reading that varies, and the runtime estimate no longer sits on exactly 63. The silence on UpsSelfTestFailed was deleted the same day rather than left to expire in September, so that rule is live again.

Two things outlast the fix. Stored metrics older than 2026-08-28 are the fabricated values rather than measurements, so a dashboard or query whose range crosses that date is reading fiction on one side of it. And the card's test schedule is on but unwatched. Read off the NMC on 2026-09-03, upsAdvTestDiagnosticSchedule is 8 (biweeklySinceLastTest), which is what keeps 1 from being a frozen last-known result — and as of #249 that is confirmed rather than assumed. The apc_ups module now walks 1.3.6.1.4.1.318.1.1.1.7.2 alongside the standard UPS-MIB, and two rules watch it: UpsSelfTestScheduleOff fires the moment the schedule reads unknown, at-turn-on or never, and UpsSelfTestStale fires when the last-test date has not moved in 21 days.

That closes a gap worth naming for its shape rather than its size. The missing battery pack was visible in a MIB already walked — upsTestResultsSummary = 4 needed no new OIDs — and the missing schedule would not have been. upsTestResultsSummary would have held 1 forever and every rule would have stayed correctly quiet; UpsBatteryUnproven could not have caught it either, because it matches 6 (noTestsInitiated) and this card reads 1. A healthy-looking metric with nothing underneath it is the failure this estate keeps finding, one level up each time.

Why SNMPv2c is still a weak point

The devices are polled with SNMPv2c, which transmits the community string in cleartext, and whoever can see the poll's last hop can read it off a single packet. Two mitigations are in place, and the third is decided per device rather than for the fleet:

  • Done: each device has its own community, confirmed live on all four, so one captured packet no longer grants read access to the whole fleet. The switch does still accept its own previous community as well — an accepted residual, recorded in SECURITY.md.
  • Done: SNMP is reachable only on the management VLAN, the switch-management LAN and — for the iLO — the lab segment, and only from the monitoring host.
  • SNMPv3 authPriv, per poll (ADR-0036). The iLO's poll is the one that matters: it is delivered into ImaginationLAN, where ADR-0014 puts the attack VM on purpose, so a guest there that ARP-spoofs 10.0.30.10 reads the community every minute. shiva moves to v3 with SHA and AES and SNMPv1 off, then the UPS card on the same procedure. The firewall cannot move without losing what it is polled for: bsnmpd is the only daemon that serves the pf MIB, and pfSense writes no v3 user for it — checked on the box on 2026-09-09. The switch stays on v2c; its agent answers v3 on the wire, and whether its UI can create a user is the unchecked half. Which devices have actually moved is recorded on #85 and in the runbook.

These communities are read-only, but "read-only" on a firewall means the complete state table and interface topology. They are credentials — except on the switch, where the agent does not check a short one on GETBULK at all (SECURITY.md has the measurement). The two that stay on v2c ride on Winterfell only, where anything that can sniff is already on the segment that holds the firewall's admin UI and the monitoring host.

The switch's management UI is HTTP, and stays that way

neo serves its management UI on port 80 and nothing on 443 — no TLS listener, and no way to import a certificate. Checked against the device on 2026-09-04 and decided in ADR-0018, which gave the switch a name (neo.matrix.elysium) and closed the certificate half of #97 as unavailable rather than pending.

So the switch admin password crosses the wire in cleartext, and it is worth being precise about where: through neo itself, which is the device the password protects. A mirrored port or a foothold on the switch sees the credential to the switch. This is the same shape as the SNMP argument above and a sharper version of it, because this credential is read-write.

What holds it: the password is unique to the device, and only Hicks and Winterfell can reach 10.7.7.0/24 at all (ADR-0013). What does not hold it: anything on the device, which is now carrying its fourth firmware limit after the undeletable community row, an SNMP agent that serves GETBULK to any community of sixteen characters or fewer (measured 2026-09-12, #84), and an SNMPv3 user page it may or may not have (ADR-0036). A TLS management interface belongs in the selection criteria whenever this switch is replaced — and the replacement that has one, a MikroTik CRS326, was bought 2026-09-13 (hardware.md, #444), with delivery estimated 2026-09-23. This section changes when it is racked, not before — that date is a courier estimate rather than an arrival, and the swap takes the whole house offline and shares a rack visit, so it buys a window to pick rather than a date this closes on.

Hardening applied to the stack

  • Anonymous Grafana access disabled; sign-up disabled; admin password from SOPS.

  • Every service except Alloy runs as a non-root uid. Prometheus and blackbox-exporter run as nobody (65534), Loki and Grafana as their own unprivileged UIDs (10001, 472), and Alertmanager and snmp-exporter as the operator's ${RENDER_UID} because each mounts a 0600 file that scripts/render-config.sh wrote. blackbox-exporter needs its user: written out because its image sets none — it was uid 0 until #330, which is how it went unnoticed for as long as it did. Alloy is uid 0 on purpose and is covered separately below: it reads the host's logs, and the capability drop rather than the uid is what bounds it.

  • Outbound probes leave VLAN 99 for the public internet, and that is a deliberate new egress path rather than a side effect (#166). blackbox-exporter makes TCP connects to 1.1.1.1:443 and 8.8.8.8:53 every 30 seconds so the estate can answer "is the internet slow, and is it us or the ISP". They are connects only — nothing is sent, nothing about the estate is disclosed beyond the fact that this address exists, which the ISP knows anyway. tcp_connect rather than an ICMP prober specifically so that blackbox-exporter keeps cap_drop: [ALL]: an icmp module needs CAP_NET_RAW, and handing one container back a capability the rest of the stack just gave up is a poor trade for a measurement TCP already provides.

  • snmp-exporter is never published to a host interface — it is reachable only on the compose network.

  • Alertmanager binds to 127.0.0.1 only. It is unauthenticated, and a silence is how monitoring gets switched off — quietly, since the record lives in the system being switched off. Nothing off-host used the port; silences are reached through Grafana. Prometheus and Loki are not in this list: they stay published for oracle's agent and remain an accepted residual. See ADR-0012.

  • The Alloy debug UI binds to 127.0.0.1 only.

  • The Docker socket is no longer mounted into Alloy (#193). It was, marked :ro, which was worth less than it looked: read-only applies to the socket file, not to the API behind it, and anything that can talk to that API can start a container with the host filesystem mounted read-write. Alloy now reaches the API through docker-socket-proxy, which allowlists the reads its three components need — containers, images, info, version, events and networks — and refuses everything else. Verified rather than assumed: POST /containers/create through the proxy returns 403 Forbidden, while GET /containers/json works.

    This moves the trust boundary rather than removing it. The proxy container holds the socket now. What it buys is that Alloy — the component with a network listener, a read-only mount of the whole rootfs and by far the largest attack surface here — no longer has a path to POST. It is not "the socket is now safe".

    oracle's agent still mounts the socket directly. docker.alloy reads the API address from DOCKER_API and falls back to the socket when it is unset, so that host keeps working unchanged until it gets a proxy of its own.

  • Alloy holds no capabilities. It runs as uid 0 with cap_drop: [ALL] and no-new-privileges, so root inside it is subject to file permissions like any other user, and joins only the group that owns /var/log/syslog so the auth and syslog sources stay readable (#188). scripts/deploy-agent.sh applies the same flags to every Docker host it deploys to, so oracle's agent is no longer the privileged copy it was until #88; on Saruman the native package runs as its own unprivileged alloy user.

  • Every service runs with a read-only root filesystem (#186). Writable state is confined to the named volumes and, where a process genuinely needs scratch space, to a sized tmpfs: 64 MiB on Grafana, which extracts plugins into /tmp, and 32 + 16 MiB on Alloy for /tmp and /root/.cache. The sizes are set rather than defaulted because a tmpfs is memory, and the default is half of host RAM — an unbounded one would be a second memory budget beside the mem_limit each service already has.

    This was established by running, not by reading: a missing tmpfs is a crash loop that restart: unless-stopped retries forever, so each service was booted read-only from its pinned digest with its real mounts before the line was written. Grafana got the full soak the issue asked for — login, all three datasources, all seven dashboards fetched by uid, an annotation written and a folder created. Alloy is included too: #186 proposed skipping it as performative next to privileged: true, and that reason left with #188.

  • The other six services hold no capabilities either. Each carries cap_drop: [ALL] and no-new-privileges:true (#187). For five of them this is belt and braces and the honest description matters: they already ran non-root with an effective set of zero, so what the drop clears is the bounding set — the capabilities a process could regain, not any it held. What that buys is that a future image bump shipping a setuid or file-capability binary cannot be used to gain one, which no-new-privileges independently forecloses. None of the six pinned images contains such a binary today; each container's filesystem was exported and scanned for the setuid and setgid bits. blackbox-exporter was the exception when #187 landed: it was running as uid 0 with the full effective set, so there the drop removed something real. #330 then gave it a non-root uid as well, which puts it in the same belt-and-braces position as the other five — the position to be in, and not a reason to drop either setting.

  • Every service runs under a real init (init: true) and a chosen task ceiling (pids_limit, 512; 1024 for Alloy) rather than the inherited systemd default of 9056. This was not theoretical: Grafana's https healthcheck was leaking two unreaped ssl_client children every 30 seconds and would have exhausted the inherited ceiling about three days after each start (#71).

  • Prometheus carries a byte ceiling as well as a time one, so a change that quietly multiplies the series count cannot consume the disk unnoticed.

  • All images are pinned to explicit versions, so an upstream compromise cannot arrive silently via :latest. Dependabot proposes the bumps; CI validates them.

  • Grafana telemetry and update checks disabled.

Alloy no longer runs privileged: true. It never needed it: cgroup: host is what makes cAdvisor see the host's cgroups, and dropping every capability changed no container metric and cost six unused series — node_rapl_*_joules_total and node_cpu_{core,package}_throttles_total, which read root-only sysfs and which nothing here references. Before that change Alloy ran as uid 0 with the full capability set, a read-only mount of /, and could therefore read ~/.config/sops/age/keys.txt directly. It can no longer.

What remains is the Docker socket, and it is the larger half. Read access to that API is enough to create a container with / mounted read-write, which is root on the host and the age key with it — so the paragraph in SECURITY.md saying file permissions are all that protect the plaintext artefacts is true of every process on the host except a compromised Alloy. Putting the socket behind a proxy that permits only the handful of GETs cAdvisor and the log discovery actually use is tracked separately; the privilege reduction above is defence in depth, not a closed door.

A writable directory inside Alloy's read scope

Scheduling the maintenance jobs (#77) put /var/lib/node_exporter/textfile_collector on the host, owned by the deploying user and read by Alloy's textfile collector through the /rootfs mount that already existed. It is 0755 with 0644 files, and it has to be: Alloy runs as root with every capability dropped, so it obeys the mode like anyone else and a 0600 file would simply be invisible to it.

The contents are four numeric gauges per job — timestamps, a duration and an exit code. Nothing secret is written there, and nothing decrypted passes through it. What it does create is a path from write access as that user to arbitrary metric names and label values in Prometheus, since the collector will parse whatever it finds. Two things bound that: only the deploying user can write, and scripts/run-scheduled.sh constrains the one operator-supplied field to ^[a-z][a-z0-9-]{0,30}$ before it becomes either a filename or a label value.

Anyone who can write there can already run the jobs themselves, so this adds no privilege — but it is a new file-backed input to the metrics pipeline, and that is worth stating rather than discovering.

What this repository deliberately does not publish

Being able to describe a network precisely is useful; publishing a complete fingerprint of a house is not. Withheld on purpose:

  • Full MAC addresses. Truncated to the OUI, which keeps the useful information (vendor, and therefore what the device is) and drops the unique identifier. Full MACs enable device tracking and, on some networks, MAC-based access control bypass. This is also why the unknown-device rules hold no allowlist: the set of known devices is whatever Loki has seen in the last seven days, so nothing in this repository has to enumerate them.
  • Owner-linked device names. Personal devices are listed by role (laptop-01) rather than by person, and a child's bedroom is not labelled.
  • Camera-to-room mapping. Knowing there are seven cameras is fine. Knowing which one covers which door is a physical-security detail.
  • The WAN address, firewall rule bodies, and Wi-Fi configuration.
  • The WireGuard endpoint and its listen port, for the same reason and with the same instinct: a repository that withholds the WAN address and then publishes the port a VPN answers on has withheld nothing. The design, the peer subnet and the rules are published (ADR-0042); where to send a packet is not.

The public IP was already redacted in the original inventory — the rest of this is the same instinct applied consistently.

Rack patch-cable colours are published, and that is a deliberate exception worth defending rather than an oversight. It is the same shape of information as the camera-to-room mapping above — a physical-security detail — but not the same reach. A room mapping is useful remotely: you learn which camera covers which door straight off this page, without ever approaching the house. A cable colour is useful only to someone already standing at the rack, who can see the cables, read the switch port labels and reach the firewall's console port regardless. It tells an attacker nothing their position has not already given them, and it tells a maintainer a great deal. See ADR-0009.