certificates/ca-key.pem — the estate's root, the one that signs Grafana's
two leaves — is the only root secret in the estate with no custody story. The
age key has ADR-0024 and back-up-the-age-key.md: a second recipient off-host
and off-estate, proved on demand against the real ciphertext, nagged by
SecretsKeyBackupUnproven at ninety days. The tier's root goes to the same
offline medium by build-the-tier-ca.md. The estate's root has nothing — not
a copy, not a sentence.
Found while deciding #436, and recorded in
ADR-0043
decision 4 with the constraints below, so that this issue starts from them
rather than re-litigating them.
What the loss actually costs
Bounded, not fatal — and worth stating accurately, because the age key's
"unbounded downside" does not apply here. The root is pathlen:0, so there is
no intermediate to keep issuing; but there is a re-mint path, and it is
already written: make certs ARGS='--ca --force', two leaves reissued, one
scp -3 through the Mac for alexander's, and a re-trust everywhere the old
ca.pem was imported (successor-handover.md, ADR-0037's cost table).
Two things about that path are the real gap:
- The trust-distribution list is unknown. ADR-0037: "the Mac's system
store and Firefox's separate one, at least, and whatever else nobody wrote
down." A re-trust of unknown size is the defect.
- The deadline is not 825 days. It is the next time a leaf is needed —
the next VLAN 30 service that wants TLS.
What the answer is not
- Not SOPS-in-git. ADR-0037 rejected that for the tier's root: "a
passphrase-encrypted CA private key in git is the exact artefact that was
purged from this repository's history." This key is one of the ones purged.
- Not a passphrase.
gen-certs.sh closes that trade deliberately, in its
header, and says how to reopen it if you disagree. This issue does not.
So the control is medium, not cryptography: one copy, on the offline medium
that already holds the age key's second copy and the tier's root, and nowhere
else. back-up-the-age-key.md's disqualifiers apply as written — not on the
monitoring host, not in a git tree, not in a synced folder.
What the proof would be
The age key's proof decrypts real ciphertext. A CA key's proof needs no
decryption at all: the public half of the backup against the public half of
ca.pem.
openssl rsa -in <backup> -pubout 2>/dev/null | sha256sum
openssl x509 -in certificates/ca.pem -pubkey -noout | sha256sum
Equal, or the copy is not this key. A sibling of scripts/verify-key-backup.sh
with the same refusal to run against the live key by device and inode, the
same "prints no secret" property, and a homelab_*_last_proof_timestamp_seconds
series so that a CaKeyBackupUnproven rule can nag the way ADR-0024's does.
One sentence from the ADR justifies the split: an age key has no re-mint path
and a CA does, which is why this key gets the age key's backup first and its
alerting second.
Order of work
- The copy, and the sentence in
generate-certificates.md's file table that
says it exists — plus successor-handover.md, where "both live only on
prometheus" stops being true for one of the two.
- Write down where
ca.pem is trusted, so the re-mint path has a list.
- The proof script and its series.
- The rule.
Refs #436, ADR-0043, ADR-0024, ADR-0037.
Corrected 2026-09-20
Steps 2–4 of the order of work are delivered by #549, merged this morning: the trust table in generate-certificates.md §4, scripts/verify-ca-key-backup.sh (public-key hash comparison, refuses the live key by device and inode, nine fixtures under make validate), scripts/ca-key-state.sh writing homelab_ca_key_backup_last_proof_timestamp_seconds, and CaKeyBackupUnproven in backup.rules.yaml with unit tests. homelab-ca-key-state.timer is installed on the monitoring host (first run 09:43 UTC today), so the rule fires within the hour on a 0 row — which is the honest state.
What is left is step 1, and it is yours with the medium in hand: the copy onto the offline medium that already holds the age key's second copy and the tier's root, then make certs-verify-backup KEY=<the copy>, then the two Mac rows in generate-certificates.md §4 confirmed from the Mac rather than from ADR-0037. The runbook is back-up-the-ca-key.md. This closes when the proof has run once and the alert clears; nothing merged shuts it.
certificates/ca-key.pem— the estate's root, the one that signs Grafana'stwo leaves — is the only root secret in the estate with no custody story. The
age key has ADR-0024 and
back-up-the-age-key.md: a second recipient off-hostand off-estate, proved on demand against the real ciphertext, nagged by
SecretsKeyBackupUnprovenat ninety days. The tier's root goes to the sameoffline medium by
build-the-tier-ca.md. The estate's root has nothing — nota copy, not a sentence.
Found while deciding #436, and recorded in
ADR-0043
decision 4 with the constraints below, so that this issue starts from them
rather than re-litigating them.
What the loss actually costs
Bounded, not fatal — and worth stating accurately, because the age key's
"unbounded downside" does not apply here. The root is
pathlen:0, so there isno intermediate to keep issuing; but there is a re-mint path, and it is
already written:
make certs ARGS='--ca --force', two leaves reissued, onescp -3through the Mac foralexander's, and a re-trust everywhere the oldca.pemwas imported (successor-handover.md, ADR-0037's cost table).Two things about that path are the real gap:
store and Firefox's separate one, at least, and whatever else nobody wrote
down." A re-trust of unknown size is the defect.
the next VLAN 30 service that wants TLS.
What the answer is not
passphrase-encrypted CA private key in git is the exact artefact that was
purged from this repository's history." This key is one of the ones purged.
gen-certs.shcloses that trade deliberately, in itsheader, and says how to reopen it if you disagree. This issue does not.
So the control is medium, not cryptography: one copy, on the offline medium
that already holds the age key's second copy and the tier's root, and nowhere
else.
back-up-the-age-key.md's disqualifiers apply as written — not on themonitoring host, not in a git tree, not in a synced folder.
What the proof would be
The age key's proof decrypts real ciphertext. A CA key's proof needs no
decryption at all: the public half of the backup against the public half of
ca.pem.Equal, or the copy is not this key. A sibling of
scripts/verify-key-backup.shwith the same refusal to run against the live key by device and inode, the
same "prints no secret" property, and a
homelab_*_last_proof_timestamp_secondsseries so that a
CaKeyBackupUnprovenrule can nag the way ADR-0024's does.One sentence from the ADR justifies the split: an age key has no re-mint path
and a CA does, which is why this key gets the age key's backup first and its
alerting second.
Order of work
generate-certificates.md's file table thatsays it exists — plus
successor-handover.md, where "both live only onprometheus" stops being true for one of the two.ca.pemis trusted, so the re-mint path has a list.Refs #436, ADR-0043, ADR-0024, ADR-0037.
Corrected 2026-09-20
Steps 2–4 of the order of work are delivered by #549, merged this morning: the trust table in
generate-certificates.md§4,scripts/verify-ca-key-backup.sh(public-key hash comparison, refuses the live key by device and inode, nine fixtures undermake validate),scripts/ca-key-state.shwritinghomelab_ca_key_backup_last_proof_timestamp_seconds, andCaKeyBackupUnproveninbackup.rules.yamlwith unit tests.homelab-ca-key-state.timeris installed on the monitoring host (first run 09:43 UTC today), so the rule fires within the hour on a 0 row — which is the honest state.What is left is step 1, and it is yours with the medium in hand: the copy onto the offline medium that already holds the age key's second copy and the tier's root, then
make certs-verify-backup KEY=<the copy>, then the two Mac rows ingenerate-certificates.md§4 confirmed from the Mac rather than from ADR-0037. The runbook isback-up-the-ca-key.md. This closes when the proof has run once and the alert clears; nothing merged shuts it.