fix(acme): stop punishing certificate issuance prepared before a DNS cutover

[mhenrixon]

Summary

Deploying a host before its DNS is repointed made the certificate arrive later, not sooner — so the working rule of thumb became "don't set it up in advance, do it at the exact moment of the cutover", which is the manual step automatic TLS exists to remove. Four defects produced that, all fixed here.

The trigger domain was never reachability-checked before an order. GetCertificate called provisionCertificate unconditionally for any registered host, and filterBatchMates deliberately exempted the triggering domain. Any HTTPS request carrying that SNI before the cutover — a smoke test, a monitor, a scanner — spent a real ACME order that could not succeed. Five in an hour trips Let's Encrypt's failed-authorization limit, and the hold that follows is what delayed the real cutover. Now the trigger is probed first: a failing probe refuses the handshake and records a cheap preflight hold having spent nothing.

Nothing lifted a hold when DNS landed. The ladder runs to 24h and persists to dynamic-domains.state, so a restart did not help either. A new release prober re-probes held domains every minute (--acme-release-probe-interval, negative to disable) and lifts the hold as soon as the domain routes back. A cutover now costs about a probe interval instead of a backoff step.

DNS-01 domains were HTTP-probed anyway. probeDomains skipped only wildcards, so a non-wildcard host in a DNS-01 zone was probed, failed while its DNS still pointed elsewhere, and was dropped from an order that would have succeeded — inverting the main advantage of DNS-01.

No visibility, no escape hatch. domains list reported only services with a domain source, so deploy-registered hosts were invisible while held. They are now listed with their certificate state and hold, holds render why they are held, and domains retry [domain] --all clears one on demand.

Closes #118

Design notes

The probe, not the quarantine, is the gate on the handshake path. That keeps the property the old unconditional pass was protecting — a just-repointed domain gets its certificate on the very next handshake rather than waiting out a stale ladder entry — while removing a burn that a quarantine check could never have stopped, because it happens on the first attempt, before any hold exists.

Holds now carry a kind, so the release prober can tell its own backoff from one the CA imposed. Rate-limit holds are never lifted early: that end time is Let's Encrypt's, the domain routing here again says nothing about it, and ordering inside the window both fails and pushes it further out. The persisted form is numeric so a state file predating the field decodes as an ACME hold, the conservative default; the status API renders it as a string.

A failing sweep records nothing. The domain is already held, and counting each sweep as a fresh failure would drive it to the top of the ladder within minutes purely for continuing to be in the state it was already held for.

Test plan

• go test ./... — green
• go test -race ./... — green (this adds a background goroutine)
• go test ./internal/server/ -run 'Quarantine|BatchGuard|DomainIssuer|CertRenewer|Release|Retry|RegisteredDomains' -count=2 — green, per the global-registry isolation problem from Global Prometheus registry breaks test isolation: -count=2 panics, cache-metrics assertions flake #104
• gofmt -l clean, go vet ./... clean, make build ok
• CLI smoke test: domains --help, domains retry --help, run --help showing the new flag
• Not done — manual end-to-end against the staging directory. See below.

New tests worth reading as the spec: TestBatchGuard_UnreachableTriggerIsRefusedWithoutBurningAnOrder, TestBatchGuard_ReachableTriggerProvisionsDespiteAStaleHold, TestBatchGuard_DNSSolvableTriggerIsNotProbed, TestReleaseProber_NeverLiftsARateLimitHold, TestReleaseProber_SkipsDomainsTheProbeCannotSpeakFor, TestReleaseProber_FailedProbeDoesNotEscalateTheLadder.

TestBatchGuard_TriggerDomainIsNeverDropped encoded the old contract and was rewritten, not deleted — the trigger is still never dropped from its own batch, but the handshake is now gated by a live probe upstream of it.

Deviations & judgment calls

Deviations from the issue

• The trigger gate went into provisionCertificate, not GetCertificate. GetCertificate is its only caller, so the two are equivalent in production — but provisionCertificate is directly callable from tests, whereas reaching GetCertificate's provisioning branch needs a tls.ClientHelloInfo carrying a real context, which crypto/tls gives no exported way to construct. One level down also covers the directory-mismatch synchronous reprovision path (--tls-staging flips), which enters through the same function.

• Lifting a hold uses a new Release, not the existing Clear. Clear wipes the failure count as well as the hold, which is right after a successful order but wrong here: a domain flapping between routing here and not would reset to the bottom of the ladder on every brief green probe and could loop against the CA indefinitely. Release zeroes Until, keeps Failures, and reports whether it actually held so callers only log real transitions. The manual domains retry deliberately still uses Clear — asking for a retry is a claim that the cause is fixed.

Judgment calls

• QuarantineStatus.Kind is a string on the wire, not the raw enum. Adding Kind to quarantineEntry broke the QuarantineStatus(entry) struct conversion, which only compiles while the two types are field-identical. Rather than mirror an unexported enum into an exported API type and emit "kind":2 to operators, the enum got a String() and QuarantineStatus is built explicitly.

• The release prober requests issuance only for dynamic domains. A deploy-registered host is issued on its handshake, and that path re-probes the trigger itself, so lifting the hold is all it needs — routing the two through different mechanisms would risk a background order racing a synchronous one for the same name. The practical cost is that a registered host's certificate arrives on the first visit after the cutover rather than strictly before it.

• stats labels changed from a bare Certified: to Dynamic certified: / Registered certified:, since there are now two counts and one unqualified label would have been ambiguous.

Discoveries

• The issuance guard is installed unconditionally whenever --acme-email is set, so the "behaviour unchanged when no guard is installed" fallback is a test-only concern in practice, not a production one.

• The renewal path already carried a cruder form of the DNS-01 guard: it probes only dynamic domains, on the reasoning that a registered host may be DNS-01-only. That rule is both too broad (never probes a registered HTTP-01 host, even when probing would help) and too narrow (always probes a dynamic domain, even one in a DNS-01 zone). The new predicate composes with it rather than replacing it. I did not widen the registered-host side — the renewal path is not what this issue is about, and changing it would alter renewal behaviour for hosts currently never probed. Worth a follow-up.

Not done

• The issue's manual staging-directory rehearsal (deploy a host pointing elsewhere, confirm no order is spent, repoint, confirm the hold lifts within a probe interval) has not been run — it needs a real domain and a live ACME staging account. Everything above is unit-level. I would not cut the v1.0.0.7 tag without it, since the gem's MINIMUM_VERSION gate makes that tag hard to walk back.

Follow-ups

• Gem-side operator surface: Pre-cutover DNS: operator surface for held certificate issuance dash#126 (dash proxy domains retry, release_probe_interval in deploy.yml, hold reporting in dash doctor). Blocked on this shipping and an image being published.
• Consider whether renewal should probe registered HTTP-01 hosts, per the discovery above.

---

Summary by cubic

Gates handshake-driven certificate issuance on a live preflight probe and adds an automatic release prober. Before, pre‑cutover handshakes spent failing ACME orders and hit rate limits; now we probe first, refuse unreachable triggers without spending an order, and lift holds as soon as DNS routes here.

New Features

• Re-probe held domains and lift holds on success; controlled by --acme-release-probe-interval (default 1m; negative disables).
• Holds record a kind (preflight, acme, rate_limited); rate-limited holds are never lifted early.
• domains list shows deploy-registered hosts with certificate state and hold reason/time.
• domains retry [domain] --all clears holds and immediately re-requests issuance.
• domains stats splits “Certified” into “Dynamic certified” and “Registered certified”.
• README documents deploying before a DNS cutover.

Bug Fixes

• Probe the trigger domain before ordering; unreachable triggers are held as preflight and the handshake is refused instead of burning an order.
• Release stale holds automatically when DNS points back; cutovers no longer wait out a backoff step.
• Skip HTTP probes for DNS-01 domains and wildcards; they remain in orders and are not wrongly dropped.
• Surface and manage holds for deploy-registered hosts, restoring visibility and a manual escape hatch.

Closes #118.

^{Written for commit ec4981d. Summary will update on new commits.}