feat: move the ingress layer from ingress-nginx to Traefik

.env.example

@@ -24,7 +24,7 @@ KUBECONFIG=
 K8S_NAMESPACE_PREFIX=fpp-user
 
 # Cluster (required for setup wizard)
-# External IP of the cluster's Ingress-NGINX LoadBalancer
+# External IP of the cluster's Traefik LoadBalancer
 CLUSTER_INGRESS_IP=
 # Root domain managed by Cloudflare (subdomains will be created under this)
 CLUSTER_DOMAIN=

CLAUDE.md

@@ -49,7 +49,7 @@ See `.env.example` for all options. Key ones:
 | `DID_HOSTING_DID` / `DID_HOSTING_PRIVATE_KEY` | — | vtafarm-api's **own** keypair (`make gen-keypair`) for the DID-hosting control API, enrolled in a daemon's ACL with `role=admin`. Not anything a daemon issued, so one keypair serves every daemon it is enrolled in. There are deliberately no DID-hosting **URLs** here — see "Shared infrastructure comes from the platform stack" below |
 | `CLOUDFLARE_API_TOKEN` | — | Cloudflare API token (`Zone:DNS:Edit` permission) |
 | `CLOUDFLARE_ZONE_ID` | — | Cloudflare Zone ID for the user's root domain |
-| `CLUSTER_INGRESS_IP` | — | External IP of the cluster's Ingress-NGINX LoadBalancer |
+| `CLUSTER_INGRESS_IP` | — | External IP of the Traefik LoadBalancer |
 | `ACME_CLUSTER_ISSUER` | `letsencrypt-http01` | The same issuer in every environment — there is no staging variant. A staging certificate passes `tls_provision` and then crash-loops the mediator, because the components resolve each other's `did:webvh` over HTTPS and reject an untrusted chain. Every environment therefore shares Let's Encrypt's unraisable allowances (5 certs per identical name set per week), so keep iteration on domains we own |
 
 ## Project Structure
@@ -504,6 +504,46 @@ Assign the correct tag so it appears in the right group in Scalar:
 
 ## Kubernetes Design
 
+### The ingress controller is Traefik
+
+`internal/k8s/traefik.go` holds everything this API knows about it. Two things
+that were per-Ingress annotations under ingress-nginx are now settings on the
+controller itself (`k8s/tls/rke2-traefik-config.yaml`), which is why the
+Ingresses we create carry **no annotations at all** beyond the one below:
+
+- **HTTP→HTTPS** is an entrypoint redirect (`ports.web.redirections`), not
+  `ssl-redirect` per Ingress.
+- **The wildcard certificate** comes from the `default` TLSStore
+  (`k8s/tls/tlsstore-default.yaml`), not `--default-ssl-certificate`. The
+  TLSStore and the Secret it names must both live in Traefik's own namespace —
+  which is why `k8s/tls/certificate.yaml` issues into `kube-system` rather than
+  `cert-manager`. Get this wrong and every managed/platform hostname silently
+  serves Traefik's self-signed certificate, which doesn't fail until
+  `step_vta_register_dids` — the components resolve each other's `did:webvh`
+  over HTTPS and reject an untrusted chain.
+- Custom domains still name their own Secret in the Ingress `tls:` block; a
+  router-level certificate wins over the store.
+
+**The dids Ingress carries a `strip-forwarded-host` Middleware**, one per user
+namespace. The daemon warn-logs every request that claims a forwarded host from
+a peer outside its trusted-CIDR set, and an ingress puts `X-Forwarded-Host` on
+every request — so without this it logs a warning per request, forever.
+Stripping rather than trusting the ingress CIDR is deliberate: pod networking is
+flat, so a CIDR wide enough to cover the ingress also lets any other pod dictate
+the request host. Traefik can actually remove the header because it writes the
+X-Forwarded-* set at the entrypoint, before middlewares run; ingress-nginx
+emits its own `proxy_set_header` ahead of any snippet and sends **both** values,
+so the same fix was impossible there.
+
+`ingressClassName` is hardcoded (`k8s.IngressClass`), not configurable — an
+environment on a different controller would also need different entrypoint
+settings, a different default certificate and a different ACME solver class, so
+one env var could never carry the switch.
+
+Sessions created before the switch keep `ingressClassName: nginx` forever (we
+only ever create Ingresses, never update them):
+`scripts/migrate-ingress-to-traefik.sh` repoints them, dry-run by default.
+
 ### Per-User Namespace Isolation
 
 Every user gets their own namespace: `vtafarm-user-{userID}`.
@@ -587,4 +627,7 @@ rules:
 - apiGroups: ["cert-manager.io"]
   resources: ["certificates"]
   verbs: ["get", "list", "watch", "create", "delete"]
+- apiGroups: ["traefik.io"]
+  resources: ["middlewares"]
+  verbs: ["get", "list", "create", "delete"]
 ```

README.md

@@ -104,7 +104,7 @@ Copy `.env.example` and adjust as needed:
 | `DB_NAME` | `vtafarm` | |
 | `JWT_SECRET` | _(required)_ | HS256 signing secret — must match the team, see below |
 | `ORCHESTRATOR_RESUME` | `true` | Re-attach interrupted sessions at startup. Set `false` locally — see [`docs/shared-dev-database.md`](docs/shared-dev-database.md) |
-| `CLUSTER_INGRESS_IP` | _(required)_ | External IP of the cluster's Ingress-NGINX LoadBalancer |
+| `CLUSTER_INGRESS_IP` | _(required)_ | External IP of the cluster's Traefik LoadBalancer |
 | `CLOUDFLARE_API_TOKEN` | _(optional)_ | Required for VTA setup wizard |
 | `CLOUDFLARE_ZONE_ID` | _(optional)_ | Required for VTA setup wizard |
 | `KUBECONFIG` | _(empty)_ | Auto-detects `~/.kube/config` when empty |
@@ -133,8 +133,8 @@ The production stack is deployed to a Kubernetes cluster (RKE2) via Helm.
 ### 1. TLS — Wildcard Certificate via cert-manager (one-time cluster setup)
 
 All VTA sessions share a single `*.firstperson.dev` wildcard certificate managed by
-cert-manager. nginx-ingress serves it as the default SSL certificate, so no
-per-Ingress TLS configuration is needed.
+cert-manager. Traefik serves it as its default certificate, so no per-Ingress TLS
+configuration is needed.
 
 #### Step 1 — Create the Cloudflare API token Secret
 
@@ -169,20 +169,66 @@ TXT record to Cloudflare, then removes it) and store the issued certificate.
 Check status with:
 
 ```bash
-kubectl get certificate -n cert-manager firstperson-dev-wildcard
+kubectl get certificate -n kube-system firstperson-dev-wildcard
 ```
 
-#### Step 4 — Configure nginx-ingress to use the wildcard cert by default
+The Certificate issues into `kube-system` — Traefik's own namespace — because a
+TLSStore can only reference a Secret alongside it. Adjust both files if Traefik
+runs elsewhere in your cluster.
 
-RKE2 manages its built-in nginx ingress controller via the `HelmChartConfig` CRD.
+#### Step 4 — Configure Traefik
+
+RKE2 manages its bundled ingress controller through the `HelmChartConfig` CRD.
+Two objects: the controller's entrypoints, and the default certificate.
 
 ```bash
-kubectl apply -f k8s/tls/rke2-ingress-nginx-config.yaml
+kubectl apply -f k8s/tls/rke2-traefik-config.yaml
+kubectl apply -f k8s/tls/tlsstore-default.yaml
 ```
 
-RKE2 will reconcile the change and restart the ingress controller automatically.
-After this, every VTA Ingress gets HTTPS automatically — no `tls:` block or
-cert-manager annotation required on individual Ingress resources.
+RKE2 reconciles the change and restarts Traefik automatically. After this every
+Ingress gets HTTPS and an HTTP→HTTPS redirect with no annotation, `tls:` block
+or cert-manager annotation of its own.
+
+Verify before going further — this is the step whose failure shows up several
+minutes later as a mediator crash loop rather than as a TLS error.
+
+**Test against the origin, not the hostname.** Managed and platform records are
+*proxied* through Cloudflare, so plain `curl https://<hostname>` reports
+Cloudflare's edge certificate (issuer: Google Trust Services) and Cloudflare's
+status code — it tells you nothing about the cluster. Pin the node IP:
+
+```bash
+IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
+
+# 1. The certificate the cluster itself serves — Let's Encrypt, not TRAEFIK DEFAULT CERT
+openssl s_client -connect "$IP":443 -servername <a dids hostname> </dev/null 2>/dev/null \
+  | openssl x509 -noout -issuer
+
+# 2. Routing. Pick a path the component actually serves: / on dids, /health on a
+#    VTA. A 404 on / from a VTA is correct and means nothing is wrong.
+curl -skI --resolve <a dids hostname>:443:"$IP" https://<a dids hostname> | head -1
+
+# 3. The redirect really applied (see the note in rke2-traefik-config.yaml —
+#    a mistyped values path fails silently)
+kubectl -n kube-system get ds rke2-traefik \
+  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep redirections
+```
+
+Note the ADDRESS column of `kubectl get ingress` stays **empty** under Traefik in
+this layout, and that is not a fault — see the `publishedService` note in
+`k8s/tls/rke2-traefik-config.yaml`. Read the CLASS column instead.
+
+#### Step 5 — Migrating an existing cluster off ingress-nginx
+
+Only when the cluster already ran sessions. vtafarm-api never updates an Ingress
+after creating it, so pre-existing ones keep `ingressClassName: nginx` and
+Traefik ignores them:
+
+```bash
+KUBE_CONTEXT=k8s-fpp-dev ./scripts/migrate-ingress-to-traefik.sh           # dry run
+KUBE_CONTEXT=k8s-fpp-dev ./scripts/migrate-ingress-to-traefik.sh --apply
+```
 
 ### 2. HashiCorp Vault (one-time, before the API)
 

docs/custom-domain-design.md

@@ -107,9 +107,9 @@ what makes the platform stack nearly free:
 
 `vta-<vta_name>.firstperson.dev` and friends. vtafarm-api creates four
 **proxied** Cloudflare A records at session-create time. TLS comes from the
-cluster-wide `*.firstperson.dev` wildcard that ingress-nginx serves as its
-`default-ssl-certificate`. No Domains record involved, nothing for the user to
-do.
+cluster-wide `*.firstperson.dev` wildcard that Traefik serves as its default
+certificate (the `default` TLSStore). No Domains record involved, nothing for
+the user to do.
 
 ### 3.2 `custom` — new
 
@@ -771,7 +771,7 @@ spec:
     solvers:
     - http01:
         ingress:
-          class: nginx
+          ingressClassName: traefik
 ```
 
 **One issuer, every environment — there is deliberately no staging twin.**
@@ -845,11 +845,11 @@ aaa.com allows letsencrypt.org.
 
 | Option | Verdict |
 | --- | --- |
-| **Cloudflare for SaaS** — users CNAME to a proxied `lb`, Cloudflare issues and renews the edge certificate | **Deferred.** Free at this scale (100 custom hostnames included on every plan, then $0.10/mo each) and it would hide the origin IP. But origin-side TLS is unresolved: with a fallback origin, Cloudflare sends SNI = the custom hostname, our nginx answers with the wildcard, and Full (strict) rejects it. Fixing that means either downgrading the whole zone to Full, or running cert-manager **as well** — i.e. this option plus all of §8. Parked in §16.3; §4.1 keeps the migration path free. |
+| **Cloudflare for SaaS** — users CNAME to a proxied `lb`, Cloudflare issues and renews the edge certificate | **Deferred.** Free at this scale (100 custom hostnames included on every plan, then $0.10/mo each) and it would hide the origin IP. But origin-side TLS is unresolved: with a fallback origin, Cloudflare sends SNI = the custom hostname, our ingress answers with the wildcard, and Full (strict) rejects it. Fixing that means either downgrading the whole zone to Full, or running cert-manager **as well** — i.e. this option plus all of §8. Parked in §16.3; §4.1 keeps the migration path free. |
 | **LE DNS-01 with `_acme-challenge` delegation** | Rejected — 4 more records for the user, and its benefits (no port-80 dependency, wildcard support) are ones we don't need. |
 | **LE DNS-01 with the user's DNS API credentials** | Rejected — a large trust ask, and provider-specific. |
 | **User-supplied certificates** | Rejected — manual renewal every 90 days. |
-| **Caddy on-demand TLS** | Rejected — the slickest answer for arbitrary hostnames, but it means replacing RKE2's bundled ingress-nginx. |
+| **Caddy on-demand TLS** | Rejected — the slickest answer for arbitrary hostnames, but it means replacing the bundled ingress controller. (Written when that was ingress-nginx; the cluster moved to Traefik afterwards, for unrelated reasons, and the verdict is unchanged — Caddy would still replace it.) |
 
 ### 8.6 On the origin IP
 
@@ -862,7 +862,7 @@ openssl s_client 157.180.68.139:443    → CN=*.firstperson.dev
 curl --resolve dids-…:443:157.180.68.139  → HTTP 200   (identical to via-Cloudflare)
 ```
 
-ingress-nginx serves the wildcard certificate to *any* direct connection on
+The ingress serves the wildcard certificate to *any* direct connection on
 :443, so internet-wide scanners (Censys, Shodan) already index the
 IP ↔ domain link. Choosing direct-to-origin therefore gives up nothing that is
 currently held. The real control would be firewalling the origin to

docs/full-stack-setup-design.md

@@ -178,8 +178,8 @@ A   vtc-{vtc_name}.{domain}        →  {CLUSTER_INGRESS_IP}
 
 DNS must exist first because the rendered recipes embed the final `https://…` URLs
 (`public_url`, `webvh_url`, `[identity].public_url`, the VTC's `base_url`) into the DID
-documents that get published. TLS is the cluster-wide wildcard default-ssl-certificate on
-nginx-ingress (same as today's VTA Ingress — no per-host `tls:` block needed).
+documents that get published. TLS is the cluster-wide wildcard, served by Traefik as its
+default certificate (same as today's VTA Ingress — no per-host `tls:` block needed).
 
 `CreateARecord` already returns a record ID; store all four for teardown.
 

docs/vta-setup-design.md

@@ -165,7 +165,7 @@ The backend calls the Cloudflare API to create DNS records **before** running an
 | --- | --- |
 | `CLOUDFLARE_API_TOKEN` | Cloudflare API token with `Zone:DNS:Edit` permission |
 | `CLOUDFLARE_ZONE_ID` | Zone ID for the user's root domain (from Cloudflare dashboard) |
-| `CLUSTER_INGRESS_IP` | External IP of the cluster's Nginx/Ingress-NGINX LoadBalancer |
+| `CLUSTER_INGRESS_IP` | External IP of the cluster's Traefik LoadBalancer |
 | `CLUSTER_DOMAIN` | Root domain the generated subdomain is appended to (e.g. `example.com`) |
 
 ### DNS Records Created
@@ -209,8 +209,8 @@ The returned record ID is stored on the session (`CFRecordID`) and used for tear
 ## Kubernetes Resource Provisioning
 
 All resources are created in the user's isolated namespace (`vtafarm-user-{userID}`). TLS is
-served by the cluster-wide **wildcard default-ssl-certificate** on nginx-ingress — there is no
-cert-manager and no per-host `tls:` block.
+served by the cluster-wide wildcard, which Traefik serves as **its default certificate** (the
+`default` TLSStore) — there is no per-host `tls:` block and nothing per-Ingress to configure.
 
 ### Resources (VTA Only)
 
@@ -221,7 +221,7 @@ cert-manager and no per-host `tls:` block.
 | `Job` (setup) | `vta-setup-{sessionID}` | runs `vta setup --from …` |
 | `Job` (provision) | `vta-provision-{sessionID}` | runs `vta import-did` (+ `did-mgmt servers add`) |
 | `Service` | `vta-{sessionID}` | ClusterIP on 8100 |
-| `Ingress` | `vta-{sessionID}` | nginx ingress (ssl-redirect annotation; wildcard TLS) |
+| `Ingress` | `vta-{sessionID}` | Traefik ingress (no annotations; wildcard TLS + redirect both come from the controller) |
 | `Deployment` (server) | `vta-{sessionID}` | long-running VTA, starts after the provision Job |
 
 ### Ingress template
@@ -232,10 +232,8 @@ kind: Ingress
 metadata:
   name: vta-{sessionID}
   namespace: vtafarm-user-{userID}
-  annotations:
-    nginx.ingress.kubernetes.io/ssl-redirect: "true"
 spec:
-  ingressClassName: nginx
+  ingressClassName: traefik
   rules:
     - host: {subdomain}.{CLUSTER_DOMAIN}
       http:

helm/vtafarm-api/templates/vtafarm-api/clusterrole.yaml

@@ -60,3 +60,9 @@ rules:
   - apiGroups: ["cert-manager.io"]
     resources: ["certificates"]
     verbs: ["get", "list", "watch", "create", "delete"]
+  # One Middleware per user namespace, stripping X-Forwarded-Host on the way to
+  # the dids daemon. Namespaced and not cross-namespace referenceable, so it
+  # cannot be a single shared object.
+  - apiGroups: ["traefik.io"]
+    resources: ["middlewares"]
+    verbs: ["get", "list", "create", "delete"]

helm/vtafarm-api/templates/vtafarm-api/ingress.yaml

@@ -3,7 +3,7 @@ kind: Ingress
 metadata:
   name: {{ .Values.name }}
 spec:
-  ingressClassName: nginx
+  ingressClassName: traefik
   rules:
     - host: {{ .Values.ingress.host }}
       http:

internal/k8s/component_resources.go

@@ -223,28 +223,27 @@ type ComponentIngressSpec struct {
 	Namespace, Name, ServiceName, Host string
 	Port                               int32
 	// TLSSecret names the Secret serving this host's certificate. Empty selects
-	// the cluster-wide wildcard that ingress-nginx serves as its
-	// default-ssl-certificate — which covers every managed and platform
-	// hostname, so only custom domains ever set this.
+	// the cluster-wide wildcard Traefik serves as its default certificate —
+	// which covers every managed and platform hostname, so only custom domains
+	// ever set this.
 	TLSSecret string
+	// StripForwardedHost attaches the namespace's strip-forwarded-host
+	// Middleware. Only the dids daemon needs it (see
+	// StripForwardedHostMiddleware); the caller must have created the
+	// Middleware first.
+	StripForwardedHost bool
 }
 
-// CreateComponentIngress creates an nginx Ingress routing Host to ServiceName
-// on Port. Idempotent.
+// CreateComponentIngress creates an Ingress routing Host to ServiceName on
+// Port. Idempotent.
 func (c *Client) CreateComponentIngress(ctx context.Context, spec ComponentIngressSpec) error {
 	pathType := networkingv1.PathTypePrefix
-	ingressClass := "nginx"
+	ingressClass := IngressClass
 
 	ingress := &networkingv1.Ingress{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      spec.Name,
 			Namespace: spec.Namespace,
-			Annotations: map[string]string{
-				// cert-manager's HTTP-01 solver claims the more specific
-				// /.well-known/acme-challenge/ path, so this doesn't interfere
-				// with issuance.
-				"nginx.ingress.kubernetes.io/ssl-redirect": "true",
-			},
 			// Deliberately NO cert-manager.io/cluster-issuer annotation. We
 			// create the Certificate ourselves (one covering all four hosts);
 			// leaving the annotation here as well would have ingress-shim make
@@ -280,6 +279,16 @@ func (c *Client) CreateComponentIngress(ctx context.Context, spec ComponentIngre
 			SecretName: spec.TLSSecret,
 		}}
 	}
+	if spec.StripForwardedHost {
+		// Traefik fails the router outright when this names a Middleware that
+		// does not exist, so the object has to be in place before the Ingress —
+		// which is why the caller creates it, rather than this function doing it
+		// on the way past.
+		ingress.Annotations = map[string]string{
+			"traefik.ingress.kubernetes.io/router.middlewares": MiddlewareRef(
+				spec.Namespace, StripForwardedHostMiddleware),
+		}
+	}
 
 	_, err := c.kube.NetworkingV1().Ingresses(spec.Namespace).Create(ctx, ingress, metav1.CreateOptions{})
 	if err != nil && !k8serrors.IsAlreadyExists(err) {