Retry OIDC provisioner initialization instead of disabling it

[AWildLeon]

An OIDC provisioner gets the openid-configuration and the JWK key set of the identity provider on Init. When the provider could not be reached, for example because step-ca started before the network was up, the provisioner was replaced by an Uninitialized one and never recovered: OIDC authentication kept failing until step-ca was restarted.

Init now marks a failure to reach the identity provider as temporary, and the provisioner gets the configuration again when it is used, using an exponential backoff between attempts. Any other initialization error, e.g. an invalid configuration, still disables the provisioner.

Fixes #2724

Name of feature:

Automatic recovery of OIDC provisioners after a temporary initialization failure.

Pain or issue this feature alleviates:

step-ca reaches its identity provider once, at startup. On a machine where step-ca starts before the network is usable — a common systemd ordering problem on homelab and cloud hosts — that single attempt fails and the provisioner is permanently replaced by an Uninitialized one. The process keeps running and looks healthy from the outside, /health is fine, systemd reports the unit as active, but every OIDC authentication returns provisioner "…" is disabled due to an initialization error forever. The only fix is a manual restart, and nothing in the running service indicates that a restart is what's needed.

After this change the provisioner retries on its own and the first login after connectivity returns succeeds.

Why is this important to the project (if not answered above):

It removes a class of silent, restart-only failures from the operator's plate. There is no configuration to add and no new failure mode: an unreachable provider is now a self-healing condition rather than a permanent one, while genuinely broken configuration still fails loudly at startup exactly as before.

Is there documentation on how to use this feature? If so, where?

None needed — no configuration surface changes and nothing for a user to enable. Three observable behaviours are worth knowing:

• The startup log line for an unreachable provider now reads failed to initialize OIDC provisioner "name", it will be retried when the provisioner is used: … instead of failed to initialize OIDC provisioner "name": ….
• While uninitialized, requests get 503 with oidc provisioner 'name' is not initialized, instead of 401 provisioner "name" is disabled due to an initialization error.
• Backoff is 5s, doubling to a 5m ceiling (oidcInitRetryInterval / oidcMaxInitRetryInterval in authority/provisioner/oidc.go). Init deliberately leaves the deadline unset, so the very first request after startup retries immediately — a brief boot-time blip recovers on the first login attempt rather than after a delay.

Happy to add a note to the docs site if maintainers want the changed log line and status code written down.

In what environments or workflows is this feature supported?

• Any step-ca with one or more OIDC provisioners, from ca.json or from the admin database / linkedca.
• All OIDC token paths, since the retry hangs off authorizeToken: x509 sign, SSH sign, revoke and SSH revoke.
• Recovery is driven by incoming requests, so it works the same under systemd, Docker and Kubernetes with no goroutine or lifecycle hook to manage.

In what environments or workflows is this feature explicitly NOT supported (if any)?

• OIDC only. It is the only provisioner type that performs network I/O in Init. The ErrRetryInit contract is general, so another provisioner could opt in later, but none does today.
• Permanent errors still disable the provisioner. An empty clientID, an unparseable endpoint, bad claims — all still produce Uninitialized, unchanged.
• Admin API create/update is unchanged. StoreProvisioner and UpdateProvisioner still reject a provisioner whose endpoint is unreachable, so an admin gets immediate feedback rather than a silently deferred failure.
• Retries are request-driven, not background. A provisioner nobody uses never retries; the first request after the backoff expires pays one connect timeout.
• No strict startup mode. Points 2 and 3 of the issue — fail the service if a provisioner cannot initialize, and surface that to systemd — are not implemented. They were workarounds for the missing recovery path, and a strict mode is a new global config surface deserving its own discussion. Glad to follow up if it is still wanted.
• Loss of Uninitialized state for this case. An unreachable OIDC provider no longer shows state: Uninitialized in the admin API, because the provisioner stays enabled. The condition is still visible in the startup log and in the 503 on every request.

Supporting links/other PRs/issues:

• Fixes OIDC provisioner unavailable after transient startup failure #2724

Notes for review

• Adds one exported symbol, provisioner.ErrRetryInit, alongside the existing ErrAllowTokenReuse / ErrNotImplemented sentinels. Init still returns an error on an unreachable provider, so callers that only check err != nil behave as before; authority.ReloadAdminResources uses errors.Is to decide whether to disable.
• configuration and keyStore moved under an initMutex, written only while holding it and read only after a successful ensureInitialized, which also serializes concurrent authorization attempts during an outage instead of letting them stampede the provider.
• Tests: unreachable-provider recovery, backoff saturation, a concurrent-initialization test under -race, and an authority-level test that an unreachable provider stays enabled while a permanent error still disables. Verified with the full suite, go vet, gofmt, and golangci-lint using the shared smallstep config, plus an end-to-end run of the built binary against a mock provider started after the CA, with a control run on master confirming it never recovers.

This code is AI-generated. While I kind of understand the codebase (a little), I'm not a Go developer but I tested it.