fix(chart): topology-gate single-replica PDBs so single-node installs stay drainable (#560)

[shujaatTracebloc]

Problem

client/templates/jobs-manager-pdb.yaml and client/templates/mysql-pdb.yaml rendered minAvailable: 1 unconditionally (gated only on podDisruptionBudget.*.create, default true). For a single-replica workload, minAvailable: 1 means the eviction API permits zero voluntary disruptions (currentHealthy 1 − 1 = 0 < 1), so kubectl drain/cordon blocks forever. On a single-node install this wedges all node maintenance (manual k3s upgrade, node drain), and the pod has nowhere to reschedule anyway.

Fix — topology-gated tri-state create

The chart already distinguishes topologies via hostPath.enabled (true = single-node/bare-metal installer; false = multi-node CSI: AKS/EKS/OpenShift) — the "storage/topology mode" the issue calls for, and the same signal SINGLE_NODE already defaults from.

podDisruptionBudget.{mysql,jobsManager}.create becomes tri-state:

• null (default) = AUTO: render on multi-node CSI (hostPath.enabled=false), skip on single-node/bare-metal (hostPath.enabled=true)
• true = force render regardless of topology
• false = never render

Rationale for gating (vs maxUnavailable: 1 or a blanket off)

On multi-node CSI the PDB is a feature: it blocks a surprise autoscaler scale-down / node upgrade from silently evicting the only mysql/jobs-manager pod, forcing a deliberate operator drain of the single-replica stateful workload. Switching to maxUnavailable: 1 there would make it a no-op, and would also require rewriting the existing minAvailable-asserting unittest. On single-node the same PDB has no upside (nowhere to reschedule) and only wedges drains — so it is skipped. Topology-gating preserves the multi-node protection, keeps single-node drainable, and adds no existing-test rewrites. Default topology is CSI (hostPath.enabled=false), so default behavior is unchanged.

Changes

• client/templates/jobs-manager-pdb.yaml, client/templates/mysql-pdb.yaml — topology-gated guard; minAvailable: 1 retained where it renders.
• client/values.yaml — create: true → create: null with tri-state docs.
• client/values.schema.json — create type boolean → ["boolean","null"], default null.
• client/Chart.yaml — version/appVersion 1.9.39 → 1.9.40 (chart-version-guard, lockstep).
• client/tests/priority_class_pdb_test.yaml — added 7 cases (single-node skip, CSI render, force-on/off overrides); all existing tests kept.

Verification

• helm lint --strict — green on aks/bm/eks/oc profiles.
• helm template — renders on all 4; PDB docs: aks/eks/oc = 2, bm = 0.
• helm unittest — 7 new PDB tests pass; the local 5 failed / 5 errored are a pre-existing helm v4.2.0 schema-message artifact (identical on clean develop); CI pins helm-unittest 0.5.2.

Closes #560

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes when PDBs exist on bare-metal and node-local installs (operational drain behavior); default multi-node CSI behavior is preserved and overrides remain explicit.

Overview
Fixes #560: unconditional minAvailable: 1 PDBs on single-replica mysql and jobs-manager made one-node clusters undrainable (kubectl drain blocks forever with nowhere to reschedule).

Topology-gated AUTO mode: podDisruptionBudget.{mysql,jobsManager}.create is now tri-state (null / true / false). When null (new default), PDBs render only on multi-node topologies and are skipped on single-node installs. Single-node detection uses a new singleNode value (null → infer from hostPath.enabled; explicit true/false overrides), deliberately not env.SINGLE_NODE (GPU fallback can be true on fixed multi-node clusters).

The node-local k3d installer path now emits singleNode: true so local-path CSI (hostPath.enabled=false) is not misclassified as multi-node. Chart 1.9.43, schema/docs updated, and helm unittest cases cover auto/override/singleNode/env.SINGLE_NODE behavior.

^{Reviewed by Cursor Bugbot for commit 49b3b5a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3e7c699. Configure here.}

[LukasWodka]

Thanks Shujaat — the diagnosis and the core fix are right, and this is a clean, well-documented, well-tested change. Verified on my side:

• Both PDBs the chart ships (mysql-pdb.yaml, jobs-manager-pdb.yaml) are gated — there are no others.
• The tri-state ternary logic is correct: explicit create wins; otherwise AUTO renders iff not single-node; and explicit singleNode wins over hostPath.enabled.
• Default hostPath.enabled=false → multi-node CSI still renders the PDB, so HA behaviour and existing installs are unchanged; the default bare-metal single-node path (hostPath.enabled=true) now correctly skips the PDB and stays drainable — the primary #560 scenario. ✅
• Chart.yaml version+appVersion bumped in lockstep 1.9.42 → 1.9.43 (one patch above develop); version-gate + helm-lint + all four template renders + helm-unittest green.

One thing I'd like resolved before approving — it's the exact case Bugbot's first thread raised, and I think it's still live:

The node-local single-node install path stays undrainable, and the shipped docs claim it's handled when it isn't.

Trace of a TB_STORAGE_MODE=node-local install:

• scripts/lib/common.sh:743-746 forces AGENTS=0 + SERVERS=1 → one schedulable node.
• scripts/lib/install-client-helm.sh:1284-1285 emits hostPath.enabled: false (local-path CSI).
• Nothing emits singleNode — git grep singleNode scripts/ returns zero hits (it only appears in the five chart files).

So in the gate: singleNode=null → $isSingleNode = $hostPathMode = false → $create = true → a minAvailable:1 PDB renders over the 1-replica pod on the single node → kubectl drain / manual k3s upgrade blocks forever. Same wedge #560 is about.

Meanwhile client/values.yaml:141 and the singleNode description in client/values.schema.json both state, in the present tense, "The installer sets singleNode=true for the node-local single-node k3d path" — but it doesn't (not here, not on develop). The new unittest proves the chart skips the PDB when singleNode: true is passed, yet no producer passes it — so the test couples to the mechanism, not to the end-to-end requirement, and a real node-local install is still undrainable.

To be clear this is not a regression — node-local was already undrainable on develop (create defaulted to true) — so landing the hostpath fix now and fast-following is defensible. But the two present-tense doc claims shouldn't ship as written.

Options (my pick = 1):

1. Wire it here: in the node-local branch of install-client-helm.sh (right next to hostPath.enabled: false), also emit singleNode: true. One line, makes the doc true, and closes #560 for node-local too. Add a manifest regen if the R8 gate flags it.
2. Land the chart fix now; soften the two present-tense claims to "pending — see follow-up #NNN" and file the installer-wiring ticket so it isn't lost.
3. If the wiring already exists on another branch I've missed, point me at it and this collapses to a one-line doc fix.

Happy to re-review the moment the node-local input is actually produced (or the claim is softened). Nice work on the singleNode / env.SINGLE_NODE decoupling in the second iteration — that part reads exactly right.

[shujaatTracebloc]

@LukasWodka thanks for the thorough trace — you're right, and I went with option 1 (wire it here) in 49b3b5a.

• scripts/lib/install-client-helm.sh now emits singleNode: true in the node-local (TB_STORAGE_MODE=node-local) values branch, right next to hostPath.enabled: false. So a node-local single-node k3d install (AGENTS=0/SERVERS=1) now resolves isSingleNode=true → the single-replica PDBs are skipped → the node stays drainable. Closes PDBs minAvailable: 1 on single-replica workloads make the single node undrainable #560 for node-local too, not just bare-metal.
• The two present-tense doc claims (values.yaml, values.schema.json) are now true as written.
• Added an end-to-end bats test (install-client-helm declares singleNode: true in the node-local storage branch) so the coverage is tied to the producer, not just the chart mechanism. Regenerated scripts/manifest.sha256 for the shell edit.

git grep singleNode scripts/ now returns the emit; verified shellcheck + bats + manifest --check clean. Ready for re-review.

[LukasWodka]

Re-reviewed at head 49b3b5a. The node-local gap from my last pass is now genuinely closed end-to-end — approving.

What changed since my comment: commit 49b3b5a ("emit singleNode: true for the node-local storage mode") wires the producer I said was missing. In the TB_STORAGE_MODE=node-local branch of scripts/lib/install-client-helm.sh (now L1287-1291), right next to hostPath.enabled: false, the installer emits singleNode: true — Option 1, as hoped.

Verified the emit is actually consumed (not just present):

• install-client-helm.sh:1252 opens cat <<EOF > "$values_file" (unquoted heredoc → the $(if … node-local …) substitution at L1275 is evaluated), so singleNode: true lands in the generated values file.
• L1404-1408 pass that file via helm upgrade --install … --values "$values_file".
• Gate then computes $isSingleNode = true → $create = false → PDB skipped → the sole node stays drainable. The #560 wedge is gone for node-local, not just bare-metal.

The present-tense doc claims are now true. values.yaml and the singleNode schema description say the installer sets singleNode=true for node-local — and now it does. No longer aspirational.

Test is end-to-end across two layers now, not mechanism-only: the new bats test asserts the producer emits singleNode: true inside the local-path block, and the helm-unittest asserts the consumer skips the PDB on singleNode:true / renders on multi-node CSI / ignores env.SINGLE_NODE. Producer + consumer together cover the requirement.

Re-verified the rest from scratch:

• Only two PDBs ship (mysql-pdb.yaml, jobs-manager-pdb.yaml); both carry the identical tri-state gate. No ungated third.
• Default hostPath.enabled=false multi-node CSI still renders minAvailable:1 → HA/eviction protection preserved, existing installs unchanged. Bare-metal single-node and node-local both skip.
• singleNode correctly stays decoupled from env.SINGLE_NODE — and this matters concretely: the installer sets env.SINGLE_NODE: "true" for every path (L1273, the GPU→CPU fallback), so keying the PDB gate on it would have mis-skipped fixed multi-node clusters. The dedicated flag avoids that.
• Chart.yaml version+appVersion 1.9.42 → 1.9.43, lockstep, one patch above develop.
• Manifest R8: recomputed sha256 of the head install-client-helm.sh matches scripts/manifest.sha256 exactly; the bats file isn't manifest-covered, so nothing else needed.
• Both Bugbot threads resolved; CI all green (helm lint/unittest + 4 template renders + version-gate + installer suite), remaining skips are the usual no-python / trigger-gated jobs.

Clean, well-documented, well-tested. Nice work on the wiring and the singleNode/env.SINGLE_NODE decoupling. LGTM. 🚀