Skip to content

docs(dha): PoC finding — CNPG scale subresource has no selector; reopen actuation mechanism #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
scooby87 wants to merge 1 commit into
base: main
Choose a base branch
from
+22 −2
Open

docs(dha): PoC finding — CNPG scale subresource has no selector; reopen actuation mechanism #53

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions design-proposals/database-horizontal-autoscaling/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,37 @@
- **Title:** `Database Horizontal Autoscaler for Cozystack`
- **Author(s):** `@scooby87`
- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit
- **Status:** Draft
- **Status:** Draft — mechanism reopened by the `2026-08-10` PoC finding below
- **PoC:** `2026-08-10` — live validation surfaced a blocking constraint in CNPG; see [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector)

## Overview

This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component.

The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder.

> **Note (2026-08-10):** a live PoC validated the metric side of this design but found that a stock HPA cannot drive the CNPG `Cluster` scale subresource (it exposes no selector), which reopens the actuation mechanism. See [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) below; the sections after it describe the mechanism as it stood before that finding.

### Why this changed

This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it.
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step reused one thing Kubernetes cannot actually provide here.

## PoC finding (2026-08-10) — blocking: the CNPG scale subresource exposes no selector

A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated the metric side of this design but uncovered a constraint the mechanism above does not survive as written.

**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA.

**The blocker.** That HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. The CNPG `Cluster` `/scale` subresource returns only `status: {replicas: N}` — no `status.selector` — and the CRD declares no `labelSelectorPath`. The Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so this fails for **every** target type (confirmed with both `AverageValue` and `Value`). It is not a calibration detail and not fixable by a version bump: upstream CNPG issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923), which requested exactly this selector for HPA/KEDA, is **closed as not planned**, and the CNPG 1.30 docs explicitly recommend against HPA for a `Cluster`.

**Consequence.** The load-bearing mechanism of this revision — a stock HPA (via KEDA) driving the CNPG `Cluster` scale subresource — cannot be built on stock CNPG; an actuation bridge is required after all. Crucially, what returns is **not** the machinery that got rev1 rejected: the ownership/enforcement layer (SSA, marker annotation, HelmRelease webhook, terminal-freeze) existed only because the chart declared `replicas`, and §3 (the chart omitting the field under autoscaling) removes it regardless of mechanism. What returns is only the small write-the-count actuator.
Comment on lines +27 to +29

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate relevant proposal files =="
git ls-files | grep -E 'design-proposals/database-horizontal-autoscaling|database-horizontal-autoscaling|Cozystack|cozystack|v45|45' || true

echo
echo "== proposal excerpt =="
if [ -f design-proposals/database-horizontal-autoscaling/README.md ]; then
  sed -n '1,120p' design-proposals/database-horizontal-autoscaling/README.md | nl -ba
fi

echo
echo "== search for CNPG versions/support matrix mentions =="
rg -n "CNPG|CloudNativePG|cloudnative-pg|1\.2[7-9]|1\.30|Cozystack|v45|compat|supported version|version matrix|versions?|version-bump|HPA|HelmRelease|replicas|status\.selector|labelSelectorPath" . \
  -g '!node_modules' -g '!dist' -g '!build' -g '!vendor' \
  | head -n 300

Repository: cozystack/community

Length of output: 318

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== proposal excerpt =="
sed -n '1,140p' design-proposals/database-horizontal-autoscaling/README.md

echo
echo "== version/matrix mentions =="
grep -RInE "CNPG|CloudNativePG|cloudnative-pg|1\.2[7-9]|1\.30|Cozystack|v45|compat|supported version|version matrix|versions|version-bump|HPA|HelmRelease|replicas|status\.selector|labelSelectorPath" . \
  --exclude-dir=node_modules --exclude-dir=dist --exclude-dir=build --exclude-dir=vendor | head -n 400

echo
echo "== release note/docs snippets in repository files =="
grep -RInE "status\.selector|labelSelectorPath|HPA|recommended against|not planned|7923" . \
  --exclude-dir=node_modules --exclude-dir=dist --exclude-dir=build --exclude-dir=vendor | head -n 200

Repository: cozystack/community

Length of output: 50378

🌐 Web query:

cloudnative-pg 1.29 release notes status.selector HPA Cluster

💡 Result:

The status.selector field for the Cluster scale subresource, which enables integration with the Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA), was introduced in CloudNativePG 1.30.0, not 1.29.0 [1][2][3]. While CloudNativePG 1.29 included significant features such as dynamic network access control via podSelectorRefs and support for PostgreSQL extensions in image catalogs [4][5], the specific capability to expose a label selector via status.selector to allow HPA and VPA to map a Cluster to its instance pods was implemented in the subsequent 1.30.0 release (Pull Request #8996) [1][6][2].

Citations:

🌐 Web query:

cloudnative-pg 1.30 HorizontalPodAutoscaler recommended against Cluster

💡 Result:

In CloudNativePG 1.30, the Cluster resource was updated to include a status.selector on its scale subresource, enabling it to be targeted by a Horizontal Pod Autoscaler (HPA) [1][2][3]. However, the official CloudNativePG documentation explicitly states that using an HPA to drive the number of instances in a Cluster is not recommended for PostgreSQL [4]. The recommendation against using HPA for a CNPG Cluster is based on the following technical reasons [4]: 1. Scaling Limitations: Increasing the number of instances in a Cluster only adds or removes standby replicas; it does not alleviate write load on the primary instance [4]. 2. Meaningless Metrics: The selector exposed by the scale subresource covers both the primary and all replicas [4]. Because these pods have fundamentally different workload profiles (write-heavy primary vs. read-only replicas), the per-pod average of CPU or memory metrics used by HPA is not a meaningful signal for scaling [4]. Adding replicas simply dilutes the average without addressing a potential bottleneck on the primary [4]. 3. Reconciliation Conflicts: HPA is generally unaware of CNPG’s internal constraints, such as synchronous replication requirements [4]. If HPA attempts to scale the cluster below the floor required by synchronous replicas (e.g., maxSyncReplicas + 1), the operator's validating webhook will reject the update, causing the HPA to continuously retry an unfulfillable state [4]. If you choose to use HPA despite these recommendations, the documentation advises that you must base the autoscaling on a custom metric that specifically reflects read-replica load and ensure that your minimum replica count (minReplicas) is set safely above the floor required by your synchronous replication configuration [4].

Citations:

Correct the CNPG version assumptions in this blocker.

Cluster.status.selector is available in CloudNativePG 1.30.0, and CloudNativePG 1.30 documents HPA as not recommended but permitted only for custom/custom-read-replica metrics. Update the blocker to narrow the version-independent text to cozystack v45’s supported CloudNativePG range, or rerun the PoC against 1.30.0+; otherwise the “not fixable by a version bump” and “1.29.2 selector fix” claims are incorrect or unsupported.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@design-proposals/database-horizontal-autoscaling/README.md` around lines 27 -
29, The blocker in the README incorrectly treats the missing selector as
version-independent. Update the discussion around the HPA/CNPG scale-subresource
limitation to reflect that CloudNativePG 1.30.0 provides Cluster.status.selector
and permits HPA for custom or custom-read-replica metrics; either scope the
claims to the CloudNativePG version range supported by cozystack v45 or validate
them with a PoC on 1.30.0+. Remove or qualify unsupported statements about
version bumps and the 1.29.2 selector fix while preserving the documented
limitations of the supported version.


**Two options to resolve (decision needed).**

- **Option A — KEDA + a thin mirror shim.** Keep KEDA's hardened decision loop by pointing its HPA at a proxy object that *does* expose a selector (a small owned CRD, or a placeholder workload), and add a tiny controller that mirrors the proxy's computed count into `Cluster.spec.instances`. Preserves the stock decision loop, but adds a shim, a proxy object, and the platform-wide KEDA dependency for a value KEDA cannot deliver end-to-end on its own.
- **Option B — a lean actuation controller, no KEDA.** A small controller reads the read-load metric from VictoriaMetrics and writes `Cluster.spec.instances` directly, applying the `min`/`max`/quorum-floor bounds and stabilization. This is close to rev1 **minus the ownership machinery** (which §3 already eliminates) and minus the aggregated-API enforcement — a much smaller component than the rejected operator, with no KEDA platform dependency, at the cost of a modest amount of stabilization logic KEDA would otherwise provide.

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Specify fail-safe metric handling for Option B.

If VictoriaMetrics returns no sample, NaN, stale data, or a timeout, the controller must not treat the result as zero. That behavior could scale CNPG to the quorum floor during a monitoring outage. Define freshness, timeout, last-known-good or no-scale behavior, and recovery behavior alongside bounds and stabilization.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@design-proposals/database-horizontal-autoscaling/README.md` at line 34,
Expand the Option B description to define fail-safe handling for missing, NaN,
stale, and timed-out VictoriaMetrics results: never interpret invalid metrics as
zero, specify freshness and query-timeout thresholds, and state whether the
controller retains the last known good value or performs no scaling during
outages. Also document how normal scaling resumes after valid fresh metrics
return, alongside the existing bounds and stabilization behavior.


Both keep §1 (the validated metric encoding) and §3 (chart omits the field). The choice is where the desired-count computation lives — stock KEDA behind a proxy, or a lean purpose-built loop — and whether to take on KEDA as a platform dependency. This reopens the mechanism decision made in the previous revision; the sections below describe the pre-finding mechanism and stand until that decision is made.

## Scope and related proposals

Expand Down