Skip to content

fix(chart): un-vacuum the order-of-ops assertion, and make the mysqlClient pin watchable - #714

Merged
LukasWodka merged 3 commits into
developfrom
fix/1941-client-two-findings
Aug 13, 2026
Merged

fix(chart): un-vacuum the order-of-ops assertion, and make the mysqlClient pin watchable#714
LukasWodka merged 3 commits into
developfrom
fix/1941-client-two-findings

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Two Bugbot Mediums on the staging promotion PR #713.

1. The order-of-operations assertion was vacuous

It required kubectl rollout restart before kubectl annotate — but #569 removed that command, and a sibling notMatchRegex forbids it on a command line. The only text it could still match was the header comments explaining the old mechanism, so it passed regardless of what order the real commands ran in.

Measured on the rendered script rather than argued. Hoisting the post-re-image annotates above the first kubectl set image — the actual regress, header comments left in place:

old assertion new assertion
real script passes passes
regressed script passes ← ships the bug green fails

A regress that recorded the digest before re-imaging would have frozen every workload on its old image with a green suite.

Now anchored to command lines and to the mechanism that exists: set imagerollout statusannotate.

Two earlier attempts at that mutation were themselves broken — one hoisted the annotates above the comments too, one popped by stale indices after inserting. Both "proved" the wrong thing. The mutation quoted above is verified to have actually moved the lines before its result is used.

2. The mysqlClient pin was never watched

images.mysqlClient carried a real digest but no repository and an empty tag — both live as template defaults. check-digest-drift.sh pairs a pin with the repo/float in its own block, found neither, and classified it UNWATCHABLE, which exits non-zero.

So the daily drift watch stays red forever while the pin it exists to compare is never checked. A guard that cannot pass teaches everyone to ignore it.

Now stated in values and consumed by the template, so the value is real config rather than decoration.

BEFORE  UNWATCHABLE: images.mysqlClient is pinned to sha256:f546e47…
AFTER   ok    tracebloc/mysql-client:prod    sha256:f546e47fb339…

Verified

  • render byte-identical across aks/bm/eks values (only POD_TOKEN_SIGNING_SECRET differs — helm regenerates it every run); mysql image line unchanged
  • helm lint clean; helm unittest 36/36, including the edited suite

Note

Low Risk
Changes are chart version bump, stricter unit tests, and values/template wiring that mirrors existing defaults; no intentional runtime behavior change to workloads or image-refresh script logic in this diff.

Overview
Bumps the client chart to 1.9.40 and fixes two CI/guard gaps that let real regressions ship with a green suite.

The image-refresh helm unittest no longer checks ordering via the removed kubectl rollout restart path (which could still match header comments). It now requires command-line order: kubectl set imagekubectl rollout status → digest recording via kubectl annotate with $annotate_args, so flap-counter annotates cannot satisfy the test if the digest is written before re-imaging.

images.mysqlClient now sets explicit repository and tag: prod in values.yaml, and the mysql deployment template reads those fields so check-digest-drift.sh can pair the pin with repo/tag instead of marking it UNWATCHABLE. Rendered mysql image stays byte-identical to the previous template defaults.

Reviewed by Cursor Bugbot for commit 18956ad. Bugbot is set up for automated code reviews on this repo. Configure here.

…lient pin watchable

Two Bugbot Mediums on the staging promotion PR #713.

1. ORDER-OF-OPS ASSERTION WAS VACUOUS. It required `kubectl rollout restart`
   before `kubectl annotate` -- but #569 REMOVED that command, and a sibling
   notMatchRegex forbids it on a command line. So the only text it could still
   match was the header comments explaining the old mechanism, and it passed
   regardless of what order the real commands ran in.

   Measured on the rendered script rather than argued: with the post-re-image
   annotates hoisted above the first `kubectl set image` -- the actual regress,
   header comments left in place -- the OLD assertion still passes (True) and
   the NEW one fails (False). On the real script the new one passes. So a
   regress that recorded the digest before re-imaging would have frozen every
   workload on its old image with a green suite.

   Now anchored to COMMAND lines and to the mechanism that exists:
   `set image` -> `rollout status` -> `annotate`.

   (Two earlier attempts at that mutation were themselves broken -- one hoisted
   the annotates above the comments too, one popped by stale indices after
   inserting. Both "proved" the wrong thing. The mutation above is verified to
   have actually moved the lines before its result is quoted.)

2. mysqlClient PIN WAS NEVER WATCHED. `images.mysqlClient` carried a real digest
   but no `repository` and an empty `tag`, because both live as template
   defaults. check-digest-drift.sh pairs a pin with the repo/float in its own
   block, found neither, and classified it UNWATCHABLE -- which exits non-zero,
   so the DAILY DRIFT WATCH STAYS RED FOREVER while the pin it exists to compare
   is never checked. A guard that cannot pass teaches everyone to ignore it.

   Stated in values and consumed by the template, so the value is real config
   rather than decoration. Both are what the templates already defaulted to.

Verified:
  * render byte-identical across aks/bm/eks values (only POD_TOKEN_SIGNING_SECRET
    differs, which helm regenerates every run); mysql image line unchanged
  * helm lint clean; helm unittest 36/36 including the edited suite
  * drift watch: BEFORE "UNWATCHABLE: images.mysqlClient is pinned to sha256:f546…"
    AFTER  "ok  tracebloc/mysql-client:prod  sha256:f546e47fb339…"
@LukasWodka
LukasWodka requested a review from saadqbal as a code owner August 13, 2026 15:17
@LukasWodka LukasWodka self-assigned this Aug 13, 2026
Comment thread client/tests/image_refresh_test.yaml Outdated
…any annotate

The order-of-ops assertion only required SOME `kubectl annotate deployment`
after set-image/rollout-status. The flap-counter-reset annotate
(`${ATTEMPT_KEY}- ${FLAP_KEY}-`) already sits there by design, so moving ONLY
the digest-recording annotate (`$annotate_args`) above the re-image kept the
suite green while the digest was recorded BEFORE the image was applied — the
freeze path this guard exists to catch. Require the `$annotate_args`
continuation line so the ordering is tied to the digest write itself, distinct
from the flap reset. Verified by mutation: the loose pattern stayed green with
the digest annotate moved above set-image; the tightened one fails. Bugbot, client#714.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 27371d4. Configure here.

The version-bump-gate (scripts/chart-version-guard.sh) requires a Chart.yaml
version bump when client/templates|values change, since the Helm repo only
publishes a NEW version — an unbumped edit ships dark or overwrites a
published one. develop is at 1.9.39; bump to 1.9.40.

backend#1468 unrelated — this is client#714.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — verified for correctness.

This PR closes two green-but-vacuous guard gaps from #713, plus the required chart bump. I re-derived each independently against the rendered script and the drift watcher rather than the description:

  • Order-of-ops assertion (image_refresh_test.yaml) — the new (?ms)^\s+kubectl set image.*^\s+kubectl rollout status.*^\s+kubectl annotate deployment[^\n]*\n\s*\$annotate_args is valid RE2 and correct on two counts. Anchoring to ^\s+kubectl means the header comments and log "...set image..." lines can no longer satisfy it — only real command lines. Requiring the $annotate_args continuation line ties the ordering to the digest annotate (script lines 507→508); the flap-reset annotate (${ATTEMPT_KEY}- ${FLAP_KEY}-, lines 497–498) carries different args and can no longer stand in. Since kubectl set image exists only in the re-image block, hoisting the digest annotate above it leaves no set image before the sole $annotate_args occurrence → no match → the freeze regress fails the suite, as intended.

  • mysqlClient watchability (values.yaml + mysql-deployment.yaml)check-digest-drift.sh's awk pairs the 4-space repository:/tag: leaves with the 4-space digest: pin within the block; both were previously empty (UNWATCHABLE, non-zero exit → daily watch red forever). Now repository: tracebloc/mysql-client + tag: prod make the pin watchable, and the :prod tag being a single-arch manifest matches the single-arch pin, so it resolves ok rather than a spurious DRIFT. The template reads .Values.images.mysqlClient.repository | default "tracebloc/mysql-client"; combined with the explicit values it renders byte-identical to the prior hardcode. Only one mysql-client image ref exists on the branch, so no straggler.

  • Chart bump 1.9.39 → 1.9.40 — required by the version-bump-gate for chart-content changes; passes.

All 21 checks green (helm lint, helm unittest, 4× template render, drift/source-of-truth, version gates), Bugbot reports no new issues, and the earlier Bugbot order-of-ops thread was already addressed in 27371d4 and resolved.

@LukasWodka
LukasWodka merged commit 5345504 into develop Aug 13, 2026
22 checks passed
@LukasWodka
LukasWodka deleted the fix/1941-client-two-findings branch August 13, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants