Skip to content

fix: update Core and updater as one release pair#628

Draft
frahlg wants to merge 4 commits into
masterfrom
codex/atomic-control-plane-pair-v2
Draft

fix: update Core and updater as one release pair#628
frahlg wants to merge 4 commits into
masterfrom
codex/atomic-control-plane-pair-v2

Conversation

@frahlg

@frahlg frahlg commented Jul 22, 2026

Copy link
Copy Markdown
Member

Why

#593 used an old release layout and conflicted with master. Master already keeps per-component rollback image IDs, checks Core /api/status before it accepts an update, and blocks Core updates when Optimizer is not healthy. This PR keeps those rules and adds the missing Core/updater pair contract.

What changes

  • Require updater /status protocol 2 or later, capability control-plane-pair-v1, and updater_version equal to the running Core version before Core opens state.
  • Require ftw-control-plane.json to name the same Core/updater release and pin both images by digest.
  • Carry the verified source revision and both digests through the Core request, updater state, and detached helper. Pull and run repo@sha256:... refs and inspect both running refs before done.
  • Give each paired update a durable transaction ID, helper owner, and heartbeat lease. A new updater waits for a fresh helper. When a helper goes stale, recovery stops the owner named in durable state and starts a recovery helper from the saved old updater image.
  • Serialize heartbeat read/update with every state write. Stop the updater heartbeat before helper handoff and stop the helper heartbeat before terminal done or failed state.
  • Fence the durable helper owner before writing a recovery claim. Re-read state after stop and preserve any terminal done or failed result, including when stop returns an error.
  • Save both old image IDs. Normal failure and stale-helper recovery restore updater first and Core second. Recovery ends failed, never done.
  • Keep Core readiness on /api/status; /api/health remains boot liveness only.
  • Add the one-time v1.10 adoption bridge. The migration script pulls exact digests, starts the new updater while old Core stays healthy, then starts Core. The old Update Center does not claim to update both.
  • Keep Optimizer outside the pair manifest and pair rollback. The existing healthy-Optimizer preflight remains.
  • Add a patch changeset and operator steps.

Compatibility

Running or requested pair Result
New Core + same-release updater, protocol 2+, pair capability, exact approved digests Ready; update may commit
New Core + v1.3.1 updater Core stays unready before state.Open; no migration
New Core + different updater version Core stays unready before state.Open; no migration
Tag moves after Check Saved manifest digests still select the bytes; a wrong running ref rolls the pair back
Detached helper heartbeat expires The named helper is stopped; recovery restores old updater then old Core; final state is failed
v1.10.0-beta.1 Core/updater + first pair-aware release Old UI is not a bridge; use the tested exact-digest updater-first migration path
Core + Optimizer 1.3.1-beta.1, 1.3.1, or 1.3.2+ on protocol 1 Compatible; Optimizer version does not need to match Core

Optimizer stays on name=ftw-optimizer, exact protocol 1, plan schema 1, and feature checks. It is not part of ftw-control-plane.json.

Checks

  • go test ./cmd/ftw-updater ./internal/selfupdate ./internal/api ./cmd/ftw -count=1 — pass
  • bash scripts/test-modular-compose.sh — pass
  • go test -race ./cmd/ftw-updater ./internal/selfupdate -count=1 — pass
  • focused stale-owner and terminal-state race tests with -race -count=20 — pass
  • make verify — pass on 3e765dd2
  • pre-commit make verify — pass
  • make verify-all — pass
  • pre-push make verify-all — pass, including Linux arm64/amd64 and Windows amd64 builds
  • GitHub CI on 3e765dd2 — pending

The focused tests cover:

  • old v1.3.1 updater rejection before POST or state open;
  • verified digest handoff from Check to Trigger and updater state/helper;
  • digest-pinned Compose refs and a moved-tag/running-digest mismatch;
  • fresh helper heartbeat during updater replacement;
  • stale restoring owned by a normal helper and by a recovery helper;
  • terminal done and failed writes racing a heartbeat read/update;
  • terminal done during successful helper stop and terminal failed during a stop error;
  • host restart after updater replacement;
  • updater-first rollback, final failed, and both old image IDs retained;
  • v1.10 updater-first adoption while old Core remains healthy;
  • readiness failure restoring both prior image IDs.

Rollout gate

Keep this PR draft. Do not merge or publish a Core beta from this head yet.

Completed prerequisites:

  • Optimizer 1.3.2 beta and stable artifacts passed; the .40 update/rollback/re-update pilot passed with safe Go-DP fallback.
  • The separate v1.10.0-beta.1 pilot passed on e9adb3113513dcd449268096b0dd637f4753ed24. It did not test this PR.

Still required:

  1. Independent review of exact head 3e765dd24ddafe512372389b1803f464f876ee65.
  2. A separate decision to build a paired beta from that approved head. Record the Core and updater index/platform digests and confirm ftw-control-plane.json uses the same source revision.
  3. Pair pilot from the current v1.10 install:
    • capture both old image IDs and the rollback backup;
    • run the one-time exact-digest updater-first bridge and prove old Core stays healthy before Core changes;
    • confirm updater protocol, capability, exact version, and digest before Core replacement;
    • confirm Core exact digest and that /api/status stays false until state and migrations are ready;
    • keep the existing healthy Optimizer unchanged;
    • inject failure after updater replacement and prove updater-first then Core rollback, both old IDs restored, and final state failed;
    • stop the detached helper or restart the host, let the lease expire, and prove the same failed recovery result with no mixed pair;
    • retry the exact candidate and confirm both digests, readiness, and no mixed version.
  4. Independent review of the pilot evidence and an explicit merge decision.

Supersedes #593. This PR does not change driver source, auth, persistent data ownership, Optimizer release logic, or the control loop. It does not use #623 as its integration path.

@frahlg
frahlg force-pushed the codex/atomic-control-plane-pair-v2 branch from 4db8548 to a9b37be Compare July 22, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant