fix(undo): peer-mode TT recycle via cluster floor + stalled-cleaner retry at report cadence#56
Merged
Merged
Conversation
added 2 commits
July 15, 2026 13:02
Completion: adds the allocator-side peer-mode guard that prevents direct COMMITTED-slot recycling, keeps ABORTED directly reusable, and adds T48-T50 unit scenarios for cleaner-owned cluster-floor recycle, ABORTED reuse, and GUC-off fail-closed behavior. Design intent: in peer mode the epoch-fenced cluster undo cleaner is the sole COMMITTED-to-FREE authority after a proven cluster floor; the allocator consumes the resulting FREE slot through Pass 1 instead of trusting a local ProcArray horizon. Handoff status: this is an unshipped WIP snapshot. No build, unit, integration, or S3 validation was run during this handoff. The peer-report/epoch proof integration, retained-pressure and rollover behavior, and production cleaner call path still require independent evidence; the receiving CC session should evaluate whether to inherit or rework it.
S3 verdict class #2 (25,715 x 'TT retention rollover failed', all on node0): when the undo cleaner's recycle stage stalls (cluster horizon unproven: missing/stale/epoch/malformed, member churn, or an F-D2 fence abort), the pass exits and the cleaner sleeps its FULL cluster.undo_cleaner_interval_ms (30s default) before re-evaluating. Stall causes normally clear at the horizon-report cadence -- one LMON tick (1s): epoch views converge and every peer re-publishes -- so a ~1s transient froze cluster-wide COMMITTED recycling for 30-60s. Field evidence (step-2 rig logs): the boot 'malformed' stall healed after exactly 60.0s = two passes on all four nodes; the 12:24 'epoch' stall (blaming a live peer during a dead->alive churn) after exactly 30.0s = one pass. Under the S3 write storm every reconfig wobble cost one such freeze; node0 burned the 256-segment pool to the hard cap. Fix (mechanism, no parameter change): undo_cleaner_run_pass() now reports whether the recycle stage failed to run against a proven floor; the main loop then re-arms the next pass at Max(cluster_lmon_main_loop_interval, 200)ms -- the cadence at which new reports and epoch convergence can actually arrive -- instead of the recycle interval. This also bounds the interval=0 (pressure- wakeup-only) mode, whose stalled pass previously blocked on a latch nobody was obliged to set. Rule 8.A untouched: every retry still recycles ONLY on a proven cluster floor; a persistent cause keeps every retry stalled (fold semantics unchanged). Test: t/402_undo_cleaner_stall_retry_2node.pl. RED on the pre-fix binary: heal latency after the stall cause cleared = 15.1s (= the full 15s test interval); GREEN: 1.1s (report cadence), 14/14. Protection negative leg: with report-drop KEPT armed the floor never advances and stall accounting keeps running -- the retry does not weaken fail-closed. Harness note: pg_reload_conf's SIGHUP latch- wakes the cleaner into an immediate pass, so the disarm used for the latency measurement is single-reload (a two-reload disarm heals the cleaner by test side effect and masks the cadence under test). Gates: cluster_unit 177/177; TAP t/214 t/222 t/270 t/370 t/402 (129 asserts); cluster_regress 13/13; PG 219/219; check-format(llvm@18) / comment-headers / scn-cmp / ges-mode / no-clog-overlay all clean. Spec: spec-5.22e stall semantics (unchanged); TT lane P1#4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two commits closing the S3 verdict class #2 ("TT retention rollover failed" undo segment-pool exhaustion, node0-concentrated):
cluster.undo_cleaner_interval_ms(30s default) before re-evaluating. The stall causes normally clear within ~one LMON tick (peers re-publish, epoch views converge), so a ~1s transient froze cluster-wide COMMITTED recycling for 30-60s per event — the pool-exhaustion amplifier. A stalled pass now re-arms atMax(lmon interval, 200ms); the interval=0 (pressure-wakeup-only) mode is bounded the same way. Fail-closed unchanged: every retry still recycles ONLY on a proven floor; a persistent cause keeps every retry stalled.Tests
t/402_undo_cleaner_stall_retry_2node.pl(new): pre-fix the heal latency after a cleared stall equals the full cleaner interval (15.1s at a 15s test interval, hard-assert red); post-fix it is the report cadence (1.1s), 14/14. Protection negative leg: with the report-drop kept armed the floor never advances and stall accounting keeps running. Harness note pinned in-test:pg_reload_conf's SIGHUP latch-wakes the cleaner, so the latency measurement uses a single-reload disarm.52364a4d42): cluster_unit 178/178; TAP t/015 t/017 t/214 t/222 t/270 t/370 t/401 t/402 = 180 asserts; cluster_regress 13/13; PG 219/219; check-format(llvm@18) / comment-headers / scn-cmp / ges-mode / no-clog-overlay all clean.Spec: spec-5.22e stall semantics (unchanged), spec-3.13 cleaner pass contract.