fix(gcs): rescue STALE master-direct ship via shared-storage fallback#55
Merged
Conversation
added 2 commits
July 15, 2026 10:19
S3 step-2 stop-capture forensics closed the branch-1 verdict: a master that retained a stale resident copy (a kept Past Image) serves it as the master-direct grant payload; the SCN self-check correctly detects STALE and fail-closes DENIED_LOST_WRITE — but every hit is a 53R93 client abort, and the stale resident never heals, so the requester wedges in a retry wall (observed 16 consecutive denials of one read in the new t/401 pre-fix run; 60x53R93 in the 4-node S3 sweep). Rescue (master-direct self-check only, found=true path; the !found state=N grant paths are untouched): on verdict STALE, probe the shared-storage page pd_block_scn (new cluster_bufmgr_read_storage_scn_ for_gcs, independent of the buffer table). If storage covers the authoritative pi_watermark_scn — the same gcs_block_lost_write_verdict SCN order the detector itself trusts — convert the reply to GRANTED_STORAGE_FALLBACK: release the ship image, carry the watermark in page_lsn (the state=N grant contract), and let the requester prove/refresh through cluster_gcs_block_fallback_verify_refresh. ANOMALY (unstamped resident) and an unverifiable/stale storage read keep the fail-closed DENIED_LOST_WRITE (Rule 8.A); lost_write_detected_count keeps meaning 'denied'. Observability: new gcs counter lost_write_master_direct_storage_fallback_count (+dump_gcs row, gcs category 111->112) and a producer rescue LOG line carrying the (expected, shipped, storage) SCN triple + requester/request_id/epoch. Injections (registry 181->183): cluster-gcs-block-stale-ship-resident forces the shipped SCN one time-step below the watermark (cluster_scn_time_predecessor — no raw SCN arithmetic) at the master-direct site ONLY; cluster-gcs-block-master-direct-fallback- storage-stale forces the rescue probe unverifiable so the refused leg is deterministically testable. Test: t/401_gcs_master_direct_storage_fallback_2node.pl (18 asserts). The cluster.read_scache=on X->S self-downgrade is the 2-node master-direct GRANTED trigger; the pre-fix RED run reproduced the branch-1 signature ORGANICALLY (verdict STALE on master-direct ship, shipped one below watermark, wm_src=invalidate-ack-slot — same shape as the S3 forensics bundle) with the rescue counter pinned 0 and the read wedged; post-fix the same drive converts (counter 0->1, zero 53R93) and the storage-stale leg still denies (detected 0->1, 53R93 seen, rescue counter unchanged). Gates: cluster_unit 177/177; TAP t/015 t/017 t/030 t/110-116 t/401 (376 asserts, baselines rolled 181->183 / 362->366 / 178->180 names / gcs 111->112); check-format(llvm@18) 0; comment-headers 0; scn-cmp/ges-mode/no-clog-overlay 0. Spec: spec-2.41-cross-node-block-version-scn-lost-write.md
… t/401 assertions Branch-1 gate review follow-ups (three P1s): - P1-1: t/401 gains the required Author header (comment-headers gate was red). - P1-2: the master-direct rescue's storage probe is the first disk I/O on the self-check path; smgrread can ereport(ERROR) on a concurrent truncate/drop or a real I/O failure, which would leak the raw-pinned live_sge ship image (not ResourceOwner-tracked) and drop the reply after the PCM transition was already applied, wedging the requester. Catch locally, LOG, and fall through to the fail-closed DENIED_LOST_WRITE arm, which releases the image and still replies. - P1-3: t/401 L3 now hard-asserts zero 53R93 on the rescued leg (was diag-only), cross-node readback agreement, and the data end-value floor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rescues the master-direct STALE self-check (found=true arm only) by probing the shared-storage page SCN: when storage provably covers the authoritative watermark the reply converts to GRANTED_STORAGE_FALLBACK instead of a 53R93 client abort. ANOMALY / unverifiable storage stays fail-closed DENIED_LOST_WRITE. The !found (state=N grant) paths are untouched.
Includes gate-review follow-ups: storage probe guarded against smgrread ERROR (ship-image raw-pin leak + dropped reply), t/401 hardened (zero-53R93 hard assertion, cross-node readback agreement, end-value floor).
Spec: spec-2.41-cross-node-block-version-scn-lost-write.md
Test plan: