feat(cluster): S3 forensics step 1a -- lost-write / GES-timeout / TT-retention attribution#52
Merged
Conversation
added 3 commits
July 14, 2026 08:59
The ownership-generation wave bumped the t/017 fault_type/hits inject counts by 6 to 179, but the wave landed 6 new injection points on top of a 174-point registry: t/015 and t/030 were correctly rolled to 180 while t/017 was left one short. Pre-existing red on main (fails identically at the branch base); align the count and label with the actual registry.
…ource split, TT retention detail Observability-only instrumentation for the S3 bench triage; no protocol, wire, or on-disk change. 53R93 lost-write (cluster_gcs_block.c): - Producer sites (master-direct ship + holder-forward ship) LOG the verdict (expected pi_watermark_scn, shipped pd_block_scn, holder-local watermark view, requester/master, request_id, epoch) -- the SCN pair is only known on the producer; correlate with the requester by (tag, request_id). - Requester 53R93 ereports (master-direct, local-master X-transfer, storage-fallback verify) gain errdetail with request identity, the local page/watermark view and the per-buffer ownership generation; the fallback detail also separates a stale storage page from a dirty-refused local refresh. GES timeout attribution (cluster_ges.c / cluster_lock_acquire.c / lock.c / cluster_debug.c): - "cluster lock acquire timeout" folds nine failure sites (reply-wait table full, master wait-queue full, outbound ring full, CV deadline, retransmit budget, native probe, convert twins) into one text. A backend-local ClusterGesTimeoutDetail records the fine source at each site, is reset at the seven-step dispatch funnel, and surfaces in the 53R70 errdetail (source/master/elapsed/attempts/conflict_holders/ effective timeout). - Five shmem breakdown counters (true-wait / capacity / send-fail / retransmit-exhausted / native-probe) split how many surfaced timeouts never actually waited; exposed via dump_ges and asserted by the new t/286 L11b/L11c on a real bounded wait. TT retention (cluster_tt_local.c): - The retention-rollover-failed and 48-slots-full ereports gain errdetail with the retention horizon vs current SCN, recycle/skip/wrap counters, segment reuse/switch/hard-cap counters and the pool cap, separating a stalled cleaner from genuine capacity pressure. Unit: stubs for the new symbols in test_cluster_debug.c and test_cluster_lock_acquire.c (standalone link surface). Local gates: cluster_unit 177, cluster_regress 13, PG regress 219/219, TAP t/017 t/108 t/110 t/116 t/280 t/285 t/286 green, check-format 0.
…eject attribution, true elapsed, TT rollover split
Review follow-up on the step-1 instrumentation; observability-only, no
protocol / wire / on-disk change.
Watermark provenance (the branch-3 discriminator):
- Every SCN-watermark feed (redeclare / take-x / grant-x / both
invalidate-ACK feeds) records {source, sender, request_id, epoch,
old->new, advanced} into a shmem ring (seq-guarded against torn
wrap-around reads); the 53R93 emit sites on the master attach the
latest record (wm_src=...), so a shipped<expected verdict can be
qualified as a late / wrong-source watermark advance without a rerun.
The master-direct producer LOG now also carries epoch; the correlation
key is the unambiguous {requester, request_id, epoch, tag} 4-tuple.
Master-reject attribution:
- The reply tails (local drain / remote REQUEST / CONVERT) returned a
master-REPLIED WORK_QUEUE_FULL / TIMEOUT rejection without recording a
timeout source, so the dominant remote-capacity class surfaced as
"unattributed". New sources master-reject-queue-full (capacity
counter) / master-reject-timeout, plus a master-side injection point
(cluster-ges-master-work-queue-full, registry 180 -> 181) driving the
new t/286 L14 semantic leg: source text asserted, elapsed_ms below the
window (the reject never waited), capacity counter delta.
True elapsed:
- All timeout-detail sites report wall-clock from a send-entry stamp
instead of the configured timeout; t/286 L11b2 asserts >= 1s on a
genuine 3s bounded wait, L14d asserts the immediate reject stayed
under the window.
TT rollover split:
- tt_rollover_fail_hard_cap/_extend count ONLY the TT rollover path (the
previously cited segment_hard_cap_fail_count belongs to the
record-extent CLAIM path -- now labeled record_extent_hard_cap_fail);
the errdetail adds cleaner status / loop iters / liveness-tick age to
separate a stalled cleaner from genuine capacity pressure. Two new
dump_undo rows (t/214 68 -> 70).
Tests: new UT test_pcm_wm_prov_ring_records_advance (a stale feed is
recorded with advanced=false and never regresses the max); t/286 L11
asserts are delta-based; t/116 L11b asserts the requester errdetail and
producer-LOG forensic fields. Unit fixtures serving ClusterPcmShared
grew their fake shmem block (4096 -> 32768) for the ring.
Baselines: t/015 t/017 t/030 inject 181; t/214 undo rows 70.
Local gates: cluster_unit 177 (clean), cluster_regress 13, PG regress
219/219, TAP 015/017/030/108/110/116/214/280/285/286, check-format 0.
sqlrush
changed the title
…r_ges link cluster_ges.c now carries the cluster-ges-master-work-queue-full injection point (S3 forensics step 1a), so the standalone test_cluster_ges binary needs cluster_injection_armed_count / _run / _should_skip stubs (cluster_inject.o is not linked). armed_count 0 keeps the macro's fast-path gate closed at runtime. Caught by fast-gate run 29299780419 (Linux enable + macOS, same undefined-reference root cause); the prior local claim was a pipe-masked exit code — re-verified with a direct exit read: cluster_unit 177/177.
sqlrush
force-pushed
the
forensics/s3-owngen
branch
from
2af5b7a to
8e0cf09
Compare
…r identity, GES timeout ledger close, TT activate-fail split - pcm: replace the step-1a 256-slot wm_prov ring (recycled in ms under S3 fan-out, so a 53R93 query could return an unrelated late feed) with a per-tag insert-once open-addressed table (8192x64B + wm_prov_lock); only feeds that ACTUALLY advance the monotone-max watermark are recorded, inside the GRD entry_lock critical section, so the record always explains the CURRENT watermark; a missed query distinguishes definite-none from table-full-inconclusive; new pcm dump key wm_prov_insert_fail_count (fill-8192-tags trigger impractical: surface asserted via t/024+t/108 key counts). - gcs: take_x/grant_x carry the transfer wire identity (holder/requester, request_id, epoch) into the provenance record; the master-direct producer LOG prints the WIRE request epoch (req->epoch) instead of cluster_epoch_get_current(); all five forensics lines gain fork=%d; 53R93 emits print wm_matches_expected=(prov.new_scn==expected_scn) instead of wm_advanced; take-X captures the shipped pd_block_scn before the slot release (the old read was use-after-release). - ges: sixth breakdown counter ges_timeout_master_reject_count closes the timeout-source ledger (every detail_set with a real source bumps exactly one bucket) + dump_ges row. - undo: third TT-rollover failure bucket tt_rollover_fail_activate_count (mark_active refused on a fresh segment) + every rollover-fail branch nudges the undo cleaner after releasing lifecycle_lock. - tests: provenance UT rewritten to the per-tag semantics (a late feed never enters the table); fake pcm shmem fixtures 32KB->1MB; t/024 + t/108 pcm keys 27->28; t/024 L6a inject baseline 173->181 (stale catch-up, t/017 family); t/214 undo rows 70->71; t/286 elapsed_ms now asserted against the client wall clock (kills hardcoded fills both directions) and L14 gains the immediate-reject bound + client-clock ceiling.
Owner
Author
|
Step 1b (user review round 3, five blocking items) pushed as c85b111:
Local gates: format 0 / unit 177 clean-build / regress 13 / TAP 12 files 300 asserts (incl. t/024 L6a stale inject baseline 173->181 catch-up) / PG 219/219. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 1a forensics substrate (scope per review: instrumentation base for the S3 triage, NOT the complete step-1 closure). Observability-only; no protocol, wire, or on-disk change.
Commit 2 (step 1)
Commit 3 (step 1a, review follow-up — closes the 5 gaps)
Step-2 first-53R93 auto-stop collector staged at RACvsRAC scripts/pgrac/mac/mac-s3-first53r93-capture.sh (harness side, not this repo).
Test plan