measure(perf): PERF-062 band ladder at HEAD — ordered streams remove the corruption (#2789) - #2802
measure(perf): PERF-062 band ladder at HEAD — ordered streams remove the corruption (#2789)#2802noahgift wants to merge 3 commits into
Conversation
…the corruption; two of the three remaining failures are the harness (#2789) #2776 has NOT merged, so the task's premise ("ordered streams should be on main") is false. main a866988 carries #2783 but the racy CU_STREAM_NON_BLOCKING default. Ran main + APR_STREAM_LEGACY=1, which is content-verified equivalent to #2776's default: the branch diff is a polarity flip of that one knob, one new default-off knob (APR_ORD9_DRAIN_SKIP), and comment-only changes elsewhere. Binary pinned and proven by content — APR_STREAM_NONBLOCKING and APR_ORD9_DRAIN_SKIP are ABSENT from the binary, positively placing it on main. The branch was not built: / is at 100% disk. Findings: 1. Ordered streams work. At c=16 the racy default returns 5 distinct answers out of 16, mostly garbage; the ordered default returns 1 out of 16, coherent. #2789's GARBAGE cells do not reproduce at HEAD under any ordered arm. 2. No arm passes the ladder as written — but the ladder cannot answer the question. Its c=1 reference is the un-warmed FIRST request, which differed from the following 14 in 5 of 5 server starts while those 14 were unanimous. Spec 4.4.2 already mandates discarding 2xc warmup requests. That alone accounts for every band-1 FAIL. 3. Bands 2 and 4 never form m=2/m=4 — the logs show two m=1 batches and an m=3. A green there reports on a batch that did not happen. 4. The surviving c>=6 disagreement is plausibly a near-tie, not a defect: across six prompts, 4 of 6 agree exactly between c=1 and c=8 at 8/8 self-consistency, and the two that differ are open-ended prompts whose alternatives are both coherent. Evidence, not proof — a logprob-margin probe would settle it. No W1 receipt recorded (I-9): the ladder is not yet clean. Refs #2789, #2706, #2767, #2790
…*.log ate them The three run logs backing every hash and every band verdict in ladder.md were silently dropped from the previous commit: .gitignore:38 is a bare *.log, so git add succeeded and committed only the summary. A summary-only record cannot be re-read against its own claims, which is the whole point of retaining it. Renamed to .txt. Same CB-510 class as the models/ vs /models/ scar: an unanchored ignore pattern quietly removing files a commit was supposed to carry.
pr-review v2.0.0 — automated adversarial review (PR-REVIEW-SKILL-002-v2)6 findings raised, 4 survived adversarial refutation (three independent verifiers per finding — correctness / scope / evidence-quality lenses, refute-by-default). Refuted findings are dropped, not shown.
1. §3.2 disqualifies the STRICT run's A2 PASS cells using the OTHER run's batch counts — the strict log shows A2 did form m=2 and m=4
Evidence Why it matters — This is one of the three headline conclusions (§7 item 2: "bands 2 and 4 never form m=2/m=4 (§3.2), so those cells report on batches that did not happen") and one half of the PR body's claim that "Two of the three remaining failures are the harness, not the runtime." On its own evidence it is false for the run it is being applied to: the strict A2 PASS at c=2 and c=4 DID exercise m=2 and m=4 batched paths. Either the disqualification is wrong, or the two runs formed different batches from the same arm — which would itself be a finding about batch-formation nondeterminism that the document does not report. Also note the strict A2 block reports m=15 where §3.2's table attributes m=15 to A3, further confirming the tables have been crossed. Why this survived refutationCould not refute — the finding is correct on the evidence. (1) ladder.md:140-144's batch table is a character-exact copy of run-selfconsistency-ladder.txt lines 4/23/42/61/80 (A0 "15 1 1 2 1 4 1 6 1 8 1 16"; A1/A2/A4 "18 1 1 3 1 6 1 8 1 16"), so §3.2 rests on the self-consistency run. (2) The cell it voids is from the other run: ladder.md:147 names A2's "PASS at c=2", and the only PASS at c=2 for A2 is ladder.md:98 in the §2 STRICT table, under a section headed "Why the strict verdict is not sound" (line 109) that precedes §4's self-consistency ladder (line 158, which uses OK/SAME, not PASS). (3) run-strict-ladder.txt ARM A2 (97-142) shows route APR_STREAM_LEGACY=1 CUBLAS_GEMM_THRESHOLD=32 (103), PASS band=2 (118), PASS band=4 (119), and "1 [PMAT-044] Batch m=2" (138) + "1 ... Batch m=4" (139). I did not take the adjacency on trust: the strict run issues 38 requests (1 ref + bands 1,2,4,6,8,16), and A2's inventory 2xm=1 + m=15 + m=2 + m=4 + m=6 + m=8 forces ref and band1 onto the two m=1 batches (A1 and A4 confirm ref/band1 stay separate m=1s), leaving m=2 for band 2 and m=4 for band 4. Across the strict run band 4 formed a full m=4 in A1, A2 and A4 - three of four ordered arms - so §7 item 2's "bands 2 and 4 never form m=2/m=4" is false for the run whose cells it disqualifies. §3.2's general point (band label != batch size) does survive via strict A1/A3/A4 band 2 and A3 band 4, but the specific claim that A2's PASS "never exercised the batched path" does not. Refutation paths all checked and negative: evidence/perf-062/ladder.md does not exist on origin/main (the 2. An ORDERED arm (A3, APR_STREAM_LEGACY=1) produced degenerate garbage at c=1/2/4 in the strict log, contradicting "GARBAGE cells do not reproduce under any ordered arm"
Evidence Why it matters — ladder.md never mentions that an ordered arm produced degenerate repeating output. §5's "Incidental" paragraph frames A3's c=1 divergence purely as a different mode hash ( Why this survived refutationConfirmed, and stronger than the finding states. SCOPE: all four evidence files are new in this PR ( 3. §5's six-prompt near-tie experiment — the load-bearing evidence for "not a defect" — has no committed log; 6 of its 8 hashes appear in zero committed file
Evidence Why it matters — §5 is the section that reclassifies the surviving failures from "GPU correctness defect" to "plausibly a near-tie" — the single biggest severity move in the document, and the one the PR title leans on. It is a separate 6-prompt x (5 + 8) replicate experiment whose entire receipt is the summary table itself. The commit that was specifically added to fix a summary-only record ( Why this survived refutationCould not refute — the finding reproduces exactly and survives every refutation avenue. (1) HASH GREP REPRODUCED DIGIT-FOR-DIGIT: extracting the four diff files via 4. Nothing in this evidence set is re-derivable from origin/main: the strict ladder script is absent from main and the self-consistency scorer is absent from the repo entirely
Evidence Why it matters — Both instruments that produced every number in this document are un-obtainable from main, so no reader can re-run either one or re-derive any cell. ladder.md:299 self-flags the strict script's absence ("A ladder that gates a release should not live on one unmerged branch") but does not flag that the self-consistency scorer — the instrument that produced the headline "1 distinct out of 16" — is not in the repo at all and carries no committed falsifier. The committed self-test proves only the strict string-equality comparator can fail. Mitigating: the self-consistency scorer does exhibit both polarities within the run itself (A0 bands report self-consistent=False, A1 bands report True), so it is not a verdict that can only pass — but that is a weaker discrimination than the committed positive control it is presented alongside. Why this survived refutationEvery sub-claim reproduced against origin, and none of the five refutation conditions hold.
NEGATIVE-CONTROL CHECK (done explicitly): not a misread. The finding's own evidence block quotes the Reviewer actor is distinct from the author actor (spec 5). This review is advisory: no finding here landed in a spec-7 blocking class. |
1 participant
PERF-062 band ladder measured at HEAD.
The task's premise was false and is corrected here. #2776 has not merged, so ordered streams are not on main. main a866988 carries #2783 but the racy
CU_STREAM_NON_BLOCKINGdefault. This ran main +APR_STREAM_LEGACY=1, which is content-verified equivalent to #2776's default: the branch diff is a polarity flip of that one knob, one new default-off knob (APR_ORD9_DRAIN_SKIP), and comment-only changes elsewhere.Binary pinned and proven by content, not by intent:
APR_STREAM_NONBLOCKINGandAPR_ORD9_DRAIN_SKIPare absent from the binary, positively placing it on main. The branch was not built —/was at 100% disk.Findings
Second commit
The three run logs backing every hash and band verdict were silently dropped from the first commit:
.gitignore:38is a bare*.log, sogit addsucceeded and committed only the summary. Renamed to.txt.Same CB-510 class as the
models/vs/models/scar — an unanchored ignore pattern quietly removing files a commit was supposed to carry. A summary-only record cannot be re-read against its own claims, which is the entire point of retaining it.Refs #2789, APR-PERF-GATE-001 (#2706)