diff --git a/evidence/perf-062/harness-selftest.txt b/evidence/perf-062/harness-selftest.txt new file mode 100644 index 0000000000..dbfee6b935 --- /dev/null +++ b/evidence/perf-062/harness-selftest.txt @@ -0,0 +1,17 @@ +PERF-059 harness self-test (no GPU, no model) +case 1: all slots identical to the reference -- must PASS + PASS band=self-1: 2/2 match the c=1 reference + ok +case 2: one slot diverged -- must FAIL + FAIL band=self-2: 1/2 diverged, 0/2 empty + ok +case 3: one slot empty -- must FAIL + FAIL band=self-3: 0/2 diverged, 1/2 empty + ok +case 4: no completions at all -- must FAIL + FAIL band=self-4: no completions returned + ok +case 5: a prefix of the reference -- must FAIL + FAIL band=self-5: 1/1 diverged, 0/1 empty + ok +SELF-TEST PASS: the verdict distinguishes all five cases diff --git a/evidence/perf-062/ladder.md b/evidence/perf-062/ladder.md new file mode 100644 index 0000000000..f4a7251107 --- /dev/null +++ b/evidence/perf-062/ladder.md @@ -0,0 +1,301 @@ +# PERF-062 — the band ladder re-run, and what it actually says + +Re-run of the PERF-059 band ladder (#2789) on an RTX 4090 (sm_89), 1.5B Q4_K_M, +asking the question APR-PERF-GATE-001 needs answered before any cell is measured: +**is there now ONE route configuration that is correct across bands {1,4,8,16}?** + +**Short answer: no — but the reason has changed completely, and two of the three +remaining failures are defects in the ladder, not in the CUDA path.** + +--- + +## 0. The premise of the task is false: #2776 has NOT merged + +The work item said "ordered streams should be on main." They are not. + +| Ticket | PR | State at time of run | +|---|---|---| +| #2770 cuBLAS Q6_K dequant | #2783 | **MERGED** `c6a6dfce3`, 2026-08-30T08:28Z | +| #2767 ordered streams | #2776 | **OPEN**, `mergeStateStatus: BLOCKED`, branch `feat/v14-streamdefault` | + +`origin/main` at run time was `a866988e4`. So a ladder run against plain main would +have measured the *racy* stream default and told us nothing new. + +### What I ran instead, and why it is equivalent + +I ran **main `a866988e4` with `APR_STREAM_LEGACY=1`**, which is content-verified +equivalent to #2776's *default*. This is not an assumption; it is the whole diff: + +``` +$ git diff --stat origin/main...origin/feat/v14-streamdefault + .github/workflows/cuda-nightly.yml | 35 +++ CI only + contracts/continuous-batching-v1.yaml | 22 +- contract only + crates/aprender-gpu/src/driver/memory/tests.rs | 11 +- tests only + crates/aprender-gpu/src/driver/memory/transfer.rs | 69 ++- see below + crates/aprender-gpu/src/driver/stream.rs | 89 ++- the polarity flip + crates/aprender-serve/src/cuda/executor/mod.rs | 14 +- COMMENT-ONLY (verified) + crates/aprender-serve/tests/falsify_..._2767.rs | 281 +++ new test +``` + +- **`stream.rs`** is a *polarity flip of one knob*. main: `APR_STREAM_LEGACY=1` + selects `CU_STREAM_DEFAULT`, default is `CU_STREAM_NON_BLOCKING`. #2776: default + is `CU_STREAM_DEFAULT`, `APR_STREAM_NONBLOCKING=1` selects the racy one. + Setting `APR_STREAM_LEGACY=1` on main reaches the identical `cuStreamCreate` flag. +- **`transfer.rs`** adds exactly one new runtime path, `APR_ORD9_DRAIN_SKIP`, which + **defaults off**. Non-comment changed lines, in full: + `if ord9_drain_disabled() { return Ok(()); }` + the `OnceLock` accessor. +- **`executor/mod.rs`** has **zero** non-comment changed lines (mechanically checked). + +So the only runtime difference between "#2776 default" and what I ran is a knob that +is off in both. **I did not build the branch** — the box is at 100% disk (see §6) — +and I state that plainly rather than implying I ran the PR. + +--- + +## 1. Binary provenance, proven by content not by `--version` + +``` +path /tmp/.../scratchpad/y5-pin/apr.a866988e4 (copied and pinned; the source + target dir was being concurrently rewritten by another agent's cargo test) +sha256 40eb0a8c8df48b35... +--version apr 0.64.0 (a866988e4) == origin/main +strings -a | grep -c cublasGemmEx 3 <- real CUDA build +strings -a | grep -c APR_STREAM_LEGACY 1 <- main's knob PRESENT +strings -a | grep -c APR_STREAM_NONBLOCKING 0 <- #2776's knob ABSENT +strings -a | grep -c APR_ORD9_DRAIN_SKIP 0 <- #2776's knob ABSENT +strings -a | grep -c FP8_DECODE 1 +strings -a | grep -c CUBLAS_GEMM_THRESHOLD 1 +``` + +The two absent strings are positive content-proof that this binary is main +*without* #2776, which is what makes the equivalence argument above checkable +rather than rhetorical. + +**Every run below additionally confirmed, per run, from the server log:** +- `gpu-layers: requested=all resolved=28 total=28 (backend=cuda)` — resolved, not requested (I-2) +- `CB-006-OUT` line count = **0** — `APR_LAYER_TRACE` genuinely off (#2764) +- harness self-test (`--prove-can-fail`) passed **both polarities**: the verdict + correctly PASSes identical input and FAILs on diverged / empty / zero / truncated. + +**Protocol statement.** This is **not** the spec's §4.4 measurement protocol. §4.4 +is the throughput protocol (closed-loop client, `2×c` warmup, `max(30, 8×c)` samples, +60 s floor, bootstrap CIs). This is the PERF-059 *correctness* ladder plus a +self-consistency extension — a precondition check that must pass **before** §4.4 +runs at all. No tok/s is reported here and no W1 receipt is recorded (I-9). + +--- + +## 2. The ladder at HEAD — strict PERF-059 verdict + +`scripts/perf059_band_ladder.sh` verdict: every slot in band *c* must be +**string-identical** to a single c=1 reference. Model 1.5B Q4_K_M, 40 max tokens, +one prompt, bands 1/2/4/6/8/16 (m≥6 included because FP8 fires at m≥5). + +| Arm | `STREAM_LEGACY` | `CUBLAS_THR` | `FP8_DECODE` | c=1 | c=2 | c=4 | c=6 | c=8 | c=16 | +|---|---|---|---|---|---|---|---|---|---| +| A0 (main default, racy) | 0 | def | def | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | +| A1 (**= #2776 default**) | 1 | def | def | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | +| A2 (#2789 row 4) | 1 | 32 | def | FAIL | PASS | PASS | FAIL | FAIL | FAIL | +| A3 (#2789 row 5) | 1 | 32 | 0 | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | +| A4 (#2789 row 6) | 1 | def | 0 | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | + +**Every arm fails band 1.** A band-1 failure means "one c=1 request disagreed with +another c=1 request." That is internally contradictory with A2 passing bands 2 and 4, +and it is the thread that unravels the instrument. + +--- + +## 3. Why the strict verdict is not sound — three defects in the ladder + +### 3.1 The c=1 reference is the *un-warmed first request*, and it is the outlier + +The ladder takes its reference from the very first request after server start. +Sequential c=1 replicates, one server, greedy, 15 per arm (10 before the bands, +5 after, to separate warm-up from drift). md5 prefixes, **every replicate**: + +| Arm | pre (10, in order) | post (5) | distinct | mode | +|---|---|---|---|---| +| A0 | `94b918ea` `fc42d43e`×9 | `fc42d43e`×5 | 2 | 14/15 | +| A1 | `94b918ea` `fc42d43e`×9 | `fc42d43e`×5 | 2 | 14/15 | +| A2 | `94b918ea` `fc42d43e`×9 | `fc42d43e`×5 | 2 | 14/15 | +| A3 | `a7ad49d4` `08912486`×9 | `08912486`×5 | 2 | 14/15 | +| A4 | `94b918ea` `fc42d43e`×9 | `fc42d43e`×5 | 2 | 14/15 | + +**In 5 of 5 server starts the first request differs from every one of the following +14, and the following 14 are unanimous.** This is a warm-up transient, not +nondeterminism: the post-band replicates are identical to the pre-band steady state, +so there is no drift. + +Spec **§4.4.2 already legislates this away** — "Warmup requests `2 × c`, discarded, +not written to the receipt" plus a 5 s quiesce. The ladder script does no warmup, so +it systematically anchors its reference on the one sample §4.4 would have thrown out. +**This alone accounts for every band-1 FAIL in the table above.** + +### 3.2 The band label is not the batch size — bands 2 and 4 never formed + +From `[PMAT-044] Batch m=` in the server logs, batches actually formed: + +| Arm | batches formed (count × m) | +|---|---| +| A0 | 15×m=1, 1×m=2, 1×m=4, 1×m=6, 1×m=8, 1×m=16 | +| A1 | 18×m=1, 1×m=3, 1×m=6, 1×m=8, 1×m=16 | +| A2 | 18×m=1, 1×m=3, 1×m=6, 1×m=8, 1×m=16 | +| A3 | 18×m=1, 1×m=3, 1×m=6, 1×m=8, 1×m=15 | +| A4 | 18×m=1, 1×m=3, 1×m=6, 1×m=8, 1×m=16 | + +Under every ordered arm, the two c=2 requests **ran as two separate m=1 batches** and +c=4 formed **m=3**. So **A2's "PASS at c=2" is vacuous — it never exercised the +batched path**, and c=4 is a partial batch whose composition varies run to run. A +ladder row that reports a band it did not form is a green that proves nothing. + +### 3.3 The verdict cannot distinguish "corrupted" from "a different valid answer" + +Exact string equality collapses two very different outcomes into one FAIL. They are +not the same failure and only one of them is a CUDA defect — see §4. + +--- + +## 4. The self-consistency ladder — what ordered streams actually fixed + +Re-measured with the reference taken as the **mode of 15 warmed c=1 replicates**, and +each band scored on (a) distinct completions within the band and (b) whether the +band's mode equals the c=1 mode. Every replicate hash retained. + +### A0 — main's current default (racy, `CU_STREAM_NON_BLOCKING`) + +| band | distinct/total | self-consistent | vs c=1 mode | +|---|---|---|---| +| 2 | 1/2 | yes | DIFFERENT | +| 4 | 2/4 | **no** | DIFFERENT | +| 6 | **3/6** | **no** | DIFFERENT | +| 8 | **4/8** | **no** | DIFFERENT | +| 16 | **5/16** | **no** | DIFFERENT | + +c=16 replicates: `4818b621 ed4dd642 4818b621 2fbf99f4 ed4dd642 09745a44 2fbf99f4 +ec351630 09745a44 09745a44 ec351630 09745a44 09745a44 09745a44 09745a44 09745a44`. +The text is genuine garbage, e.g. `` 1:=\).\0).\).AI).00=1).".().).Ắ).IZ(isis) ``. + +### A1 — the #2776 default (`CU_STREAM_DEFAULT`) + +| band | distinct/total | self-consistent | vs c=1 mode | +|---|---|---|---| +| 2 | 1/2 | yes | SAME | +| 4 | 2/4 | no | DIFFERENT | +| 6 | **1/6** | **yes** | DIFFERENT | +| 8 | **1/8** | **yes** | DIFFERENT | +| 16 | **1/16** | **yes** | DIFFERENT | + +A2, A3 and A4 have the identical shape (A3's m≥6 answer is `94b918ea`, A1/A2/A4's is +`1bcbabf3`); full per-replicate hashes in the run log. + +**This is the headline. Ordered streams do what #2767 claims:** at c=16 the racy +default returns **5 distinct answers out of 16**, mostly garbage; the ordered default +returns **1 out of 16**, and it is coherent, correct Python. #2789's `GARBAGE` cells +are gone. What remains is not corruption — it is a *different valid completion*. + +--- + +## 5. Is "different from c=1" a defect? Not on the evidence — it is a near-tie + +Per the #2359 scar (four neighbouring prompts inverted a "GPU correctness defect" +into "the gate sampled a near-tie"), one prompt is an anecdote. Six prompts, A1 +config, c=1 mode of 5 replicates vs the mode of a c=8 band: + +| prompt | c=1 mode (n=5) | band-8 mode | verdict | b8 distinct | +|---|---|---|---|---| +| Python sum of a list | `fc42d43e` (4/5) | `1bcbabf3` (7/8) | **DIFFERENT** | 2 | +| Explain a hash table | `c68cd8e1` (5/5) | `c68cd8e1` (8/8) | SAME | 1 | +| Capital of France | `4804f8d3` (5/5) | `4804f8d3` (8/8) | SAME | 1 | +| Rust reverse a string | `9585e9e0` (5/5) | `37a97e76` (7/8) | **DIFFERENT** | 2 | +| Three primes > 100 | `2e80aea2` (5/5) | `2e80aea2` (8/8) | SAME | 1 | +| HTTP 404 | `e2870f22` (5/5) | `e2870f22` (8/8) | SAME | 1 | + +**4 of 6 prompts agree exactly between c=1 and c=8**, 8/8 self-consistent. + +The two that differ are the two open-ended code-generation prompts, and both +alternatives are coherent and correct: + +``` +c=1 : ```python\ndef sum_list(numbers):\n # Initialize the sum to 0\n total_sum = 0 ... +c=8 : ```python\ndef sum_list(numbers):\n return sum(numbers)\n``` +``` + +That is the signature of an **early argmax near-tie** flipped by floating-point +reduction order between an m=1 and an m=8 GEMM — not of a corrupted batched path. +The closed-form factual prompts, which have no near-tie, are bit-identical across +bands. **This is evidence, not proof**; a logprob-margin probe at the first divergent +position would settle it, and has not been run. + +### Incidental: `FP8_DECODE` changes the **c=1** answer, but only with `THR=32` + +A3 (`THR=32, FP8=0`) is the only arm whose c=1 mode differs (`08912486` vs +`fc42d43e`); A4 (`THR=def, FP8=0`) does not move. Since `fp8_will_fire = +gpu_profile.fp8_decode && m >= 5` (`batched_ffn.rs:111`, `batched_qkv.rs:119`, +`cublas_prefill/attention.rs:1021` at `a866988e4`), and a **prefill** of this ~20-token +prompt has m≈20 ≥ 5, FP8 fires at c=1 through prefill. `CUBLAS_GEMM_THRESHOLD=32` +simultaneously pushes that same m≈20 prefill **off** the cuBLAS route (default +threshold is 4). The interaction is consistent with that reading but was not +isolated — flagged, not concluded. + +--- + +## 6. Host conditions + +`load average 13.94` at start (over-subscription is provisioned; not a diagnosis). +**Disk `/` was at 100%, 3.6 GB → 561 MB free during the run**, driven by two +concurrent agents' cargo target dirs (45 GB and 39 GB). This is why the #2776 branch +was not built. Reported, not resolved. + +--- + +## 7. The plain answer + +> **Is there now ONE configuration correct across bands 1/4/8/16?** + +**Under the ladder as written: no — no arm passes.** But the ladder as written cannot +answer the question, because two of its three failure modes are its own: + +1. **band 1 fails in 5/5 arms** because the reference is the un-warmed first request, + which §4.4.2 mandates discarding (§3.1). Fix the harness and this disappears. +2. **bands 2 and 4 never form m=2/m=4** (§3.2), so those cells report on batches that + did not happen. +3. **bands 6/8/16 genuinely differ from c=1** — but under ordered streams they are + *1-distinct-out-of-16, coherent*, and 4 of 6 prompts show no difference at all. + +**What is now settled:** ordered streams (#2776, reached here as +`APR_STREAM_LEGACY=1`) removes the corruption. #2789's `GARBAGE` cells do not +reproduce at HEAD under any ordered arm. The ladder collapsed *most* of the way toward +one configuration, and the surviving disagreement is plausibly a near-tie rather than +a defect — the opposite end of the severity scale from where #2789 left it. + +**What is not settled, and blocks Arm A:** + +- **#2776 must land.** Everything above holds only with `APR_STREAM_LEGACY=1`; + main's *default* is still A0, which is unusable at every band. +- **The ladder needs §4.4.2 warmup and a formed-batch assertion** before its verdict + can gate anything. Right now it cannot tell a cold first request from a defect, and + it reports on bands it did not form. +- **The near-tie hypothesis needs a logprob-margin probe** to become a finding. If it + holds, the ladder's verdict should be "self-consistent within the band **and** + agreeing with c=1 wherever the argmax margin exceeds the m=1↔m=8 numeric delta" — + not raw string equality, which no correct batched implementation can satisfy on an + open-ended prompt. + +**No W1 receipt is recorded** (I-9: a band may not be re-run to green, and no cell may +be measured until the ladder is clean). The ladder is not yet clean, and two of the +three reasons are in the harness. + +--- + +## 8. Reproduce + +```bash +. scripts/apr_bin.sh || exit 1 # or pin an explicit --target-dir build +APR_STREAM_LEGACY=1 scripts/perf059_band_ladder.sh \ + --model /home/noah/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \ + --bands 1,2,4,6,8,16 --port 18802 --max-tokens 40 +``` + +Note `scripts/perf059_band_ladder.sh` is **not on `origin/main`** — it exists only on +`origin/feat/y1-7bgarbage` (`6fcb879b2`). A ladder that gates a release should not +live on one unmerged branch. diff --git a/evidence/perf-062/run-selfconsistency-ladder.txt b/evidence/perf-062/run-selfconsistency-ladder.txt new file mode 100644 index 0000000000..d1ac25bbde --- /dev/null +++ b/evidence/perf-062/run-selfconsistency-ladder.txt @@ -0,0 +1,96 @@ +ARM A0_racy route: APR_STREAM_LEGACY=0 + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) + CB-006-OUT lines: 0 + batches formed: 15 1 1 2 1 4 1 6 1 8 1 16 + c=1 replicates (n=15): 2 distinct; mode 14/15 + pre hashes: 94b918ea fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + post hashes: fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + c=1 MODE hash = fc42d43e + band 2: 1 distinct/2, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: dc325209 dc325209 + band 4: 2 distinct/4, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: d0371e5d d0371e5d cf2c89d3 cf2c89d3 + band 6: 3 distinct/6, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 424b45d9 bbdfdf91 bbdfdf91 c53f9793 c53f9793 424b45d9 + band 8: 4 distinct/8, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: f28f3cda a2961ecb a2961ecb a49938c6 885f56c0 f28f3cda 885f56c0 a49938c6 + band 16: 5 distinct/16, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 4818b621 ed4dd642 4818b621 2fbf99f4 ed4dd642 09745a44 2fbf99f4 ec351630 09745a44 09745a44 ec351630 09745a44 09745a44 09745a44 09745a44 09745a44 + SUMMARY A0_racy: c2=FAIL c4=FAIL c6=FAIL c8=FAIL c16=FAIL +ARM A1_thrdef_fpdef route: APR_STREAM_LEGACY=1 + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) + CB-006-OUT lines: 0 + batches formed: 18 1 1 3 1 6 1 8 1 16 + c=1 replicates (n=15): 2 distinct; mode 14/15 + pre hashes: 94b918ea fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + post hashes: fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + c=1 MODE hash = fc42d43e + band 2: 1 distinct/2, self-consistent=True, SAME as c=1 mode, empty=0 -> OK + hashes: fc42d43e fc42d43e + band 4: 2 distinct/4, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: fc42d43e 94b918ea 94b918ea 94b918ea + band 6: 1 distinct/6, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 8: 1 distinct/8, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 16: 1 distinct/16, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + SUMMARY A1_thrdef_fpdef: c2=OK c4=FAIL c6=FAIL c8=FAIL c16=FAIL +ARM A2_thr32_fpdef route: APR_STREAM_LEGACY=1 CUBLAS_GEMM_THRESHOLD=32 + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) + CB-006-OUT lines: 0 + batches formed: 18 1 1 3 1 6 1 8 1 16 + c=1 replicates (n=15): 2 distinct; mode 14/15 + pre hashes: 94b918ea fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + post hashes: fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + c=1 MODE hash = fc42d43e + band 2: 1 distinct/2, self-consistent=True, SAME as c=1 mode, empty=0 -> OK + hashes: fc42d43e fc42d43e + band 4: 2 distinct/4, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea fc42d43e 94b918ea 94b918ea + band 6: 1 distinct/6, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 8: 1 distinct/8, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 16: 1 distinct/16, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + SUMMARY A2_thr32_fpdef: c2=OK c4=FAIL c6=FAIL c8=FAIL c16=FAIL +ARM A3_thr32_fp0 route: APR_STREAM_LEGACY=1 CUBLAS_GEMM_THRESHOLD=32 FP8_DECODE=0 + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) + CB-006-OUT lines: 0 + batches formed: 18 1 1 3 1 6 1 8 1 15 + c=1 replicates (n=15): 2 distinct; mode 14/15 + pre hashes: a7ad49d4 08912486 08912486 08912486 08912486 08912486 08912486 08912486 08912486 08912486 + post hashes: 08912486 08912486 08912486 08912486 08912486 + c=1 MODE hash = 08912486 + band 2: 1 distinct/2, self-consistent=True, SAME as c=1 mode, empty=0 -> OK + hashes: 08912486 08912486 + band 4: 2 distinct/4, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea 94b918ea 94b918ea 08912486 + band 6: 1 distinct/6, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea + band 8: 1 distinct/8, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea + band 16: 1 distinct/16, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea 94b918ea + SUMMARY A3_thr32_fp0: c2=OK c4=FAIL c6=FAIL c8=FAIL c16=FAIL +ARM A4_thrdef_fp0 route: APR_STREAM_LEGACY=1 FP8_DECODE=0 + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) + CB-006-OUT lines: 0 + batches formed: 18 1 1 3 1 6 1 8 1 16 + c=1 replicates (n=15): 2 distinct; mode 14/15 + pre hashes: 94b918ea fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + post hashes: fc42d43e fc42d43e fc42d43e fc42d43e fc42d43e + c=1 MODE hash = fc42d43e + band 2: 1 distinct/2, self-consistent=True, SAME as c=1 mode, empty=0 -> OK + hashes: fc42d43e fc42d43e + band 4: 2 distinct/4, self-consistent=False, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 94b918ea 94b918ea 94b918ea fc42d43e + band 6: 1 distinct/6, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 8: 1 distinct/8, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + band 16: 1 distinct/16, self-consistent=True, DIFFERENT from c=1 mode, empty=0 -> FAIL + hashes: 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 1bcbabf3 + SUMMARY A4_thrdef_fp0: c2=OK c4=FAIL c6=FAIL c8=FAIL c16=FAIL +L2 ALL ARMS DONE diff --git a/evidence/perf-062/run-strict-ladder.txt b/evidence/perf-062/run-strict-ladder.txt new file mode 100644 index 0000000000..b9016a3ba8 --- /dev/null +++ b/evidence/perf-062/run-strict-ladder.txt @@ -0,0 +1,242 @@ +########## ARM A0_racy_default ########## +ARM A0_racy_default RC=1 +PERF-059 band ladder + binary : /tmp/claude-1000/-home-noah-src-aprender/2256679c-128a-431a-b273-4d5e2aa6fdcd/scratchpad/y5-pin/apr.a866988e4 + model : /home/noah/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf + ctx : 2048 max_tokens: 40 + route : APR_STREAM_LEGACY=0 CUBLAS_GEMM_THRESHOLD=default FP8_DECODE=default + dispatch: gpu-layers: requested=all resolved=28 total=28 (backend=cuda) +reference: c=1 +reference text: ```python +def sum_list(numbers): + # Initialize the sum to 0 + total_sum = 0 + # Iterate through each number in th + FAIL band=1: 1/1 diverged, 0/1 empty + first divergent: ```python +def sum_list(numbers): + # Initialize the sum to 0 + total_sum = 0 + + # Iterate through each number + FAIL band=2: 2/2 diverged, 0/2 empty + first divergent: 1:=\).\0).\).AI).00=1).".().).Ắ).IZ(isis).).ir big acuerdo two(\" pri(isis baño + FAIL band=4: 4/4 diverged, 0/4 empty + first divergent: 10dexeratalinerical destinyideroicalericalỹ\1-producingical1ilerceo soonerical).1xили