ARCH: quant-type dispatch is duplicated across ~30 files with no single source of truth — propose a dedicated consolidation release

[alfredodeza]

tl;dr

Aprender has no equivalent of llama.cpp's ggml_type_traits[] — the single
table that tells the entire codebase what a quant type is (block size,
byte size, dequant fn) once, so every operation and every model architecture
gets it for free. Instead, ~30 files in crates/aprender-serve/src/ each
independently match on quant type. The result is that "does architecture X
work" is not actually one question — it's architectures × quant_types × backends × call_sites, and today most of that matrix is untested. Five
closed tickets (#1749, #1789, #2535, #3341, #3091) are the same defect class
recurring in different cells of that matrix, each discovered by a user
crash rather than by a gate that could have covered all of them at once.

This is not a "fix Qwen" ticket — #3090/#3091/#3413 stay where they are for
the specific model-family defects. This is the structural ticket: the
dispatch architecture itself is the reason those keep happening, and it
will keep happening for every future model (Qwen4 preview is already sitting
in a Downloads folder untested) unless the dispatch is consolidated once.

Evidence

1. The size gap between "new architecture" costs, aprender vs llama.cpp

llama.cpp's entire Qwen3-MoE architecture — tensor shapes, attention, RoPE,
MoE routing — is 179 lines (src/models/qwen3moe.cpp in a local llama.cpp
checkout, current as of 3173a5647 2026-08-29). It's short because it's
almost entirely calls into shared, quant-agnostic building blocks
(build_qkv, build_attn, build_moe_ffn, ggml_rope_ext) that every one
of llama.cpp's ~50 architectures reuses unchanged.

Aprender's Qwen3-MoE-specific code alone
(crates/aprender-serve/src/gguf/qwen3_moe_load.rs +
crates/aprender-serve/src/infer/qwen3_moe_generate.rs) is 1,678 lines —
before counting anything it depends on.

2. Quant-type dispatch is scattered, not centralized

$ grep -rln "GGUF_TYPE_Q4_K\|GGUF_TYPE_Q4_0\|match qtype\|match.*\.qtype" \
    crates/aprender-serve/src/ --include="*.rs" | grep -v test | wc -l
30

30 files, each with its own quant-type match arms:
gguf/transformer.rs, gguf/inference/fused_matmul_into.rs,
gguf/inference/matmul_fused.rs, gguf/qwen3_moe_load.rs,
gguf/metadata.rs, cuda/executor/layers/gemv_dispatch.rs,
gpu/adapters/wgpu_adapter.rs, and 23 more (full list in the investigation
that produced this ticket — available on request / in session transcript).

llama.cpp's equivalent is ggml_type_traits[]
(ggml/src/ggml.c:~750-900) — one array, one entry per type
(type_size, blck_size, to_float, from_float), read generically by
every operation in the compute graph.

Aprender has a partial version of this idea —
QuantBlockFormat (crates/aprender-serve/src/quantize/format_trait.rs) —
but only 5 of the ~30 call sites are known to route through it consistently,
and the trait itself only models affine quantization (x = d*s*q - dmin*m: Q4_0/Q8_0/Q4_K/Q5_K/Q6_K). It cannot represent the IQ family
(IQ2_XXS/XS, IQ3_XXS/S, IQ4_NL/XS), which are codebook/lattice
quantization — each block indexes into a fixed grid table, not a per-block
affine scale. That's a second trait shape needed, not an extension of the
first.

3. The universe aprender tracks is smaller than the universe it needs to track

$ grep -rohE "GGUF_TYPE_[A-Z0-9_]+" crates/aprender-serve/src/ | sort -u
GGUF_TYPE_ARRAY  GGUF_TYPE_BF16  GGUF_TYPE_F16  GGUF_TYPE_F32
GGUF_TYPE_INT32  GGUF_TYPE_Q2_K  GGUF_TYPE_Q3_K  GGUF_TYPE_Q4_0
GGUF_TYPE_Q4_1   GGUF_TYPE_Q4_K  GGUF_TYPE_Q5_0  GGUF_TYPE_Q5_1
GGUF_TYPE_Q5_K   GGUF_TYPE_Q6_K  GGUF_TYPE_Q8_0  GGUF_TYPE_STRING
GGUF_TYPE_UINT32

16 constants named anywhere in the crate. The GGUF spec defines 30+
(missing here: the entire IQ1/IQ2/IQ3/IQ4 family, Q8_1, Q8_K, TQ1_0/TQ2_0).
Every one of those missing types is a RealizarError::UnsupportedOperation
waiting to be hit by the next real-world download, in whichever of the 30
files happens to load the tensor first.

4. The ticket pattern — same defect, different cells of the matrix

Issue	Architecture	Broke because
#1749	MoE (generic)	matmul_fused.rs:211 index OOB on MoE GGUF
#1789	Qwen3-Coder-30B MoE	Same line, F32 weight this time
#2535	MoE (generic)	Misdiagnosed as "truncated/corrupt" — the real cause was an unhandled tensor shape/qtype combination
#3341	Qwen3-Coder-30B MoE	Q4_0 experts refused — the MoE loader's qtype list only had Q4_K/Q6_K
#3091	Qwen3.5 (qwen35)	IQ2_XS/IQ4_XS refused in tensor_byte_size — that function's qtype list only had 9 of the ~30 real types

Five tickets, five separate fixes, one root cause each time: "this
particular file's quant-type list was incomplete." None of these fixes
closed the class — each is scoped to the one file that crashed. The next
new model (Qwen4-preview architecture is already sitting untested in
~/Downloads/Qwen3.8-Flash-Next-UD-Q2_K_XL-...gguf, per #3413's sibling
investigation) will find the next incomplete list, in some 31st file.

Proposed consolidation

1. A QuantCodebookFormat trait alongside the existing
   QuantBlockFormat, covering lattice/codebook quant types. Grid tables
   are portable data — llama.cpp's ggml-quants.c iq2xs_grid/iq3xs_grid
   etc. are the reference, not code to be creatively reinvented.
2. A single dispatch entry point analogous to ggml_type_traits[]:
   given a qtype: u32, return {byte_size, dequant_to_f32, family} in one
   place. Every one of the ~30 call sites is refactored to call through it
   instead of maintaining its own arm list.
3. A completeness gate: a test (or pv contract) that enumerates every
   GGUF_TYPE_* the format spec defines and asserts the dispatch table has
   an entry for each — so a missing type is a CI failure at merge time, not
   a user's crash report three months later. This is the mechanical
   backstop; CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091's own history shows the honor-system approach (each file
   remembering to update its own list) has failed five times already.
4. Per-backend fan-out is then additive, not multiplicative: CPU
   SIMD/CUDA/wgpu each implement the trait once per type; the ~30
   call-site refactor happens once regardless of how many backends exist.

Why this should be its own release, not squeezed into the current cadence

Per docs/specifications/06x-release-schedule.md, this project runs a
2-3 day release cadence, 24/7 (v0.66.0 → v0.70.0 in the current
train). That cadence is built for landing scoped, independently-shippable
fixes — exactly the shape of #3091, #3341, #3413. It is the wrong cadence
for this ticket, for three reasons:

• Blast radius. The refactor touches ~30 files across the CPU, CUDA,
  and wgpu backends simultaneously — nearly every file in
  crates/aprender-serve/src/gguf/, quantize/, cuda/, and gpu/.
  Landing it incrementally alongside unrelated model-fix PRs on a 2-3 day
  cadence means every one of those PRs is rebasing against a moving
  dispatch layer, and every dequant/matmul kernel needs re-verification
  against the same parity bar this project already holds itself to (per
  CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091's own history: cosine ≥0.98 vs llama.cpp, per quant type, per
  backend). That verification matrix (≥9 missing quant types × 3 backends)
  is itself real work, not a rename.
• Regression risk to the one thing that currently works. Qwen2.5-Coder
  (README's demonstrated, BEATS.md-measured, GPU-parity model) runs through
  several of these same 30 files today. A consolidation that isn't done in
  one dedicated, fully-gated pass risks the exact kind of silent regression
  this ticket exists to prevent — on the one model line that currently has
  real proof behind it.
• It competes for the same attention the model-specific tickets need.
  GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090 (GPU), P0: dense Qwen3 (Qwen3ForCausalLM) has no current proof it runs — last verification is 7 months stale and CI never covers it #3413 (dense Qwen3 proof), and whatever Qwen4-preview turns
  into all sit downstream of this dispatch layer. Doing them first means
  redoing parts of them once this lands; doing this first means every
  future model ticket gets cheaper, not just the current ones.

Ask: reserve one full release cycle (e.g. a dedicated milestone, not
squeezed as a rider into 0.69/0.70/0.71) where this consolidation is the
only architectural change landing, with model-specific bug fixes limited
to true P0 hotfixes during that window. The exit criterion is the
completeness gate in (3) above going green, plus a full parity re-run of
the existing Qwen2.5-Coder GPU-beat measurement to prove no regression.

Related

• CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091, GPU: implement Gated DeltaNet / SSM inference for Qwen3.5 / Qwen3.8 hybrid GGUFs #3090, feat: Qwen3.5 (qwen35) SSM/Gated Delta Net inference support #3038 — Qwen3.5 CPU/GPU (the immediate trigger for this
  investigation)
• P0: dense Qwen3 (Qwen3ForCausalLM) has no current proof it runs — last verification is 7 months stale and CI never covers it #3413 — dense Qwen3 proof-of-life gap (downstream of the same dispatch
  layer)
• defect: apr bench panics on MoE GGUF models (matmul_fused.rs:211 index OOB) #1749, apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight #1789, MoE GGUF misdiagnosed as 'truncated/corrupt … file is incomplete' — the file is provably complete, and apr validate agrees #2535, CPU+GPU: Qwen3-Coder-30B MoE (Q4_0) fails with empty-buffer error on blk.0.ffn_gate.weight — possible #1789 regression #3341 — prior instances of the same defect class
• Docs: publish a GPU-vs-CPU model architecture support table (no user-facing doc exists today) #3077 — GPU/CPU model support table (would become truthful, not
  aspirational, once (3)'s completeness gate exists)

[noahgift]

PP-QUANT-001 — Phase 0/1 ground truth (research run, 2026-09-17). Verdict: STOPPED(framing-falsified)

Measured on origin/main @ c79a53223, v0.68.0 tagged 2026-09-17T06:35Z; llama.cpp @ 3173a5647. Marks: [V] verified by the command shown · [C] computed from a cited artifact · [U] unverified. Original figures are struck, never silently replaced. No code, spec, or issue body was edited.

1. The ticket's numbers, re-derived

claim	original	measured
dispatch files (serve, non-test)	30	27 non-test / 58 total name GGUF_TYPE_* — gguf/ 11, gguf/inference 10, gguf/cuda 4, convert 1, gpu/adapters 1 [V] git grep -l -E 'GGUF_TYPE_[A-Z0-9_]+' origin/main -- 'crates/aprender-serve/src/*.rs'
type constants	16	13 tensor-type consts + 4 GGUF metadata value-type names (UINT32 STRING INT32 ARRAY) — two different id spaces counted as one [V]
spec universe	"30+"	GGML_TYPE_COUNT = 43: 35 live, 8 removed (4, 5, 31–33, 36–38). New since January: NVFP4=40, Q1_0=41, Q2_0=42 [V] git show 3173a5647:ggml/include/ggml.h
QuantBlockFormat	"5 of ~30 call sites"	5 impls (Q4K Q5K Q6K Q4_0 Q8_0); used in 5 non-test files, all inside quantize/ — zero users outside it [V]
LOC gap	1,678 vs 179	confirmed: 973 + 705 vs 179 [V] — but see §3

2. The universe is larger than the ticket says — three enums, not one crate [V]

enum	ids named
serve GgmlQuantType (gguf/types.rs:101)	16 — 0–3, 6–14, 16, 17, 30
core GgmlType (format/gguf/types.rs:55, the import path)	12 — 0–3, 8, 12, 14, 24–28 (no Q5_0/Q5_1/Q2_K/Q3_K/Q5_K)
compute GgmlType (inference/gguf.rs:27)	15 — 0–3, 6–15, 30

Plus 14 further Quant*/Quantization* enums across cbtop, core, cuda-edge, qa-runner, registry, train. 19 of 35 live spec ids are unnamed even in serve. A single table inside aprender-serve/src/quantize/ cannot be the source of truth while the importer (core) and the kernels (compute) keep their own; it has to sit below all three. Also: bare-integer dispatch (12 =>, qtype == 2) exists beside the constants — a regex upper bound of 558 non-test lines across 10+ crates [U] (noisy; needs a case table) — so a ban gate keyed on GGUF_TYPE_* alone would not see it.

3. Five-whys on the cited instances: 1 of 5 is "local qtype list incomplete" [C] (from each fix PR's own root-cause text; diffs not re-read)

issue	terminal cause per its fix	class
#1749	dense forward ran on an MoE model → placeholder FFN tensors with empty buffers (#1751)	arch dispatch
#1789	same placeholder; serve endpoint never routed to the MoE path (#1790, #1806)	arch dispatch
#2535	validator ignored the documented MoE-placeholder contract — #2541: "the architecture dispatch was never at fault … only the validator was wrong"	placeholder contract
#3341	matvec_for_qtype fallback arm on Q4_0 experts; fixed by single-source SUPPORTED_EXPERT_QTYPES + drift test (#3405)	qtype list
#3091	entire qwen35 architecture refused; fix is a new forward pass (#3114)	arch

Four of five share a different class: MoE placeholder tensors (byte_size=0, qtype=0) reaching dense-path consumers. A quant traits table would not have prevented #1749, #1789, #2535 or #3091. The duplication in §2 is real and worth fixing, but the defect history does not support it as the cause of the user-found crashes, and it does not by itself justify a must-carry slot. The run's own rule for this outcome is: report, do not plan a release around it.

4. The dedicated-release ask

Not needed either way. CI on main: p50 31 min, p95 62 min (n=31 successful of the last 50 push runs) [V] → ~69 serial slots per 72 h window, and the merge queue builds 3 in parallel. A table → ban gate → per-directory migration ordering handles rebase churn without stopping the train.

5. Proposed re-scope (for #3421 / PR #3420 — not applied; both are human-authored)

1. Split the evidence: a placeholder-tensor ticket (type-level: a dense consumer cannot receive an MoE placeholder) owns defect: apr bench panics on MoE GGUF models (matmul_fused.rs:211 index OOB) #1749/apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight #1789/MoE GGUF misdiagnosed as 'truncated/corrupt … file is incomplete' — the file is provably complete, and apr validate agrees #2535; PP-QUANT-001 keeps CPU+GPU: Qwen3-Coder-30B MoE (Q4_0) fails with empty-buffer error on blk.0.ffn_gate.weight — possible #1789 regression #3341 only.
2. PP-QUANT-001 Phase 0 inventories three crates, not one, and separates tensor-type ids from metadata value-type ids before any gate is designed.
3. The LOC gap (1,678 vs 179) is a graph-builder gap, not a quant gap — its own ticket.
4. Design positions (one closed enum + traits table + tier-0 dequant fallback, bit-exact dequant fixtures, resolve-at-load) are unreviewed: no quorum ran. Nothing here is a ruling.

Not done: per-call-site table, spec revision, quorum, child tickets. Also noted: #3091 is still open in the 0.68.0 milestone though 0.68.0 has shipped; APR-RELEASE-001 on main has no §1.5 must-carry table for a 0.69 row to land in.

[alfredodeza]

This ticket is the umbrella problem statement for "new architectures are too
expensive to add" — two sub-problems have since been evidenced and are tracked
under it, each with its own spec and epic (design docs stay separate; this issue
is the one narrative entry point):

• Quant-type dispatch (this ticket's original scope) — docs/specifications/PP-QUANT-001-MASTER.md, epic EPIC: PP-QUANT-001 — quant-type dispatch, Phase 0/1 (0.69 train) #3421, milestone 0.69.0 for Phase 0/1 (done: PR spec(PP-QUANT-001): quant-type dispatch consolidation — design, contract, Phase 0 gate #3420), milestone Inference dispatch & architecture consolidation for Phase 2/3 (the 30-file migration + parity re-proof — the large-blast-radius part, not squeezed into a train).
• Shared-block forward-path duplication (ARCH: shared attention/FFN building blocks exist but 0% adopted — new architectures still cost ~1,600 lines each #3422) — the bigger factor: of this ticket's own 1,678-line Qwen3-MoE count, only ~85 lines are quant dispatch; the rest is forward logic that reimplements attention/softmax/RoPE per architecture despite a working shared layer (attention.rs, ffn_block.rs) already existing and sitting at 0% adoption. docs/specifications/PP-ARCH-001-MASTER.md, epic EPIC: Inference dispatch & architecture consolidation (dedicated cycle) #3423, milestone Inference dispatch & architecture consolidation.

Both land out of the same dedicated milestone once past their low-risk Phase 0/1
work, tracked jointly by epic #3423. #3422 is not independent scope — it's the
other half of what this ticket asked for ("easier to add new model support").

[noahgift]

Follow-up (2026-09-17, PMAT-3427) — corrections to my 09:20Z comment, and where this now lives

Corrections to my own numbers. (a) I classed #3091 as "arch", giving a 4/1 split from PR bodies. Read from the fix diffs, plus today's reopen, it is 3 placeholder / 2 qtype-list — so "framing falsified" was too strong: the qtype-list class is real and recurring, it is just the smaller of two. (b) Spec ids: 35 live + 8 removed, not 34 + 9. (c) Live ids named by no aprender enum: 13; unnamed in serve alone: 19.

New since then [V]

• CPU: Qwen3.5 / Qwen3.8 hybrid GGUFs are refused entirely — no fallback exists #3091 reproduces on the published v0.68.0 asset with a 338 MB public model (unsloth/Qwen3.5-0.8B-GGUF, UD-IQ2_XXS → exit 8, type 16; UD-Q4_K_XL → type 23). tensor_byte_size has 11 arms and lacks Q3_K and Q5_1, which the crate defines; there are three such functions.
• The placeholder mechanism: dense_ffn_placeholder is byte_size: 0, qtype: 0, and 0 is F32. feat(apr-cli): bench MoE dispatch — routes Qwen3-MoE GGUFs through forward_qwen3_moe (#1749) #1751, fix(serve): #1789 matmul defensive guard against empty / undersized weights #1790 and fix(serve): a complete MoE model was rejected as "truncated/corrupt" (#2535) #2541 each guard or re-route one consumer; none removes it. 63 non-test files read the dense FFN fields; 14 carry no guard token, including the CUDA and wgpu consumers.
• PR spec(PP-QUANT-001): quant-type dispatch consolidation — design, contract, Phase 0 gate #3420's oracle and my constant-keyed scan each miss what the other finds (30 vs 27 files, 24 shared, union 33) — detail in the review there.

Where it lives

what	where
0.69.0 must-carry slice M	#3429 fixtures → #3430 one GgmlType + TRAITS[43] → #3431 reconciliation gate → #3432 byte size + single refusal
placeholder class (new sibling epic)	#3428 — T1 #3433, T2 #3434 (0.69.0), T3 #3435 (0.70.0)
ratcheted follow-on (0.70.0)	retype per directory #3436–#3440 · tier-0 dequant 35/35 #3441 · upstream drift job #3442 · generated support table #3443
process	#3445 a tag was cut with open milestone issues · #3446 APR-RELEASE-001 has no must-carry section on main
rule + row + tables	draft PR #3447 (06x-release-schedule.md §1.1a; receipt with Tables G and S)
decisions needed from a human	one comment on #3421

The shared-block / LOC-gap half is Alfredo's #3422 (PP-ARCH-001); my duplicate #3444 is closed in its favour.

Still true: no dedicated release is needed for throughput reasons (CI p95 62 min, ~69 slots per 72 h, queue 3 wide). Still not done: no independent quorum has reviewed the design positions — everything above is one reviewer's measurement, and PR #3447 stays draft until that happens.

[noahgift]

Filed #3945 as the test half of this ticket: an enumerated (qtype × op × backend × shape) conformance table derived from the dispatch registry, so admitting a quant type automatically creates its proof obligations.

Your framing is the specification and I am citing it verbatim there — "'does architecture X work' is not actually one question — it's architectures × quant_types × backends × call_sites, and today most of that matrix is untested." What was missing is the thing that enumerates it; llama.cpp pairs ggml_type_traits[] with test-backend-ops, which auto-enumerates (op × type × shape) and compares backend against CPU by NMSE. You asked for the first half; #3945 is the second.

The 0.69.1 release measured the cost this week: the GPU-eligibility whitelist turned out to be recorded in nine places (the commit that warned about six undercounted), the ninth was invisible to three audits because it names a type nothing had changed, and dtype.rs — which calls itself the single source of truth — contradicted its own expression in both directions. Also noted on #3421: its Phase 0 checkbox is marked complete while PR #3420 is still open and the contract is not on main.