From 7de9459b3225f40b449a1621516f3bb5e6ccc64f Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Wed, 16 Sep 2026 16:55:10 +0200 Subject: [PATCH 1/6] =?UTF-8?q?feat(crux):=20AutoGluon=20becomes=20a=20CRU?= =?UTF-8?q?X=20competitor=20=E2=80=94=20category=20O,=2024=20contracts,=20?= =?UTF-8?q?25=20tickets,=20registry=20edit=20+=20mutation=20proof?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Research of ../autogluon (1.6.3 @ 77946149) as a competitive-research source for aprender, landed the way category N landed linfa and burn (#3169): the competitor is admitted to the CLOSED registry, every story is a real contract with falsification gates, every story is a registry row, and every missing/partial story has a GitHub issue and a roadmap fragment. What AutoGluon is, measured from the tree rather than recalled: three predictors. TabularPredictor (65 public methods, 11 presets, 24 model families of which 8 are tabular foundation models added in 1.4-1.6), TimeSeriesPredictor (Chronos-2/Toto-2 pretrained, 30+ local/deep models, 16 metrics incl. WQL/MASE/RMSSE, auto backtesting since 1.5) and MultiModalPredictor. Evidence under evidence/crux/autogluon/. What aprender has, measured at eb262f8eb: automl/ is a single-estimator hyperparameter tuner (TPE, grid, random, DE, TimeBudget, EarlyStopping); time_series/ is one univariate ARIMA; encoders, calibration, SHAP/LIME/ permutation importance and KFold/cross_validate exist as building blocks. No predictor-level fit(label), no leaderboard, no bagging, stacking or greedy weighted-ensemble selection, no panel forecasting, no quantile forecast metrics. The gap is the AutoML UX, not the algorithms. Category O — AutoML Parity — 24 stories: 9 P0 (the README hello-world: fit(label), problem-type inference, presets, leaderboard, feature pipeline, weighted ensemble, time budget, panel forecaster, quantile metrics), 9 P1 (bagging, stacking, importance, threshold calibration, deployment artifact, tabular foundation model, backtesting, local baselines, pretrained forecaster), 6 P2 (refit_full, distill, infer_limit, fit diagnostics, memory-aware fit, covariates). MultiModalPredictor, autogluon.cloud, MLZero and Ray-parallel fits are CUT on the epic with reasons. CRUX_COMPETITORS: [&str; 14] -> [&str; 15] + autogluon Not a BEAT pillar: aprender claims no pinned-benchmark win over AutoGluon. Both registry tests that keep BEAT_INCUMBENTS and CRUX_COMPETITORS apart are extended, not worked around. Tickets: epic #3370, stories #3371-#3394, label pareto-autogluon. Roadmap: 25 fragments under docs/roadmaps/entries/, roadmap.yaml regenerated by the aggregator (idempotent check passes). Spec: docs/specifications/crux-competitive-research-ux-workflows.md v2.2 -> v2.3 — §3 gains rows for linfa+burn (category N, which #3169 never recorded there) and AutoGluon; §5 gains Category O; §6 notes that coverage_intake in the YAML is the source of truth. coverage_intake 267 -> 291 (partial 72 -> 77, missing 152 -> 171). Verification: - pv built from THIS tree validates 25/25 (24 new + master). The stale ~/.cargo/bin/pv rejects crux-O-01 with CRUX-002 — the behavioural delta proves the registry edit engaged. - Mutation-verified: deleting "autogluon" from CRUX_COMPETITORS turns competitor_registry_covers_the_corpus_vocabulary RED with "autogluon is used by contracts/ and must stay in CRUX_COMPETITORS" and the_real_crux_registry_rows_are_all_in_domain RED. Restored: 20/20. - cargo test -p aprender-contracts --lib: 1526 passed, 0 failed. - Every falsification gate is LIVE-PENDING prose (no `::`), so strict-test-binding has nothing to refuse; the obligations are RECORDED as unfalsifiable-by-absence, not satisfied. - README CONTRACT_COUNT regenerated 1835 -> 1866 by readme_sync.sh. - Guards: roadmap fragment/ids/sorted/additive/completion, contract test-binding and enforcement, shell-lint ratchet, hardcoded paths, readme claims, grep -q ratchet — all rc=0. Pmat-Ticket: PMAT-3370 Co-Authored-By: Claude Fable 5.1 --- README.md | 4 +- contracts/crux-O-01-v1.yaml | 94 ++++ contracts/crux-O-02-v1.yaml | 75 +++ contracts/crux-O-03-v1.yaml | 75 +++ contracts/crux-O-04-v1.yaml | 94 ++++ contracts/crux-O-05-v1.yaml | 94 ++++ contracts/crux-O-06-v1.yaml | 75 +++ contracts/crux-O-07-v1.yaml | 75 +++ contracts/crux-O-08-v1.yaml | 75 +++ contracts/crux-O-09-v1.yaml | 75 +++ contracts/crux-O-10-v1.yaml | 75 +++ contracts/crux-O-11-v1.yaml | 75 +++ contracts/crux-O-12-v1.yaml | 75 +++ contracts/crux-O-13-v1.yaml | 75 +++ contracts/crux-O-14-v1.yaml | 75 +++ contracts/crux-O-15-v1.yaml | 56 ++ contracts/crux-O-16-v1.yaml | 75 +++ contracts/crux-O-17-v1.yaml | 75 +++ contracts/crux-O-18-v1.yaml | 75 +++ contracts/crux-O-19-v1.yaml | 75 +++ contracts/crux-O-20-v1.yaml | 75 +++ contracts/crux-O-21-v1.yaml | 75 +++ contracts/crux-O-22-v1.yaml | 75 +++ contracts/crux-O-23-v1.yaml | 94 ++++ contracts/crux-O-24-v1.yaml | 75 +++ .../crux-competitive-research-ux-v1.yaml | 46 +- .../src/schema/crux_intake_tests.rs | 4 + .../src/schema/validator.rs | 12 +- docs/roadmaps/entries/PMAT-3370.yaml | 22 + docs/roadmaps/entries/PMAT-3371.yaml | 21 + docs/roadmaps/entries/PMAT-3372.yaml | 21 + docs/roadmaps/entries/PMAT-3373.yaml | 21 + docs/roadmaps/entries/PMAT-3374.yaml | 21 + docs/roadmaps/entries/PMAT-3375.yaml | 21 + docs/roadmaps/entries/PMAT-3376.yaml | 21 + docs/roadmaps/entries/PMAT-3377.yaml | 21 + docs/roadmaps/entries/PMAT-3378.yaml | 21 + docs/roadmaps/entries/PMAT-3379.yaml | 21 + docs/roadmaps/entries/PMAT-3380.yaml | 21 + docs/roadmaps/entries/PMAT-3381.yaml | 21 + docs/roadmaps/entries/PMAT-3382.yaml | 21 + docs/roadmaps/entries/PMAT-3383.yaml | 21 + docs/roadmaps/entries/PMAT-3384.yaml | 21 + docs/roadmaps/entries/PMAT-3385.yaml | 21 + docs/roadmaps/entries/PMAT-3386.yaml | 21 + docs/roadmaps/entries/PMAT-3387.yaml | 21 + docs/roadmaps/entries/PMAT-3388.yaml | 21 + docs/roadmaps/entries/PMAT-3389.yaml | 21 + docs/roadmaps/entries/PMAT-3390.yaml | 21 + docs/roadmaps/entries/PMAT-3391.yaml | 21 + docs/roadmaps/entries/PMAT-3392.yaml | 21 + docs/roadmaps/entries/PMAT-3393.yaml | 21 + docs/roadmaps/entries/PMAT-3394.yaml | 21 + docs/roadmaps/roadmap.yaml | 526 ++++++++++++++++++ .../crux-competitive-research-ux-workflows.md | 43 +- evidence/crux/autogluon/README.md | 3 + evidence/crux/autogluon/api-surface.md | 46 ++ evidence/crux/autogluon/hello.sh | 14 + evidence/crux/autogluon/readme-verbs.txt | 23 + scripts/crux_scaffold_contracts.py | 1 + 60 files changed, 3095 insertions(+), 10 deletions(-) create mode 100644 contracts/crux-O-01-v1.yaml create mode 100644 contracts/crux-O-02-v1.yaml create mode 100644 contracts/crux-O-03-v1.yaml create mode 100644 contracts/crux-O-04-v1.yaml create mode 100644 contracts/crux-O-05-v1.yaml create mode 100644 contracts/crux-O-06-v1.yaml create mode 100644 contracts/crux-O-07-v1.yaml create mode 100644 contracts/crux-O-08-v1.yaml create mode 100644 contracts/crux-O-09-v1.yaml create mode 100644 contracts/crux-O-10-v1.yaml create mode 100644 contracts/crux-O-11-v1.yaml create mode 100644 contracts/crux-O-12-v1.yaml create mode 100644 contracts/crux-O-13-v1.yaml create mode 100644 contracts/crux-O-14-v1.yaml create mode 100644 contracts/crux-O-15-v1.yaml create mode 100644 contracts/crux-O-16-v1.yaml create mode 100644 contracts/crux-O-17-v1.yaml create mode 100644 contracts/crux-O-18-v1.yaml create mode 100644 contracts/crux-O-19-v1.yaml create mode 100644 contracts/crux-O-20-v1.yaml create mode 100644 contracts/crux-O-21-v1.yaml create mode 100644 contracts/crux-O-22-v1.yaml create mode 100644 contracts/crux-O-23-v1.yaml create mode 100644 contracts/crux-O-24-v1.yaml create mode 100644 docs/roadmaps/entries/PMAT-3370.yaml create mode 100644 docs/roadmaps/entries/PMAT-3371.yaml create mode 100644 docs/roadmaps/entries/PMAT-3372.yaml create mode 100644 docs/roadmaps/entries/PMAT-3373.yaml create mode 100644 docs/roadmaps/entries/PMAT-3374.yaml create mode 100644 docs/roadmaps/entries/PMAT-3375.yaml create mode 100644 docs/roadmaps/entries/PMAT-3376.yaml create mode 100644 docs/roadmaps/entries/PMAT-3377.yaml create mode 100644 docs/roadmaps/entries/PMAT-3378.yaml create mode 100644 docs/roadmaps/entries/PMAT-3379.yaml create mode 100644 docs/roadmaps/entries/PMAT-3380.yaml create mode 100644 docs/roadmaps/entries/PMAT-3381.yaml create mode 100644 docs/roadmaps/entries/PMAT-3382.yaml create mode 100644 docs/roadmaps/entries/PMAT-3383.yaml create mode 100644 docs/roadmaps/entries/PMAT-3384.yaml create mode 100644 docs/roadmaps/entries/PMAT-3385.yaml create mode 100644 docs/roadmaps/entries/PMAT-3386.yaml create mode 100644 docs/roadmaps/entries/PMAT-3387.yaml create mode 100644 docs/roadmaps/entries/PMAT-3388.yaml create mode 100644 docs/roadmaps/entries/PMAT-3389.yaml create mode 100644 docs/roadmaps/entries/PMAT-3390.yaml create mode 100644 docs/roadmaps/entries/PMAT-3391.yaml create mode 100644 docs/roadmaps/entries/PMAT-3392.yaml create mode 100644 docs/roadmaps/entries/PMAT-3393.yaml create mode 100644 docs/roadmaps/entries/PMAT-3394.yaml create mode 100644 evidence/crux/autogluon/README.md create mode 100644 evidence/crux/autogluon/api-surface.md create mode 100644 evidence/crux/autogluon/hello.sh create mode 100644 evidence/crux/autogluon/readme-verbs.txt diff --git a/README.md b/README.md index 0d802a47a5..45e714f570 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ publishing — all backed by YAML provable contracts that fail CI on drift. | Metric | Count | Source of truth | |-------:|------:|---| | Workspace crates | **79** workspace crates | `cargo metadata --no-deps` (NOT `ls crates/` — 4 are `exclude`d, 1 has no Cargo.toml) | -| Provable contracts | **1842** provable contracts | `find contracts/ -name '*.yaml'` (generated by `make readme-sync`, guarded by `scripts/check_readme_claims.sh`) | +| Provable contracts | **1866** provable contracts | `find contracts/ -name '*.yaml'` (generated by `make readme-sync`, guarded by `scripts/check_readme_claims.sh`) | | CLI commands | **110** CLI commands | `apr --help` | | Book CLI chapters | **112** chapters | `ls book/src/cli/*.md` | | Book lib chapters | **71** chapters | `ls book/src/lib/*.md` (parity with `pub mod`) | @@ -262,7 +262,7 @@ falsification_tests: prediction: apr validate bad-model.apr exits non-zero ``` -The tree carries 1842 contracts across inference, training, quantization, attention, FFN, +The tree carries 1866 contracts across inference, training, quantization, attention, FFN, tokenization, model formats, CLI safety — and this README itself. ## Migration from old crates diff --git a/contracts/crux-O-01-v1.yaml b/contracts/crux-O-01-v1.yaml new file mode 100644 index 0000000000..e317f539e5 --- /dev/null +++ b/contracts/crux-O-01-v1.yaml @@ -0,0 +1,94 @@ +# CRUX-O-01 — One-call tabular AutoML: fit(label) -> predict on a CSV +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3371, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-01 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3371 + description: > + One-call tabular AutoML: fit(label) -> predict on a CSV. Competitor verb: TabularPredictor(label="class").fit("train.csv"); predictor.predict("test.csv"). Gap measured in aprender at eb262f8eb: No predictor-level AutoML entry point exists. crates/aprender-core/src/automl/ is a hyperparameter TUNER (AutoTuner, TPE, GridSearch, RandomSearch, DESearch, TimeBudget) that tunes ONE estimator the caller already chose; nothing takes a labelled table and returns a fitted model. `apr train` is causal-LM pre-training only (crates/apr-cli/src/commands/train.rs:1-5) and `apr finetune --task classify` is text classification. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3371' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + fit_on_a_labelled_csv_returns_a_predictor_whose_predictions_: + formula: | + on the iris fixture (crates/aprender-core/src/datasets/iris.csv) accuracy >= 0.90 AND the majority-class baseline is asserted at 0.333 in the same test + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline" + the_same_call_works_for_a_regression_label_without_a_problem: + formula: | + a numeric label column yields a regressor whose R^2 on a held-out split exceeds 0.5 AND the mean-predictor baseline is asserted at ~0.0 + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the same call works for a regression label without a problem_type argument" + apr_automl_fit_is_reachable_from_the_cli: + formula: | + `apr automl fit --label class train.csv --out model.apr` exits 0 and writes a loadable artifact; a missing --label exits 2 with a message naming the flag + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "apr automl fit is reachable from the CLI" + +falsification_tests: +- id: FALSIFY-CRUX-O-01-001 + rule: "fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline" + prediction: "on the iris fixture (crates/aprender-core/src/datasets/iris.csv) accuracy >= 0.90 AND the majority-class baseline is asserted at 0.333 in the same test" + test: >- + LIVE-PENDING - fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline. No test surface exists today because the capability is unimplemented: One-call tabular AutoML: fit(label) -> predict on a CSV (aprender#3371, CRUX-O-01). PROMOTE by authoring a test named fit_beats_majority_baseline_on_iris in module `automl/predictor/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline' is violated — the autogluon parity claim for CRUX-O-01 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-01-002 + rule: "the same call works for a regression label without a problem_type argument" + prediction: "a numeric label column yields a regressor whose R^2 on a held-out split exceeds 0.5 AND the mean-predictor baseline is asserted at ~0.0" + test: >- + LIVE-PENDING - the same call works for a regression label without a problem_type argument. No test surface exists today because the capability is unimplemented: One-call tabular AutoML: fit(label) -> predict on a CSV (aprender#3371, CRUX-O-01). PROMOTE by authoring a test named fit_infers_regression_from_numeric_label in module `automl/predictor/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the same call works for a regression label without a problem_type argument' is violated — the autogluon parity claim for CRUX-O-01 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-01-003 + rule: "apr automl fit is reachable from the CLI" + prediction: "`apr automl fit --label class train.csv --out model.apr` exits 0 and writes a loadable artifact; a missing --label exits 2 with a message naming the flag" + test: >- + LIVE-PENDING - apr automl fit is reachable from the CLI. No test surface exists today because the capability is unimplemented: One-call tabular AutoML: fit(label) -> predict on a CSV (aprender#3371, CRUX-O-01). PROMOTE by authoring a test named cli_automl_fit_roundtrip in module `commands/automl_tests` of crate `apr-cli`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'apr automl fit is reachable from the CLI' is violated — the autogluon parity claim for CRUX-O-01 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline" +- type: invariant + property: "the same call works for a regression label without a problem_type argument" +- type: invariant + property: "apr automl fit is reachable from the CLI" + +kani_harnesses: +- id: KH-CRUX-O-01-001 + obligation: fit_on_a_labelled_csv_returns_a_predictor_whose_predictions_ + property: fit_on_a_labelled_csv_returns_a_predictor_whose_predictions__holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-01-002 + obligation: the_same_call_works_for_a_regression_label_without_a_problem + property: the_same_call_works_for_a_regression_label_without_a_problem_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-01-003 + obligation: apr_automl_fit_is_reachable_from_the_cli + property: apr_automl_fit_is_reachable_from_the_cli_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-02-v1.yaml b/contracts/crux-O-02-v1.yaml new file mode 100644 index 0000000000..efe32eda60 --- /dev/null +++ b/contracts/crux-O-02-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-02 — Problem-type inference: binary / multiclass / regression / quantile from the label column +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3372, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-02 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3372 + description: > + Problem-type inference: binary / multiclass / regression / quantile from the label column. Competitor verb: predictor.problem_type (inferred in fit unless problem_type= given). Gap measured in aprender at eb262f8eb: No function infers a task from a label column. Estimators are chosen by type name (LogisticRegression vs LinearRegression); DataFrame in crates/aprender-core/src/data/mod.rs carries ColumnStats but no label-kind classifier. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3372' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + two_unique_label_values_infer_binary__3___n_small_cardinalit: + formula: | + a table of (label column, expected kind) fixtures including the ambiguous cases {0,1} as int, {0.0,1.0} as float, and 30 unique floats over 1000 rows all classify as documented + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression" + the_override_wins_and_a_contradictory_override_is_rejected: + formula: | + problem_type=regression on a string label column is an Err naming the column, not a silent cast + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the override wins and a contradictory override is rejected" + +falsification_tests: +- id: FALSIFY-CRUX-O-02-001 + rule: "two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression" + prediction: "a table of (label column, expected kind) fixtures including the ambiguous cases {0,1} as int, {0.0,1.0} as float, and 30 unique floats over 1000 rows all classify as documented" + test: >- + LIVE-PENDING - two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression. No test surface exists today because the capability is unimplemented: Problem-type inference: binary / multiclass / regression / quantile from the label column (aprender#3372, CRUX-O-02). PROMOTE by authoring a test named label_kind_table in module `automl/problem_type/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression' is violated — the autogluon parity claim for CRUX-O-02 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-02-002 + rule: "the override wins and a contradictory override is rejected" + prediction: "problem_type=regression on a string label column is an Err naming the column, not a silent cast" + test: >- + LIVE-PENDING - the override wins and a contradictory override is rejected. No test surface exists today because the capability is unimplemented: Problem-type inference: binary / multiclass / regression / quantile from the label column (aprender#3372, CRUX-O-02). PROMOTE by authoring a test named override_contradiction_is_error in module `automl/problem_type/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the override wins and a contradictory override is rejected' is violated — the autogluon parity claim for CRUX-O-02 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression" +- type: invariant + property: "the override wins and a contradictory override is rejected" + +kani_harnesses: +- id: KH-CRUX-O-02-001 + obligation: two_unique_label_values_infer_binary__3___n_small_cardinalit + property: two_unique_label_values_infer_binary__3___n_small_cardinalit_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-02-002 + obligation: the_override_wins_and_a_contradictory_override_is_rejected + property: the_override_wins_and_a_contradictory_override_is_rejected_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-03-v1.yaml b/contracts/crux-O-03-v1.yaml new file mode 100644 index 0000000000..13dc0c8df0 --- /dev/null +++ b/contracts/crux-O-03-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-03 — Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3373, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-03 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3373 + description: > + Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget. Competitor verb: fit(..., presets="best_quality") # tabular/src/autogluon/tabular/configs/presets_configs.py. Gap measured in aprender at eb262f8eb: No preset vocabulary. AutoTuner takes a SearchSpace the caller hand-builds (crates/aprender-core/src/automl/params.rs); there is no named bundle of {models, bagging, stacking, time_limit}. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3373' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + every_preset_name_resolves_to_a_portfolio_and_unknown_names_: + formula: | + the five quality presets each yield a non-empty ordered model list; `presets="bestest"` is an Err whose message contains all five valid names + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "every preset name resolves to a portfolio and unknown names are rejected with the valid list" + presets_are_ordered__a_higher_preset_never_fits_fewer_model_: + formula: | + for medium < good < high < best the family count is monotone non-decreasing, asserted pairwise + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "presets are ordered: a higher preset never fits FEWER model families than the one below it" + +falsification_tests: +- id: FALSIFY-CRUX-O-03-001 + rule: "every preset name resolves to a portfolio and unknown names are rejected with the valid list" + prediction: "the five quality presets each yield a non-empty ordered model list; `presets=\"bestest\"` is an Err whose message contains all five valid names" + test: >- + LIVE-PENDING - every preset name resolves to a portfolio and unknown names are rejected with the valid list. No test surface exists today because the capability is unimplemented: Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget (aprender#3373, CRUX-O-03). PROMOTE by authoring a test named preset_table_resolves in module `automl/presets/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'every preset name resolves to a portfolio and unknown names are rejected with the valid list' is violated — the autogluon parity claim for CRUX-O-03 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-03-002 + rule: "presets are ordered: a higher preset never fits FEWER model families than the one below it" + prediction: "for medium < good < high < best the family count is monotone non-decreasing, asserted pairwise" + test: >- + LIVE-PENDING - presets are ordered: a higher preset never fits FEWER model families than the one below it. No test surface exists today because the capability is unimplemented: Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget (aprender#3373, CRUX-O-03). PROMOTE by authoring a test named preset_monotone_families in module `automl/presets/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'presets are ordered: a higher preset never fits FEWER model families than the one below it' is violated — the autogluon parity claim for CRUX-O-03 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "every preset name resolves to a portfolio and unknown names are rejected with the valid list" +- type: invariant + property: "presets are ordered: a higher preset never fits FEWER model families than the one below it" + +kani_harnesses: +- id: KH-CRUX-O-03-001 + obligation: every_preset_name_resolves_to_a_portfolio_and_unknown_names_ + property: every_preset_name_resolves_to_a_portfolio_and_unknown_names__holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-03-002 + obligation: presets_are_ordered__a_higher_preset_never_fits_fewer_model_ + property: presets_are_ordered__a_higher_preset_never_fits_fewer_model__holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-04-v1.yaml b/contracts/crux-O-04-v1.yaml new file mode 100644 index 0000000000..4b28ca9ccc --- /dev/null +++ b/contracts/crux-O-04-v1.yaml @@ -0,0 +1,94 @@ +# CRUX-O-04 — Leaderboard: per-model validation/test score, fit time, predict time and stack level +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3374, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-04 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3374 + description: > + Leaderboard: per-model validation/test score, fit time, predict time and stack level. Competitor verb: predictor.leaderboard(test_data, extra_info=True). Gap measured in aprender at eb262f8eb: No leaderboard type. GridSearchCVResult in model_selection/ ranks parameter settings of one estimator; TuneResult in automl/tuner.rs is a single best trial. Nothing tabulates several fitted models with timings. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3374' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_leaderboard_is_sorted_by_validation_score_descending_and: + formula: | + columns == [model, score_val, score_test?, pred_time_val, fit_time, stack_level, fit_order] and score_val is non-increasing row to row + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the leaderboard is sorted by validation score descending and its columns are fixed" + timings_are_measured__not_defaulted: + formula: | + every fit_time and pred_time_val is > 0 after a real fit; a leaderboard built with no fit has zero rows, not zero timings + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "timings are measured, not defaulted" + apr_automl_leaderboard_prints_the_same_table_from_a_saved_ar: + formula: | + `apr automl leaderboard model.apr --json` emits the rows byte-equal to the in-process leaderboard + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "apr automl leaderboard prints the same table from a saved artifact" + +falsification_tests: +- id: FALSIFY-CRUX-O-04-001 + rule: "the leaderboard is sorted by validation score descending and its columns are fixed" + prediction: "columns == [model, score_val, score_test?, pred_time_val, fit_time, stack_level, fit_order] and score_val is non-increasing row to row" + test: >- + LIVE-PENDING - the leaderboard is sorted by validation score descending and its columns are fixed. No test surface exists today because the capability is unimplemented: Leaderboard: per-model validation/test score, fit time, predict time and stack level (aprender#3374, CRUX-O-04). PROMOTE by authoring a test named leaderboard_sorted_and_typed in module `automl/leaderboard/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the leaderboard is sorted by validation score descending and its columns are fixed' is violated — the autogluon parity claim for CRUX-O-04 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-04-002 + rule: "timings are measured, not defaulted" + prediction: "every fit_time and pred_time_val is > 0 after a real fit; a leaderboard built with no fit has zero rows, not zero timings" + test: >- + LIVE-PENDING - timings are measured, not defaulted. No test surface exists today because the capability is unimplemented: Leaderboard: per-model validation/test score, fit time, predict time and stack level (aprender#3374, CRUX-O-04). PROMOTE by authoring a test named leaderboard_timings_are_measured in module `automl/leaderboard/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'timings are measured, not defaulted' is violated — the autogluon parity claim for CRUX-O-04 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-04-003 + rule: "apr automl leaderboard prints the same table from a saved artifact" + prediction: "`apr automl leaderboard model.apr --json` emits the rows byte-equal to the in-process leaderboard" + test: >- + LIVE-PENDING - apr automl leaderboard prints the same table from a saved artifact. No test surface exists today because the capability is unimplemented: Leaderboard: per-model validation/test score, fit time, predict time and stack level (aprender#3374, CRUX-O-04). PROMOTE by authoring a test named cli_leaderboard_matches_library in module `commands/automl_tests` of crate `apr-cli`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'apr automl leaderboard prints the same table from a saved artifact' is violated — the autogluon parity claim for CRUX-O-04 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the leaderboard is sorted by validation score descending and its columns are fixed" +- type: invariant + property: "timings are measured, not defaulted" +- type: invariant + property: "apr automl leaderboard prints the same table from a saved artifact" + +kani_harnesses: +- id: KH-CRUX-O-04-001 + obligation: the_leaderboard_is_sorted_by_validation_score_descending_and + property: the_leaderboard_is_sorted_by_validation_score_descending_and_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-04-002 + obligation: timings_are_measured__not_defaulted + property: timings_are_measured__not_defaulted_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-04-003 + obligation: apr_automl_leaderboard_prints_the_same_table_from_a_saved_ar + property: apr_automl_leaderboard_prints_the_same_table_from_a_saved_ar_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-05-v1.yaml b/contracts/crux-O-05-v1.yaml new file mode 100644 index 0000000000..e490ee142c --- /dev/null +++ b/contracts/crux-O-05-v1.yaml @@ -0,0 +1,94 @@ +# CRUX-O-05 — Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3375, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-05 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: partial + github_issue: 3375 + description: > + Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate). Competitor verb: AutoMLPipelineFeatureGenerator # features/src/autogluon/features/generators/auto_ml_pipeline.py. Gap measured in aprender at eb262f8eb: Encoders exist (LabelEncoder, OneHotEncoder, OrdinalEncoder, StandardScaler, PolynomialFeatures in crates/aprender-core/src/preprocessing/) but every one is applied by hand to a column the caller already typed. There is no pass that reads a raw DataFrame, infers each column's kind, and emits a fitted transform. Datetime expansion and text n-gram features do not exist. TfidfVectorizer exists in text/ but is not wired to a tabular pipeline. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3375' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + column_kinds_are_inferred_from_raw_values: + formula: | + a fixture CSV with int, float, low-cardinality string, high-cardinality string, ISO datetime and free-text columns is typed as {numeric, numeric, categorical, text, datetime, text} exactly + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "column kinds are inferred from raw values" + constant_and_duplicate_columns_are_dropped_and_the_drop_is_r: + formula: | + a column with one unique value and an exact duplicate of another column are both absent from the transformed output AND named in the fit report + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "constant and duplicate columns are dropped and the drop is reported" + the_fitted_pipeline_is_deterministic_under_transform: + formula: | + transform(train) then transform(train) are byte-identical and transform(test) never sees a category unseen at fit as anything but the reserved unknown code + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the fitted pipeline is deterministic under transform" + +falsification_tests: +- id: FALSIFY-CRUX-O-05-001 + rule: "column kinds are inferred from raw values" + prediction: "a fixture CSV with int, float, low-cardinality string, high-cardinality string, ISO datetime and free-text columns is typed as {numeric, numeric, categorical, text, datetime, text} exactly" + test: >- + LIVE-PENDING - column kinds are inferred from raw values. No test surface exists today because the capability is partial: Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) (aprender#3375, CRUX-O-05). PROMOTE by authoring a test named infer_feature_kinds_fixture in module `automl/features/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'column kinds are inferred from raw values' is violated — the autogluon parity claim for CRUX-O-05 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-05-002 + rule: "constant and duplicate columns are dropped and the drop is reported" + prediction: "a column with one unique value and an exact duplicate of another column are both absent from the transformed output AND named in the fit report" + test: >- + LIVE-PENDING - constant and duplicate columns are dropped and the drop is reported. No test surface exists today because the capability is partial: Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) (aprender#3375, CRUX-O-05). PROMOTE by authoring a test named drop_unique_and_duplicate_reported in module `automl/features/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'constant and duplicate columns are dropped and the drop is reported' is violated — the autogluon parity claim for CRUX-O-05 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-05-003 + rule: "the fitted pipeline is deterministic under transform" + prediction: "transform(train) then transform(train) are byte-identical and transform(test) never sees a category unseen at fit as anything but the reserved unknown code" + test: >- + LIVE-PENDING - the fitted pipeline is deterministic under transform. No test surface exists today because the capability is partial: Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) (aprender#3375, CRUX-O-05). PROMOTE by authoring a test named pipeline_transform_deterministic in module `automl/features/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the fitted pipeline is deterministic under transform' is violated — the autogluon parity claim for CRUX-O-05 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "column kinds are inferred from raw values" +- type: invariant + property: "constant and duplicate columns are dropped and the drop is reported" +- type: invariant + property: "the fitted pipeline is deterministic under transform" + +kani_harnesses: +- id: KH-CRUX-O-05-001 + obligation: column_kinds_are_inferred_from_raw_values + property: column_kinds_are_inferred_from_raw_values_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-05-002 + obligation: constant_and_duplicate_columns_are_dropped_and_the_drop_is_r + property: constant_and_duplicate_columns_are_dropped_and_the_drop_is_r_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-05-003 + obligation: the_fitted_pipeline_is_deterministic_under_transform + property: the_fitted_pipeline_is_deterministic_under_transform_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-06-v1.yaml b/contracts/crux-O-06-v1.yaml new file mode 100644 index 0000000000..750fbebeeb --- /dev/null +++ b/contracts/crux-O-06-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-06 — K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3376, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-06 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3376 + description: > + K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof). Competitor verb: fit(..., num_bag_folds=8); predictor.predict_proba_oof(). Gap measured in aprender at eb262f8eb: KFold and StratifiedKFold exist (model_selection/) and cross_validate scores them, but no wrapper trains one child per fold, keeps all children, averages them at predict time and exposes the out-of-fold matrix. RandomForest bags trees internally and is not reusable for other estimators. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3376' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + oof_predictions_cover_every_training_row_exactly_once: + formula: | + for n rows and k folds the OOF matrix has n rows, no NaN, and each row was predicted by the one child that did not see it (asserted through a fold-id trace) + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "OOF predictions cover every training row exactly once" + bagged_prediction_is_the_mean_of_the_children: + formula: | + predict_proba of the bag equals the elementwise mean of the k children's predict_proba within 1e-12 + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "bagged prediction is the mean of the children" + +falsification_tests: +- id: FALSIFY-CRUX-O-06-001 + rule: "OOF predictions cover every training row exactly once" + prediction: "for n rows and k folds the OOF matrix has n rows, no NaN, and each row was predicted by the one child that did not see it (asserted through a fold-id trace)" + test: >- + LIVE-PENDING - OOF predictions cover every training row exactly once. No test surface exists today because the capability is unimplemented: K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) (aprender#3376, CRUX-O-06). PROMOTE by authoring a test named oof_covers_each_row_once in module `automl/bagging/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'OOF predictions cover every training row exactly once' is violated — the autogluon parity claim for CRUX-O-06 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-06-002 + rule: "bagged prediction is the mean of the children" + prediction: "predict_proba of the bag equals the elementwise mean of the k children's predict_proba within 1e-12" + test: >- + LIVE-PENDING - bagged prediction is the mean of the children. No test surface exists today because the capability is unimplemented: K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) (aprender#3376, CRUX-O-06). PROMOTE by authoring a test named bag_predict_is_child_mean in module `automl/bagging/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'bagged prediction is the mean of the children' is violated — the autogluon parity claim for CRUX-O-06 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "OOF predictions cover every training row exactly once" +- type: invariant + property: "bagged prediction is the mean of the children" + +kani_harnesses: +- id: KH-CRUX-O-06-001 + obligation: oof_predictions_cover_every_training_row_exactly_once + property: oof_predictions_cover_every_training_row_exactly_once_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-06-002 + obligation: bagged_prediction_is_the_mean_of_the_children + property: bagged_prediction_is_the_mean_of_the_children_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-07-v1.yaml b/contracts/crux-O-07-v1.yaml new file mode 100644 index 0000000000..e76ba2a33c --- /dev/null +++ b/contracts/crux-O-07-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-07 — Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3377, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-07 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3377 + description: > + Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking). Competitor verb: fit(..., num_stack_levels=1, dynamic_stacking="auto"). Gap measured in aprender at eb262f8eb: No stacking. crates/aprender-core/src/stack/ is a deployment-health module (StackHealth, InferenceConfig), not a model stacker. ensemble/ holds MixtureOfExperts gating only. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3377' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + level_2_features_are_oof__never_in_sample: + formula: | + a mutation that feeds in-sample level-1 predictions to level 2 is detected by the leak test: the L2 holdout score on a pure-noise label rises above chance (asserted RED) while the OOF path stays at chance + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "level-2 features are OOF, never in-sample" + dynamic_stacking_falls_back_when_stacking_hurts: + formula: | + on a fixture where L2 holdout score < L1 holdout score, the final model is the L1 ensemble and the decision is recorded in the fit summary + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "dynamic stacking falls back when stacking hurts" + +falsification_tests: +- id: FALSIFY-CRUX-O-07-001 + rule: "level-2 features are OOF, never in-sample" + prediction: "a mutation that feeds in-sample level-1 predictions to level 2 is detected by the leak test: the L2 holdout score on a pure-noise label rises above chance (asserted RED) while the OOF path stays at chance" + test: >- + LIVE-PENDING - level-2 features are OOF, never in-sample. No test surface exists today because the capability is unimplemented: Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) (aprender#3377, CRUX-O-07). PROMOTE by authoring a test named stack_uses_oof_not_insample in module `automl/stacking/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'level-2 features are OOF, never in-sample' is violated — the autogluon parity claim for CRUX-O-07 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-07-002 + rule: "dynamic stacking falls back when stacking hurts" + prediction: "on a fixture where L2 holdout score < L1 holdout score, the final model is the L1 ensemble and the decision is recorded in the fit summary" + test: >- + LIVE-PENDING - dynamic stacking falls back when stacking hurts. No test surface exists today because the capability is unimplemented: Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) (aprender#3377, CRUX-O-07). PROMOTE by authoring a test named dynamic_stacking_fallback_recorded in module `automl/stacking/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'dynamic stacking falls back when stacking hurts' is violated — the autogluon parity claim for CRUX-O-07 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "level-2 features are OOF, never in-sample" +- type: invariant + property: "dynamic stacking falls back when stacking hurts" + +kani_harnesses: +- id: KH-CRUX-O-07-001 + obligation: level_2_features_are_oof__never_in_sample + property: level_2_features_are_oof__never_in_sample_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-07-002 + obligation: dynamic_stacking_falls_back_when_stacking_hurts + property: dynamic_stacking_falls_back_when_stacking_hurts_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-08-v1.yaml b/contracts/crux-O-08-v1.yaml new file mode 100644 index 0000000000..cc1700640e --- /dev/null +++ b/contracts/crux-O-08-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-08 — Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3378, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-08 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3378 + description: > + Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection). Competitor verb: fit_weighted_ensemble=True (default) # core/src/autogluon/core/models/greedy_ensemble/ensemble_selection.py. Gap measured in aprender at eb262f8eb: No ensemble-selection algorithm. MixtureOfExperts learns a gating network (ensemble/moe.rs); nothing performs the forward greedy selection with replacement over base-model validation predictions that yields non-negative weights summing to 1. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3378' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + weights_are_a_probability_vector_and_the_ensemble_never_scor: + formula: | + sum(w)=1, all w>=0, and validation metric(ensemble) >= max over members within 1e-9, on 3 fixtures + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "weights are a probability vector and the ensemble never scores below its best member" + selection_is_greedy_with_replacement_and_reproducible: + formula: | + with ensemble_size=25 the weight of a member equals its selection count / 25; two runs on the same inputs produce identical weights + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "selection is greedy with replacement and reproducible" + +falsification_tests: +- id: FALSIFY-CRUX-O-08-001 + rule: "weights are a probability vector and the ensemble never scores below its best member" + prediction: "sum(w)=1, all w>=0, and validation metric(ensemble) >= max over members within 1e-9, on 3 fixtures" + test: >- + LIVE-PENDING - weights are a probability vector and the ensemble never scores below its best member. No test surface exists today because the capability is unimplemented: Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) (aprender#3378, CRUX-O-08). PROMOTE by authoring a test named weights_simplex_and_no_regression in module `automl/ensemble_selection/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'weights are a probability vector and the ensemble never scores below its best member' is violated — the autogluon parity claim for CRUX-O-08 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-08-002 + rule: "selection is greedy with replacement and reproducible" + prediction: "with ensemble_size=25 the weight of a member equals its selection count / 25; two runs on the same inputs produce identical weights" + test: >- + LIVE-PENDING - selection is greedy with replacement and reproducible. No test surface exists today because the capability is unimplemented: Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) (aprender#3378, CRUX-O-08). PROMOTE by authoring a test named greedy_with_replacement_counts in module `automl/ensemble_selection/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'selection is greedy with replacement and reproducible' is violated — the autogluon parity claim for CRUX-O-08 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "weights are a probability vector and the ensemble never scores below its best member" +- type: invariant + property: "selection is greedy with replacement and reproducible" + +kani_harnesses: +- id: KH-CRUX-O-08-001 + obligation: weights_are_a_probability_vector_and_the_ensemble_never_scor + property: weights_are_a_probability_vector_and_the_ensemble_never_scor_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-08-002 + obligation: selection_is_greedy_with_replacement_and_reproducible + property: selection_is_greedy_with_replacement_and_reproducible_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-09-v1.yaml b/contracts/crux-O-09-v1.yaml new file mode 100644 index 0000000000..5bfadb368f --- /dev/null +++ b/contracts/crux-O-09-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-09 — Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3379, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-09 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: partial + github_issue: 3379 + description: > + Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share. Competitor verb: fit(..., time_limit=3600). Gap measured in aprender at eb262f8eb: TimeBudget and EarlyStopping exist in automl/tuner.rs but budget ONE tuner. There is no allocation of a global limit across an ordered portfolio, no per-model time share, and no 'skip the rest' when the budget is exhausted. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3379' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_wall_clock_of_fit_never_exceeds_time_limit_by_more_than_: + formula: | + with time_limit=5s on a portfolio that would take >60s unconstrained, elapsed <= 5s + 1s AND at least one model was skipped with reason=budget in the summary + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the wall-clock of fit never exceeds time_limit by more than the tolerance" + the_budget_is_redistributed_when_a_model_finishes_early: + formula: | + a model that uses 10% of its share returns the remainder to the pool; the next model's share is asserted larger than the naive equal split + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the budget is redistributed when a model finishes early" + +falsification_tests: +- id: FALSIFY-CRUX-O-09-001 + rule: "the wall-clock of fit never exceeds time_limit by more than the tolerance" + prediction: "with time_limit=5s on a portfolio that would take >60s unconstrained, elapsed <= 5s + 1s AND at least one model was skipped with reason=budget in the summary" + test: >- + LIVE-PENDING - the wall-clock of fit never exceeds time_limit by more than the tolerance. No test surface exists today because the capability is partial: Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share (aprender#3379, CRUX-O-09). PROMOTE by authoring a test named portfolio_respects_global_budget in module `automl/budget/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the wall-clock of fit never exceeds time_limit by more than the tolerance' is violated — the autogluon parity claim for CRUX-O-09 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-09-002 + rule: "the budget is redistributed when a model finishes early" + prediction: "a model that uses 10% of its share returns the remainder to the pool; the next model's share is asserted larger than the naive equal split" + test: >- + LIVE-PENDING - the budget is redistributed when a model finishes early. No test surface exists today because the capability is partial: Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share (aprender#3379, CRUX-O-09). PROMOTE by authoring a test named unused_share_is_redistributed in module `automl/budget/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the budget is redistributed when a model finishes early' is violated — the autogluon parity claim for CRUX-O-09 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the wall-clock of fit never exceeds time_limit by more than the tolerance" +- type: invariant + property: "the budget is redistributed when a model finishes early" + +kani_harnesses: +- id: KH-CRUX-O-09-001 + obligation: the_wall_clock_of_fit_never_exceeds_time_limit_by_more_than_ + property: the_wall_clock_of_fit_never_exceeds_time_limit_by_more_than__holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-09-002 + obligation: the_budget_is_redistributed_when_a_model_finishes_early + property: the_budget_is_redistributed_when_a_model_finishes_early_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-10-v1.yaml b/contracts/crux-O-10-v1.yaml new file mode 100644 index 0000000000..bbabd53a5a --- /dev/null +++ b/contracts/crux-O-10-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-10 — Predictor-level permutation feature importance with p-values and confidence intervals +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3380, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-10 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: partial + github_issue: 3380 + description: > + Predictor-level permutation feature importance with p-values and confidence intervals. Competitor verb: predictor.feature_importance(test_data, num_shuffle_sets=10). Gap measured in aprender at eb262f8eb: PermutationImportance exists in crates/aprender-core/src/interpret/ for a single estimator. Missing: the predictor-level call on raw (pre-pipeline) columns, num_shuffle_sets repeats, stddev / p-value / p99 columns, and subsampling. AutoGluon 1.6 also cut this call's memory 25x (#5645). + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3380' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + importance_is_reported_per_raw_input_column: + formula: | + on a fixture whose datetime column expands to 4 features, the importance table has one row for the datetime column, not four + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "importance is reported per raw input column" + a_pure_noise_column_has_importance_statistically_indistingui: + formula: | + with num_shuffle_sets=10 the noise column's p-value > 0.05 AND the signal column's p-value < 0.01 on the same run + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a pure-noise column has importance statistically indistinguishable from zero" + +falsification_tests: +- id: FALSIFY-CRUX-O-10-001 + rule: "importance is reported per raw input column" + prediction: "on a fixture whose datetime column expands to 4 features, the importance table has one row for the datetime column, not four" + test: >- + LIVE-PENDING - importance is reported per raw input column. No test surface exists today because the capability is partial: Predictor-level permutation feature importance with p-values and confidence intervals (aprender#3380, CRUX-O-10). PROMOTE by authoring a test named importance_on_raw_columns in module `automl/importance/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'importance is reported per raw input column' is violated — the autogluon parity claim for CRUX-O-10 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-10-002 + rule: "a pure-noise column has importance statistically indistinguishable from zero" + prediction: "with num_shuffle_sets=10 the noise column's p-value > 0.05 AND the signal column's p-value < 0.01 on the same run" + test: >- + LIVE-PENDING - a pure-noise column has importance statistically indistinguishable from zero. No test surface exists today because the capability is partial: Predictor-level permutation feature importance with p-values and confidence intervals (aprender#3380, CRUX-O-10). PROMOTE by authoring a test named noise_column_pvalue in module `automl/importance/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a pure-noise column has importance statistically indistinguishable from zero' is violated — the autogluon parity claim for CRUX-O-10 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "importance is reported per raw input column" +- type: invariant + property: "a pure-noise column has importance statistically indistinguishable from zero" + +kani_harnesses: +- id: KH-CRUX-O-10-001 + obligation: importance_is_reported_per_raw_input_column + property: importance_is_reported_per_raw_input_column_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-10-002 + obligation: a_pure_noise_column_has_importance_statistically_indistingui + property: a_pure_noise_column_has_importance_statistically_indistingui_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-11-v1.yaml b/contracts/crux-O-11-v1.yaml new file mode 100644 index 0000000000..2132d3123c --- /dev/null +++ b/contracts/crux-O-11-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-11 — Decision-threshold calibration for binary metrics (calibrate_decision_threshold) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3381, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-11 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: partial + github_issue: 3381 + description: > + Decision-threshold calibration for binary metrics (calibrate_decision_threshold). Competitor verb: fit(..., calibrate_decision_threshold="auto"); predictor.calibrate_decision_threshold(metric="f1"). Gap measured in aprender at eb262f8eb: Probability calibration exists (PlattScaling, IsotonicRegression, TemperatureScaling in calibration.rs) but nothing searches the decision threshold that maximises f1 / balanced_accuracy / mcc on validation data and stores it on the predictor (`decision_threshold`, `set_decision_threshold`). + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3381' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_calibrated_threshold_beats_0_5_on_the_calibration_metric: + formula: | + on an imbalanced fixture (5% positives) f1 at the calibrated threshold > f1 at 0.5 by >= 0.05 absolute + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search" + the_threshold_is_persisted_with_the_model: + formula: | + save then load reproduces predict() bit-identically including the threshold; a mutation that resets the threshold to 0.5 on load turns this RED + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the threshold is persisted with the model" + +falsification_tests: +- id: FALSIFY-CRUX-O-11-001 + rule: "the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search" + prediction: "on an imbalanced fixture (5% positives) f1 at the calibrated threshold > f1 at 0.5 by >= 0.05 absolute" + test: >- + LIVE-PENDING - the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search. No test surface exists today because the capability is partial: Decision-threshold calibration for binary metrics (calibrate_decision_threshold) (aprender#3381, CRUX-O-11). PROMOTE by authoring a test named calibrated_threshold_beats_half in module `automl/threshold/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search' is violated — the autogluon parity claim for CRUX-O-11 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-11-002 + rule: "the threshold is persisted with the model" + prediction: "save then load reproduces predict() bit-identically including the threshold; a mutation that resets the threshold to 0.5 on load turns this RED" + test: >- + LIVE-PENDING - the threshold is persisted with the model. No test surface exists today because the capability is partial: Decision-threshold calibration for binary metrics (calibrate_decision_threshold) (aprender#3381, CRUX-O-11). PROMOTE by authoring a test named threshold_persisted_roundtrip in module `automl/threshold/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the threshold is persisted with the model' is violated — the autogluon parity claim for CRUX-O-11 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search" +- type: invariant + property: "the threshold is persisted with the model" + +kani_harnesses: +- id: KH-CRUX-O-11-001 + obligation: the_calibrated_threshold_beats_0_5_on_the_calibration_metric + property: the_calibrated_threshold_beats_0_5_on_the_calibration_metric_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-11-002 + obligation: the_threshold_is_persisted_with_the_model + property: the_threshold_is_persisted_with_the_model_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-12-v1.yaml b/contracts/crux-O-12-v1.yaml new file mode 100644 index 0000000000..fb7576e04a --- /dev/null +++ b/contracts/crux-O-12-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-12 — refit_full: retrain the selected models on train+validation after model selection +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3382, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-12 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3382 + description: > + refit_full: retrain the selected models on train+validation after model selection. Competitor verb: fit(..., refit_full=True, set_best_to_refit_full=True); predictor.refit_full(). Gap measured in aprender at eb262f8eb: No refit-on-full-data step. cross_validate and grid_search return scores; the model returned is the one fitted on a fold, not on all rows. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3382' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_refit_model_saw_every_row: + formula: | + the refit estimator's training-row count equals n_train + n_val, asserted through the fitted row count it reports + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the refit model saw every row" + refit_keeps_the_selected_hyperparameters_and_drops_the_bag_c: + formula: | + the artifact after refit has one child per selected model and its hyperparameters are byte-equal to the pre-refit winner's + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "refit keeps the selected hyperparameters and drops the bag children" + +falsification_tests: +- id: FALSIFY-CRUX-O-12-001 + rule: "the refit model saw every row" + prediction: "the refit estimator's training-row count equals n_train + n_val, asserted through the fitted row count it reports" + test: >- + LIVE-PENDING - the refit model saw every row. No test surface exists today because the capability is unimplemented: refit_full: retrain the selected models on train+validation after model selection (aprender#3382, CRUX-O-12). PROMOTE by authoring a test named refit_sees_all_rows in module `automl/refit/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the refit model saw every row' is violated — the autogluon parity claim for CRUX-O-12 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-12-002 + rule: "refit keeps the selected hyperparameters and drops the bag children" + prediction: "the artifact after refit has one child per selected model and its hyperparameters are byte-equal to the pre-refit winner's" + test: >- + LIVE-PENDING - refit keeps the selected hyperparameters and drops the bag children. No test surface exists today because the capability is unimplemented: refit_full: retrain the selected models on train+validation after model selection (aprender#3382, CRUX-O-12). PROMOTE by authoring a test named refit_keeps_params_drops_children in module `automl/refit/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'refit keeps the selected hyperparameters and drops the bag children' is violated — the autogluon parity claim for CRUX-O-12 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the refit model saw every row" +- type: invariant + property: "refit keeps the selected hyperparameters and drops the bag children" + +kani_harnesses: +- id: KH-CRUX-O-12-001 + obligation: the_refit_model_saw_every_row + property: the_refit_model_saw_every_row_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-12-002 + obligation: refit_keeps_the_selected_hyperparameters_and_drops_the_bag_c + property: refit_keeps_the_selected_hyperparameters_and_drops_the_bag_c_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-13-v1.yaml b/contracts/crux-O-13-v1.yaml new file mode 100644 index 0000000000..a9ea70615f --- /dev/null +++ b/contracts/crux-O-13-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-13 — Model distillation: compress the ensemble into one fast student +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3383, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-13 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3383 + description: > + Model distillation: compress the ensemble into one fast student. Competitor verb: predictor.distill(time_limit=..., augment_method="spunge"). Gap measured in aprender at eb262f8eb: Distillation exists only for LLMs in the training crate (knowledge distillation of transformers); nothing trains a single tabular student on the teacher ensemble's soft labels with data augmentation. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3383' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_student_is_faster_and_within_tolerance_of_the_teacher: + formula: | + student pred_time < 0.25 x teacher pred_time AND student validation score >= teacher score - 0.02 on a fixture + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the student is faster and within tolerance of the teacher" + augmentation_produces_rows_the_training_set_does_not_contain: + formula: | + with augment_method=spunge the student's training set has > n_train rows and the added rows are not row-equal to any original + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "augmentation produces rows the training set does not contain" + +falsification_tests: +- id: FALSIFY-CRUX-O-13-001 + rule: "the student is faster and within tolerance of the teacher" + prediction: "student pred_time < 0.25 x teacher pred_time AND student validation score >= teacher score - 0.02 on a fixture" + test: >- + LIVE-PENDING - the student is faster and within tolerance of the teacher. No test surface exists today because the capability is unimplemented: Model distillation: compress the ensemble into one fast student (aprender#3383, CRUX-O-13). PROMOTE by authoring a test named student_faster_within_tolerance in module `automl/distill/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the student is faster and within tolerance of the teacher' is violated — the autogluon parity claim for CRUX-O-13 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-13-002 + rule: "augmentation produces rows the training set does not contain" + prediction: "with augment_method=spunge the student's training set has > n_train rows and the added rows are not row-equal to any original" + test: >- + LIVE-PENDING - augmentation produces rows the training set does not contain. No test surface exists today because the capability is unimplemented: Model distillation: compress the ensemble into one fast student (aprender#3383, CRUX-O-13). PROMOTE by authoring a test named augmented_rows_are_novel in module `automl/distill/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'augmentation produces rows the training set does not contain' is violated — the autogluon parity claim for CRUX-O-13 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the student is faster and within tolerance of the teacher" +- type: invariant + property: "augmentation produces rows the training set does not contain" + +kani_harnesses: +- id: KH-CRUX-O-13-001 + obligation: the_student_is_faster_and_within_tolerance_of_the_teacher + property: the_student_is_faster_and_within_tolerance_of_the_teacher_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-13-002 + obligation: augmentation_produces_rows_the_training_set_does_not_contain + property: augmentation_produces_rows_the_training_set_does_not_contain_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-14-v1.yaml b/contracts/crux-O-14-v1.yaml new file mode 100644 index 0000000000..a05ea6978c --- /dev/null +++ b/contracts/crux-O-14-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-14 — Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3384, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-14 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3384 + description: > + Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file. Competitor verb: predictor.clone_for_deployment(path); predictor.persist(). Gap measured in aprender at eb262f8eb: Single estimators serialize to .apr (bundle/, serialization/). No artifact holds a fitted feature pipeline (O-05) plus several models plus ensemble weights plus a decision threshold and loads them as one predictor. Nothing prunes non-selected models from disk. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3384' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + a_deployment_clone_is_smaller_and_predicts_identically: + formula: | + clone_for_deployment artifact bytes < 0.5 x full artifact bytes AND predict() on the clone is bit-identical to predict() on the original for the test fixture + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a deployment clone is smaller and predicts identically" + the_artifact_is_self_describing_and_refuses_a_schema_drift: + formula: | + load() on an artifact whose feature schema disagrees with the input columns is an Err naming the first mismatched column, not a wrong prediction + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the artifact is self-describing and refuses a schema drift" + +falsification_tests: +- id: FALSIFY-CRUX-O-14-001 + rule: "a deployment clone is smaller and predicts identically" + prediction: "clone_for_deployment artifact bytes < 0.5 x full artifact bytes AND predict() on the clone is bit-identical to predict() on the original for the test fixture" + test: >- + LIVE-PENDING - a deployment clone is smaller and predicts identically. No test surface exists today because the capability is unimplemented: Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file (aprender#3384, CRUX-O-14). PROMOTE by authoring a test named deployment_clone_smaller_identical in module `automl/artifact/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a deployment clone is smaller and predicts identically' is violated — the autogluon parity claim for CRUX-O-14 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-14-002 + rule: "the artifact is self-describing and refuses a schema drift" + prediction: "load() on an artifact whose feature schema disagrees with the input columns is an Err naming the first mismatched column, not a wrong prediction" + test: >- + LIVE-PENDING - the artifact is self-describing and refuses a schema drift. No test surface exists today because the capability is unimplemented: Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file (aprender#3384, CRUX-O-14). PROMOTE by authoring a test named artifact_refuses_schema_drift in module `automl/artifact/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the artifact is self-describing and refuses a schema drift' is violated — the autogluon parity claim for CRUX-O-14 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "a deployment clone is smaller and predicts identically" +- type: invariant + property: "the artifact is self-describing and refuses a schema drift" + +kani_harnesses: +- id: KH-CRUX-O-14-001 + obligation: a_deployment_clone_is_smaller_and_predicts_identically + property: a_deployment_clone_is_smaller_and_predicts_identically_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-14-002 + obligation: the_artifact_is_self_describing_and_refuses_a_schema_drift + property: the_artifact_is_self_describing_and_refuses_a_schema_drift_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-15-v1.yaml b/contracts/crux-O-15-v1.yaml new file mode 100644 index 0000000000..b70a81b5d8 --- /dev/null +++ b/contracts/crux-O-15-v1.yaml @@ -0,0 +1,56 @@ +# CRUX-O-15 — Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3385, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-15 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3385 + description: > + Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size). Competitor verb: fit(..., infer_limit=0.001, infer_limit_batch_size=10000). Gap measured in aprender at eb262f8eb: No model or ensemble is ever excluded for being slow at predict time; leaderboard timings (O-04) do not exist to compare against. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3385' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + the_selected_ensemble_respects_the_per_row_latency_limit: + formula: | + with infer_limit=L the measured pred_time per row of the final model <= L AND at least one faster-but-worse model was preferred over a slower-but-better one (asserted on a fixture built to force the trade) + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the selected ensemble respects the per-row latency limit" + +falsification_tests: +- id: FALSIFY-CRUX-O-15-001 + rule: "the selected ensemble respects the per-row latency limit" + prediction: "with infer_limit=L the measured pred_time per row of the final model <= L AND at least one faster-but-worse model was preferred over a slower-but-better one (asserted on a fixture built to force the trade)" + test: >- + LIVE-PENDING - the selected ensemble respects the per-row latency limit. No test surface exists today because the capability is unimplemented: Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size) (aprender#3385, CRUX-O-15). PROMOTE by authoring a test named infer_limit_prunes_slow_models in module `automl/budget/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the selected ensemble respects the per-row latency limit' is violated — the autogluon parity claim for CRUX-O-15 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "the selected ensemble respects the per-row latency limit" + +kani_harnesses: +- id: KH-CRUX-O-15-001 + obligation: the_selected_ensemble_respects_the_per_row_latency_limit + property: the_selected_ensemble_respects_the_per_row_latency_limit_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-16-v1.yaml b/contracts/crux-O-16-v1.yaml new file mode 100644 index 0000000000..56c001d51a --- /dev/null +++ b/contracts/crux-O-16-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-16 — Fit diagnostics: fit_summary, model_failures and learning curves +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3386, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-16 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3386 + description: > + Fit diagnostics: fit_summary, model_failures and learning curves. Competitor verb: predictor.fit_summary(); predictor.model_failures(); fit(..., learning_curves=True). Gap measured in aprender at eb262f8eb: ProgressCallback in automl/tuner.rs streams trial results; there is no post-fit summary object listing models trained, models failed with their error, per-model hyperparameters and per-iteration validation curves. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3386' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + a_model_that_raises_during_fit_is_recorded__not_swallowed_an: + formula: | + a portfolio containing a deliberately failing model finishes; model_failures() has exactly one row naming the model and the error string; the leaderboard omits it + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a model that raises during fit is recorded, not swallowed and not fatal" + learning_curves_have_one_point_per_boosting_epoch_iteration: + formula: | + for a GBM with n_estimators=50 and learning_curves=true the curve has 50 validation points, monotone non-increasing after early-stopping's best iteration is asserted absent + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "learning curves have one point per boosting/epoch iteration" + +falsification_tests: +- id: FALSIFY-CRUX-O-16-001 + rule: "a model that raises during fit is recorded, not swallowed and not fatal" + prediction: "a portfolio containing a deliberately failing model finishes; model_failures() has exactly one row naming the model and the error string; the leaderboard omits it" + test: >- + LIVE-PENDING - a model that raises during fit is recorded, not swallowed and not fatal. No test surface exists today because the capability is unimplemented: Fit diagnostics: fit_summary, model_failures and learning curves (aprender#3386, CRUX-O-16). PROMOTE by authoring a test named failing_model_is_recorded_not_fatal in module `automl/summary/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a model that raises during fit is recorded, not swallowed and not fatal' is violated — the autogluon parity claim for CRUX-O-16 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-16-002 + rule: "learning curves have one point per boosting/epoch iteration" + prediction: "for a GBM with n_estimators=50 and learning_curves=true the curve has 50 validation points, monotone non-increasing after early-stopping's best iteration is asserted absent" + test: >- + LIVE-PENDING - learning curves have one point per boosting/epoch iteration. No test surface exists today because the capability is unimplemented: Fit diagnostics: fit_summary, model_failures and learning curves (aprender#3386, CRUX-O-16). PROMOTE by authoring a test named learning_curve_length_matches_iterations in module `automl/summary/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'learning curves have one point per boosting/epoch iteration' is violated — the autogluon parity claim for CRUX-O-16 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "a model that raises during fit is recorded, not swallowed and not fatal" +- type: invariant + property: "learning curves have one point per boosting/epoch iteration" + +kani_harnesses: +- id: KH-CRUX-O-16-001 + obligation: a_model_that_raises_during_fit_is_recorded__not_swallowed_an + property: a_model_that_raises_during_fit_is_recorded__not_swallowed_an_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-16-002 + obligation: learning_curves_have_one_point_per_boosting_epoch_iteration + property: learning_curves_have_one_point_per_boosting_epoch_iteration_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-17-v1.yaml b/contracts/crux-O-17-v1.yaml new file mode 100644 index 0000000000..2c46e58e04 --- /dev/null +++ b/contracts/crux-O-17-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-17 — Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3387, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-17 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3387 + description: > + Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class). Competitor verb: hyperparameters={"TABPFNV2": {}, "TABICL": {}, "MITRA": {}} # in the extreme_quality preset. Gap measured in aprender at eb262f8eb: No tabular in-context learner. nn/ and the inference crates run causal LMs; nothing consumes (X_train, y_train, X_test) as one context and predicts without gradient steps. AutoGluon 1.4-1.6 added TabPFNv2/2.5/2.6/3, TabICL/v2, Mitra, TabDPT, Nori and made them the extreme preset. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3387' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + zero_gradient_prediction_matches_a_pinned_reference_within_t: + formula: | + loading a pinned small checkpoint via apr pull and predicting a 100-row fixture matches the committed reference probabilities within 1e-4 + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "zero-gradient prediction matches a pinned reference within tolerance" + the_context_limit_is_enforced__not_silently_truncated: + formula: | + a context of rows > the model's documented max is an Err naming the limit; a mutation that truncates instead turns this RED + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the context limit is enforced, not silently truncated" + +falsification_tests: +- id: FALSIFY-CRUX-O-17-001 + rule: "zero-gradient prediction matches a pinned reference within tolerance" + prediction: "loading a pinned small checkpoint via apr pull and predicting a 100-row fixture matches the committed reference probabilities within 1e-4" + test: >- + LIVE-PENDING - zero-gradient prediction matches a pinned reference within tolerance. No test surface exists today because the capability is unimplemented: Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) (aprender#3387, CRUX-O-17). PROMOTE by authoring a test named icl_matches_pinned_reference in module `models/tabular_fm/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'zero-gradient prediction matches a pinned reference within tolerance' is violated — the autogluon parity claim for CRUX-O-17 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-17-002 + rule: "the context limit is enforced, not silently truncated" + prediction: "a context of rows > the model's documented max is an Err naming the limit; a mutation that truncates instead turns this RED" + test: >- + LIVE-PENDING - the context limit is enforced, not silently truncated. No test surface exists today because the capability is unimplemented: Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) (aprender#3387, CRUX-O-17). PROMOTE by authoring a test named context_limit_is_an_error in module `models/tabular_fm/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the context limit is enforced, not silently truncated' is violated — the autogluon parity claim for CRUX-O-17 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "zero-gradient prediction matches a pinned reference within tolerance" +- type: invariant + property: "the context limit is enforced, not silently truncated" + +kani_harnesses: +- id: KH-CRUX-O-17-001 + obligation: zero_gradient_prediction_matches_a_pinned_reference_within_t + property: zero_gradient_prediction_matches_a_pinned_reference_within_t_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-17-002 + obligation: the_context_limit_is_enforced__not_silently_truncated + property: the_context_limit_is_enforced__not_silently_truncated_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-18-v1.yaml b/contracts/crux-O-18-v1.yaml new file mode 100644 index 0000000000..6ac97f5bb8 --- /dev/null +++ b/contracts/crux-O-18-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-18 — Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3388, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-18 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3388 + description: > + Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit. Competitor verb: fit(..., memory_limit="auto") # ag.max_memory_usage_ratio; 1.6 calibrated CPU/GPU estimates. Gap measured in aprender at eb262f8eb: No estimator reports an expected peak memory before fitting and nothing checks a limit. AutoGluon 1.6 spent 4 PRs on this (#5757, #5768, #5791, GPU budgeting for parallel folds). + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3388' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + a_model_whose_estimate_exceeds_the_limit_is_skipped_with_rea: + formula: | + with memory_limit=64MiB on a fixture where the estimate for the largest model is > 64MiB, that model is absent from the leaderboard and present in the skip list with its estimate + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a model whose estimate exceeds the limit is skipped with reason=memory" + the_estimate_is_not_a_constant: + formula: | + the estimate for a 10x larger fixture is asserted larger than for the base fixture; a mutation returning a constant turns this RED + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "the estimate is not a constant" + +falsification_tests: +- id: FALSIFY-CRUX-O-18-001 + rule: "a model whose estimate exceeds the limit is skipped with reason=memory" + prediction: "with memory_limit=64MiB on a fixture where the estimate for the largest model is > 64MiB, that model is absent from the leaderboard and present in the skip list with its estimate" + test: >- + LIVE-PENDING - a model whose estimate exceeds the limit is skipped with reason=memory. No test surface exists today because the capability is unimplemented: Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit (aprender#3388, CRUX-O-18). PROMOTE by authoring a test named memory_limit_skips_and_records in module `automl/budget/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a model whose estimate exceeds the limit is skipped with reason=memory' is violated — the autogluon parity claim for CRUX-O-18 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-18-002 + rule: "the estimate is not a constant" + prediction: "the estimate for a 10x larger fixture is asserted larger than for the base fixture; a mutation returning a constant turns this RED" + test: >- + LIVE-PENDING - the estimate is not a constant. No test surface exists today because the capability is unimplemented: Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit (aprender#3388, CRUX-O-18). PROMOTE by authoring a test named estimate_scales_with_data in module `automl/budget/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'the estimate is not a constant' is violated — the autogluon parity claim for CRUX-O-18 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "a model whose estimate exceeds the limit is skipped with reason=memory" +- type: invariant + property: "the estimate is not a constant" + +kani_harnesses: +- id: KH-CRUX-O-18-001 + obligation: a_model_whose_estimate_exceeds_the_limit_is_skipped_with_rea + property: a_model_whose_estimate_exceeds_the_limit_is_skipped_with_rea_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-18-002 + obligation: the_estimate_is_not_a_constant + property: the_estimate_is_not_a_constant_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-19-v1.yaml b/contracts/crux-O-19-v1.yaml new file mode 100644 index 0000000000..4e9ff29c33 --- /dev/null +++ b/contracts/crux-O-19-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-19 — Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3389, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-19 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3389 + description: > + Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq. Competitor verb: TimeSeriesPredictor(prediction_length=48, freq="h").fit(TimeSeriesDataFrame). Gap measured in aprender at eb262f8eb: crates/aprender-core/src/time_series/mod.rs is one struct, ARIMA, on one f32 series (fit/forecast/order). There is no panel container keyed by item and timestamp, no frequency, no horizon-first API and no per-item forecast. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3389' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + a_panel_of_n_items_forecasts_n_x_prediction_length_rows: + formula: | + on a fixture of 5 items with 200 hourly points each and prediction_length=24 the output has exactly 120 rows, each (item_id, timestamp) unique, timestamps continuing each item's last stamp at freq=h + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a panel of N items forecasts N x prediction_length rows" + irregular_timestamps_are_rejected_or_regularised__never_sile: + formula: | + an item with a missing hour is an Err naming the item and gap unless fill=forward is passed, in which case the filled row is flagged + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "irregular timestamps are rejected or regularised, never silently misaligned" + +falsification_tests: +- id: FALSIFY-CRUX-O-19-001 + rule: "a panel of N items forecasts N x prediction_length rows" + prediction: "on a fixture of 5 items with 200 hourly points each and prediction_length=24 the output has exactly 120 rows, each (item_id, timestamp) unique, timestamps continuing each item's last stamp at freq=h" + test: >- + LIVE-PENDING - a panel of N items forecasts N x prediction_length rows. No test surface exists today because the capability is unimplemented: Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq (aprender#3389, CRUX-O-19). PROMOTE by authoring a test named panel_forecast_shape in module `forecast/predictor/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a panel of N items forecasts N x prediction_length rows' is violated — the autogluon parity claim for CRUX-O-19 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-19-002 + rule: "irregular timestamps are rejected or regularised, never silently misaligned" + prediction: "an item with a missing hour is an Err naming the item and gap unless fill=forward is passed, in which case the filled row is flagged" + test: >- + LIVE-PENDING - irregular timestamps are rejected or regularised, never silently misaligned. No test surface exists today because the capability is unimplemented: Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq (aprender#3389, CRUX-O-19). PROMOTE by authoring a test named irregular_index_is_error in module `forecast/predictor/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'irregular timestamps are rejected or regularised, never silently misaligned' is violated — the autogluon parity claim for CRUX-O-19 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "a panel of N items forecasts N x prediction_length rows" +- type: invariant + property: "irregular timestamps are rejected or regularised, never silently misaligned" + +kani_harnesses: +- id: KH-CRUX-O-19-001 + obligation: a_panel_of_n_items_forecasts_n_x_prediction_length_rows + property: a_panel_of_n_items_forecasts_n_x_prediction_length_rows_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-19-002 + obligation: irregular_timestamps_are_rejected_or_regularised__never_sile + property: irregular_timestamps_are_rejected_or_regularised__never_sile_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-20-v1.yaml b/contracts/crux-O-20-v1.yaml new file mode 100644 index 0000000000..694349d2dd --- /dev/null +++ b/contracts/crux-O-20-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-20 — Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3390, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-20 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 5 # 1..5, critical priority in pmat work + intake_status: missing + github_issue: 3390 + description: > + Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE). Competitor verb: TimeSeriesPredictor(eval_metric="WQL", quantile_levels=[0.1,0.5,0.9]). Gap measured in aprender at eb262f8eb: ARIMA.forecast returns a point path; metrics/probabilistic.rs holds classification-probability metrics, and metrics/regression.rs has no seasonal-scaled (MASE/RMSSE) or quantile (WQL/MQL) losses. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3390' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + quantile_forecasts_are_monotone_in_the_quantile_level: + formula: | + for every (item, step) q0.1 <= q0.5 <= q0.9, asserted over the whole fixture; a mutation that shuffles the quantile columns turns this RED + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "quantile forecasts are monotone in the quantile level" + each_metric_matches_the_autogluon_reference_value_on_a_pinne: + formula: | + WQL, MASE, SMAPE, RMSSE, WAPE and MQL computed on a committed (y_true, y_pred, quantiles) fixture equal the reference values produced by autogluon.timeseries.metrics within 1e-9 + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "each metric matches the AutoGluon reference value on a pinned fixture" + +falsification_tests: +- id: FALSIFY-CRUX-O-20-001 + rule: "quantile forecasts are monotone in the quantile level" + prediction: "for every (item, step) q0.1 <= q0.5 <= q0.9, asserted over the whole fixture; a mutation that shuffles the quantile columns turns this RED" + test: >- + LIVE-PENDING - quantile forecasts are monotone in the quantile level. No test surface exists today because the capability is unimplemented: Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) (aprender#3390, CRUX-O-20). PROMOTE by authoring a test named quantiles_monotone in module `forecast/metrics/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'quantile forecasts are monotone in the quantile level' is violated — the autogluon parity claim for CRUX-O-20 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-20-002 + rule: "each metric matches the AutoGluon reference value on a pinned fixture" + prediction: "WQL, MASE, SMAPE, RMSSE, WAPE and MQL computed on a committed (y_true, y_pred, quantiles) fixture equal the reference values produced by autogluon.timeseries.metrics within 1e-9" + test: >- + LIVE-PENDING - each metric matches the AutoGluon reference value on a pinned fixture. No test surface exists today because the capability is unimplemented: Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) (aprender#3390, CRUX-O-20). PROMOTE by authoring a test named metrics_match_autogluon_reference in module `forecast/metrics/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'each metric matches the AutoGluon reference value on a pinned fixture' is violated — the autogluon parity claim for CRUX-O-20 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "quantile forecasts are monotone in the quantile level" +- type: invariant + property: "each metric matches the AutoGluon reference value on a pinned fixture" + +kani_harnesses: +- id: KH-CRUX-O-20-001 + obligation: quantile_forecasts_are_monotone_in_the_quantile_level + property: quantile_forecasts_are_monotone_in_the_quantile_level_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-20-002 + obligation: each_metric_matches_the_autogluon_reference_value_on_a_pinne + property: each_metric_matches_the_autogluon_reference_value_on_a_pinne_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-21-v1.yaml b/contracts/crux-O-21-v1.yaml new file mode 100644 index 0000000000..63608abc91 --- /dev/null +++ b/contracts/crux-O-21-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-21 — Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3391, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-21 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3391 + description: > + Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions. Competitor verb: fit(..., num_val_windows="auto", refit_every_n_windows="auto"); predictor.backtest_predictions(). Gap measured in aprender at eb262f8eb: No time-aware validation. KFold shuffles rows; nothing cuts the last k horizons of each item as expanding-window validation sets, and there is no API that returns the validation forecasts for inspection. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3391' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + validation_windows_never_contain_a_timestamp_later_than_the_: + formula: | + for num_val_windows=3 and val_step_size=h, each window's max training timestamp < min validation timestamp per item, asserted for every item and window + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "validation windows never contain a timestamp later than the training cut for that window" + backtest_predictions_rows_align_with_backtest_targets: + formula: | + the two frames have identical (window, item_id, timestamp) index sets and no NaN targets + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "backtest_predictions rows align with backtest_targets" + +falsification_tests: +- id: FALSIFY-CRUX-O-21-001 + rule: "validation windows never contain a timestamp later than the training cut for that window" + prediction: "for num_val_windows=3 and val_step_size=h, each window's max training timestamp < min validation timestamp per item, asserted for every item and window" + test: >- + LIVE-PENDING - validation windows never contain a timestamp later than the training cut for that window. No test surface exists today because the capability is unimplemented: Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions (aprender#3391, CRUX-O-21). PROMOTE by authoring a test named windows_never_leak_future in module `forecast/backtest/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'validation windows never contain a timestamp later than the training cut for that window' is violated — the autogluon parity claim for CRUX-O-21 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-21-002 + rule: "backtest_predictions rows align with backtest_targets" + prediction: "the two frames have identical (window, item_id, timestamp) index sets and no NaN targets" + test: >- + LIVE-PENDING - backtest_predictions rows align with backtest_targets. No test surface exists today because the capability is unimplemented: Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions (aprender#3391, CRUX-O-21). PROMOTE by authoring a test named backtest_frames_align in module `forecast/backtest/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'backtest_predictions rows align with backtest_targets' is violated — the autogluon parity claim for CRUX-O-21 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "validation windows never contain a timestamp later than the training cut for that window" +- type: invariant + property: "backtest_predictions rows align with backtest_targets" + +kani_harnesses: +- id: KH-CRUX-O-21-001 + obligation: validation_windows_never_contain_a_timestamp_later_than_the_ + property: validation_windows_never_contain_a_timestamp_later_than_the__holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-21-002 + obligation: backtest_predictions_rows_align_with_backtest_targets + property: backtest_predictions_rows_align_with_backtest_targets_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-22-v1.yaml b/contracts/crux-O-22-v1.yaml new file mode 100644 index 0000000000..92d02732ec --- /dev/null +++ b/contracts/crux-O-22-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-22 — Known covariates, past covariates and static features in forecasting +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3392, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-22 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 3 # 1..5, medium priority in pmat work + intake_status: missing + github_issue: 3392 + description: > + Known covariates, past covariates and static features in forecasting. Competitor verb: TimeSeriesPredictor(known_covariates_names=["holiday"]); train_data.static_features = df. Gap measured in aprender at eb262f8eb: ARIMA is univariate. No container carries per-item static features or time-varying covariates, and no model consumes them. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3392' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + a_known_covariate_that_fully_determines_the_target_is_used: + formula: | + on a fixture where y = 10*holiday + noise, a model given known_covariates has MASE < 0.5 x the same model without them + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "a known covariate that fully determines the target is used" + future_covariates_missing_for_the_horizon_is_an_error: + formula: | + predict() without known_covariates for all prediction_length steps is an Err naming the first missing timestamp + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "future covariates missing for the horizon is an error" + +falsification_tests: +- id: FALSIFY-CRUX-O-22-001 + rule: "a known covariate that fully determines the target is used" + prediction: "on a fixture where y = 10*holiday + noise, a model given known_covariates has MASE < 0.5 x the same model without them" + test: >- + LIVE-PENDING - a known covariate that fully determines the target is used. No test surface exists today because the capability is unimplemented: Known covariates, past covariates and static features in forecasting (aprender#3392, CRUX-O-22). PROMOTE by authoring a test named known_covariate_reduces_error in module `forecast/covariates/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'a known covariate that fully determines the target is used' is violated — the autogluon parity claim for CRUX-O-22 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-22-002 + rule: "future covariates missing for the horizon is an error" + prediction: "predict() without known_covariates for all prediction_length steps is an Err naming the first missing timestamp" + test: >- + LIVE-PENDING - future covariates missing for the horizon is an error. No test surface exists today because the capability is unimplemented: Known covariates, past covariates and static features in forecasting (aprender#3392, CRUX-O-22). PROMOTE by authoring a test named missing_future_covariates_is_error in module `forecast/covariates/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'future covariates missing for the horizon is an error' is violated — the autogluon parity claim for CRUX-O-22 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "a known covariate that fully determines the target is used" +- type: invariant + property: "future covariates missing for the horizon is an error" + +kani_harnesses: +- id: KH-CRUX-O-22-001 + obligation: a_known_covariate_that_fully_determines_the_target_is_used + property: a_known_covariate_that_fully_determines_the_target_is_used_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-22-002 + obligation: future_covariates_missing_for_the_horizon_is_an_error + property: future_covariates_missing_for_the_horizon_is_an_error_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-23-v1.yaml b/contracts/crux-O-23-v1.yaml new file mode 100644 index 0000000000..7d76dcf42f --- /dev/null +++ b/contracts/crux-O-23-v1.yaml @@ -0,0 +1,94 @@ +# CRUX-O-23 — Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3393, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-23 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: partial + github_issue: 3393 + description: > + Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston. Competitor verb: hyperparameters={"SeasonalNaive": {}, "AutoETS": {}, "Theta": {}, "AutoARIMA": {}, "Croston": {}}. Gap measured in aprender at eb262f8eb: ARIMA exists (fixed order; no auto-order search). SeasonalNaive, ETS, Theta and the intermittent-demand family (Croston, ADIDA, IMAPA) are absent. AutoGluon runs these as its 'local' tier (timeseries/models/local/) and they anchor every leaderboard. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3393' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + seasonalnaive_is_exactly_the_lag_m_copy: + formula: | + forecast[t] == y[t - m] for the whole horizon on a fixture with m=24, byte-equal + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "SeasonalNaive is exactly the lag-m copy" + autoarima_selects_the_planted_order: + formula: | + on a fixture generated from ARIMA(2,1,1) the selected (p,d,q) equals (2,1,1) in >= 9 of 10 seeds + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "AutoARIMA selects the planted order" + ets_and_theta_match_the_statsforecast_reference: + formula: | + point forecasts on the AirPassengers fixture equal the statsforecast reference within 1e-6 + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "ETS and Theta match the statsforecast reference" + +falsification_tests: +- id: FALSIFY-CRUX-O-23-001 + rule: "SeasonalNaive is exactly the lag-m copy" + prediction: "forecast[t] == y[t - m] for the whole horizon on a fixture with m=24, byte-equal" + test: >- + LIVE-PENDING - SeasonalNaive is exactly the lag-m copy. No test surface exists today because the capability is partial: Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston (aprender#3393, CRUX-O-23). PROMOTE by authoring a test named seasonal_naive_is_lag_copy in module `forecast/local/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'SeasonalNaive is exactly the lag-m copy' is violated — the autogluon parity claim for CRUX-O-23 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-23-002 + rule: "AutoARIMA selects the planted order" + prediction: "on a fixture generated from ARIMA(2,1,1) the selected (p,d,q) equals (2,1,1) in >= 9 of 10 seeds" + test: >- + LIVE-PENDING - AutoARIMA selects the planted order. No test surface exists today because the capability is partial: Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston (aprender#3393, CRUX-O-23). PROMOTE by authoring a test named autoarima_recovers_planted_order in module `forecast/local/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'AutoARIMA selects the planted order' is violated — the autogluon parity claim for CRUX-O-23 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-23-003 + rule: "ETS and Theta match the statsforecast reference" + prediction: "point forecasts on the AirPassengers fixture equal the statsforecast reference within 1e-6" + test: >- + LIVE-PENDING - ETS and Theta match the statsforecast reference. No test surface exists today because the capability is partial: Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston (aprender#3393, CRUX-O-23). PROMOTE by authoring a test named ets_theta_match_statsforecast in module `forecast/local/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'ETS and Theta match the statsforecast reference' is violated — the autogluon parity claim for CRUX-O-23 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "SeasonalNaive is exactly the lag-m copy" +- type: invariant + property: "AutoARIMA selects the planted order" +- type: invariant + property: "ETS and Theta match the statsforecast reference" + +kani_harnesses: +- id: KH-CRUX-O-23-001 + obligation: seasonalnaive_is_exactly_the_lag_m_copy + property: seasonalnaive_is_exactly_the_lag_m_copy_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-23-002 + obligation: autoarima_selects_the_planted_order + property: autoarima_selects_the_planted_order_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-23-003 + obligation: ets_and_theta_match_the_statsforecast_reference + property: ets_and_theta_match_the_statsforecast_reference_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-O-24-v1.yaml b/contracts/crux-O-24-v1.yaml new file mode 100644 index 0000000000..7790b58f59 --- /dev/null +++ b/contracts/crux-O-24-v1.yaml @@ -0,0 +1,75 @@ +# CRUX-O-24 — Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning +# CRUX category O: AutoML Parity (AutoGluon 1.6.3). +# Filed from the 2026-09-16 competitive sweep of ../autogluon. Tracks GitHub #3394, epic #3370. +# registry: false — this contract carries real falsification gates and claims no +# exemption (operator ruling 2026-08-21, "no contract exemptions"). + +metadata: + id: CRUX-O-24 + version: "1.0.0" + created: "2026-09-16" + updated: "2026-09-16" + author: PAIML Engineering + registry: false + status: draft + parent_contracts: + - crux-competitive-research-ux-v1 + category: "O — AutoML Parity" + competitor: autogluon + demand_score: 4 # 1..5, high priority in pmat work + intake_status: missing + github_issue: 3394 + description: > + Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning. Competitor verb: hyperparameters={"Chronos2": {"fine_tune": True}} # 1.5 spotlight; 1.6 adds Toto-2. Gap measured in aprender at eb262f8eb: apr pull fetches LLM checkpoints (Category A) and the inference crates run decoder transformers, but no forecasting head exists: nothing tokenises a numeric series into the model's input, samples a horizon, and maps it back to quantiles. + references: + - 'master: contracts/crux-competitive-research-ux-v1.yaml — §5 + §12' + - 'epic: https://github.com/paiml/aprender/issues/3370' + - 'story: https://github.com/paiml/aprender/issues/3394' + - 'competitor: https://github.com/autogluon/autogluon (1.6.3)' + - 'evidence: evidence/crux/autogluon/api-surface.md' + +equations: + zero_shot_output_matches_a_pinned_reference: + formula: | + a pinned small checkpoint pulled by `apr pull` forecasts the committed fixture within 1e-3 of the reference quantiles + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "zero-shot output matches a pinned reference" + fine_tuning_changes_the_weights_and_improves_the_in_domain_m: + formula: | + after fine_tune on the fixture the WQL improves by >= 5% relative AND at least one weight tensor differs from the pulled checkpoint + domain: "the fixture and inputs named in the prediction" + codomain: "PASS iff the prediction holds; the mutation named in the prediction (where one is named) turns it RED" + invariants: + - "fine-tuning changes the weights and improves the in-domain metric" + +falsification_tests: +- id: FALSIFY-CRUX-O-24-001 + rule: "zero-shot output matches a pinned reference" + prediction: "a pinned small checkpoint pulled by `apr pull` forecasts the committed fixture within 1e-3 of the reference quantiles" + test: >- + LIVE-PENDING - zero-shot output matches a pinned reference. No test surface exists today because the capability is unimplemented: Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning (aprender#3394, CRUX-O-24). PROMOTE by authoring a test named zero_shot_matches_reference in module `forecast/pretrained/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'zero-shot output matches a pinned reference' is violated — the autogluon parity claim for CRUX-O-24 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" +- id: FALSIFY-CRUX-O-24-002 + rule: "fine-tuning changes the weights and improves the in-domain metric" + prediction: "after fine_tune on the fixture the WQL improves by >= 5% relative AND at least one weight tensor differs from the pulled checkpoint" + test: >- + LIVE-PENDING - fine-tuning changes the weights and improves the in-domain metric. No test surface exists today because the capability is unimplemented: Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning (aprender#3394, CRUX-O-24). PROMOTE by authoring a test named finetune_improves_and_changes_weights in module `forecast/pretrained/tests` of crate `aprender-core`; this gate then binds to it and the contract moves from draft to active. Until then the obligation is RECORDED and unfalsifiable-by-absence, NOT satisfied. + if_fails: "rule 'fine-tuning changes the weights and improves the in-domain metric' is violated — the autogluon parity claim for CRUX-O-24 is FALSE and the contract MUST be re-verified against the competitor's canonical implementation" + +proof_obligations: +- type: invariant + property: "zero-shot output matches a pinned reference" +- type: invariant + property: "fine-tuning changes the weights and improves the in-domain metric" + +kani_harnesses: +- id: KH-CRUX-O-24-001 + obligation: zero_shot_output_matches_a_pinned_reference + property: zero_shot_output_matches_a_pinned_reference_holds_on_bounded_input + bound: 4 +- id: KH-CRUX-O-24-002 + obligation: fine_tuning_changes_the_weights_and_improves_the_in_domain_m + property: fine_tuning_changes_the_weights_and_improves_the_in_domain_m_holds_on_bounded_input + bound: 4 diff --git a/contracts/crux-competitive-research-ux-v1.yaml b/contracts/crux-competitive-research-ux-v1.yaml index dd058981e9..6151302d8b 100644 --- a/contracts/crux-competitive-research-ux-v1.yaml +++ b/contracts/crux-competitive-research-ux-v1.yaml @@ -39,6 +39,7 @@ metadata: - "https://github.com/huggingface/transformers" - "https://github.com/vllm-project/vllm" - "https://github.com/mlfoundations/open_clip" + - "https://github.com/autogluon/autogluon" # ───────────────────────────────────────────────────────────── # Demand scoring rubric (see subspec §4) @@ -100,6 +101,11 @@ evidence_sources: capability_matrix: "evidence/crux/openclaw/capability-matrix.yaml" gaps: "evidence/crux/openclaw/gaps.md" interpretation_note: "Resolved 2026-04-18: OpenCLAW is openclaw.ai (local-first personal AI assistant / agent orchestration). NOT OpenCLIP. Category J rewritten accordingly." + autogluon: + readme_verbs: "evidence/crux/autogluon/readme-verbs.txt" + canonical_flow: "evidence/crux/autogluon/hello.sh" + api_surface: "evidence/crux/autogluon/api-surface.md" + interpretation_note: "AutoGluon 1.6.3 (../autogluon @ 77946149), surveyed 2026-09-16. Category O = AutoML parity (tabular + forecasting). MultiModalPredictor CUT on epic #3370." openclip: readme_verbs: "evidence/crux/openclip/readme-verbs.txt" top_issues: "evidence/crux/openclip/top-issues.json" @@ -460,15 +466,46 @@ stories: - { id: CRUX-N-16, title: "Barnes-Hut t-SNE (O(n log n))", competitor: linfa, demand_score: 3, status: partial, contract: crux-N-16-v1.yaml } - { id: CRUX-N-17, title: "Vision ops: NMS, GPU connected components", competitor: burn, demand_score: 2, status: partial, contract: crux-N-17-v1.yaml } + # Category O — AutoML Parity (24) — AutoGluon 1.6.3 (aprender#3370, 2026-09-16). + # Competitor source: ../autogluon at 77946149. Three products (TabularPredictor, + # TimeSeriesPredictor, MultiModalPredictor); MultiModal is CUT on the epic. + # aprender's automl/ is a single-estimator tuner and time_series/ is one ARIMA; + # the gap is the predictor-level UX (fit(label) -> leaderboard -> deploy), not + # the algorithms. Not a BEAT pillar. Registry edit: CRUX_COMPETITORS. + - { id: CRUX-O-01, title: "One-call tabular AutoML: fit(label) -> predict on a CSV", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-01-v1.yaml } + - { id: CRUX-O-02, title: "Problem-type inference: binary / multiclass / regression / quantile fr", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-02-v1.yaml } + - { id: CRUX-O-03, title: "Quality presets (medium / good / high / best / extreme) that name a mo", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-03-v1.yaml } + - { id: CRUX-O-04, title: "Leaderboard: per-model validation/test score, fit time, predict time a", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-04-v1.yaml } + - { id: CRUX-O-05, title: "Automatic feature-type inference and the AutoML feature pipeline (nume", competitor: autogluon, demand_score: 5, status: partial, contract: crux-O-05-v1.yaml } + - { id: CRUX-O-06, title: "K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oo", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-06-v1.yaml } + - { id: CRUX-O-07, title: "Multi-layer stack ensembling with a leakage guard (num_stack_levels, a", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-07-v1.yaml } + - { id: CRUX-O-08, title: "Greedy weighted-ensemble selection over fitted models (Caruana ensembl", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-08-v1.yaml } + - { id: CRUX-O-09, title: "Time-budgeted portfolio fit: time_limit split across models, each mode", competitor: autogluon, demand_score: 5, status: partial, contract: crux-O-09-v1.yaml } + - { id: CRUX-O-10, title: "Predictor-level permutation feature importance with p-values and confi", competitor: autogluon, demand_score: 4, status: partial, contract: crux-O-10-v1.yaml } + - { id: CRUX-O-11, title: "Decision-threshold calibration for binary metrics (calibrate_decision_", competitor: autogluon, demand_score: 4, status: partial, contract: crux-O-11-v1.yaml } + - { id: CRUX-O-12, title: "refit_full: retrain the selected models on train+validation after mode", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-12-v1.yaml } + - { id: CRUX-O-13, title: "Model distillation: compress the ensemble into one fast student", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-13-v1.yaml } + - { id: CRUX-O-14, title: "Deployment artifact: clone_for_deployment / keep_only_best / save_spac", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-14-v1.yaml } + - { id: CRUX-O-15, title: "Inference-latency constraint during model selection (infer_limit, infe", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-15-v1.yaml } + - { id: CRUX-O-16, title: "Fit diagnostics: fit_summary, model_failures and learning curves", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-16-v1.yaml } + - { id: CRUX-O-17, title: "Tabular foundation model: in-context prediction with a pretrained tran", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-17-v1.yaml } + - { id: CRUX-O-18, title: "Memory-aware fit: per-model memory estimate and a memory_limit that sk", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-18-v1.yaml } + - { id: CRUX-O-19, title: "Multi-series forecasting predictor: (item_id, timestamp) panel data, p", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-19-v1.yaml } + - { id: CRUX-O-20, title: "Probabilistic forecasts: quantile_levels and the forecasting metric fa", competitor: autogluon, demand_score: 5, status: missing, contract: crux-O-20-v1.yaml } + - { id: CRUX-O-21, title: "Rolling-window backtesting: num_val_windows, refit_every_n_windows, ba", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-21-v1.yaml } + - { id: CRUX-O-22, title: "Known covariates, past covariates and static features in forecasting", competitor: autogluon, demand_score: 3, status: missing, contract: crux-O-22-v1.yaml } + - { id: CRUX-O-23, title: "Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Cro", competitor: autogluon, demand_score: 4, status: partial, contract: crux-O-23-v1.yaml } + - { id: CRUX-O-24, title: "Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by a", competitor: autogluon, demand_score: 4, status: missing, contract: crux-O-24-v1.yaml } + # ───────────────────────────────────────────────────────────── # Coverage — intake v2.0.0 (verified by awk over §5 of subspec) # ───────────────────────────────────────────────────────────── coverage_intake: supported: 43 - partial: 72 - missing: 152 + partial: 77 + missing: 171 unclear: 0 - total: 267 + total: 291 notes: | ID gaps at C-14, F-10, H-04, I-05, K-06 are intentional — those five stories were dropped pre-v1.0.0 as duplicative (see inline @@ -477,6 +514,9 @@ coverage_intake: 5 partial. It is the first category whose competitors are Rust-native frameworks rather than Python/C++ incumbents, which is why it required a CRUX_COMPETITORS registry edit rather than reuse of an existing source. + Category O (24 rows, aprender#3370) was added 2026-09-16: 19 missing, + 5 partial. Competitor autogluon (1.6.3) — AutoML parity: predictor-level + tabular AutoML and multi-series forecasting. Same registry edit. # ───────────────────────────────────────────────────────────── # Falsification conditions diff --git a/crates/aprender-contracts/src/schema/crux_intake_tests.rs b/crates/aprender-contracts/src/schema/crux_intake_tests.rs index 4a6728c509..ed3250c1fa 100644 --- a/crates/aprender-contracts/src/schema/crux_intake_tests.rs +++ b/crates/aprender-contracts/src/schema/crux_intake_tests.rs @@ -216,6 +216,8 @@ fn competitor_registry_covers_the_corpus_vocabulary() { // 17 contracts: burn ×7, linfa ×10. "burn", "linfa", + // Category O — AutoML Parity (aprender#3370, 2026-09-16). 24 contracts. + "autogluon", ] { assert!( CRUX_COMPETITORS.contains(&required), @@ -258,6 +260,8 @@ fn beat_incumbents_cannot_name_the_crux_corpus() { // is why category N required a registry edit rather than a reuse. "burn", "linfa", + // AutoGluon: not a BEAT pillar either; scikit-learn stays the pillar. + "autogluon", ] { assert!(!beat_accepts(c), "BEAT_INCUMBENTS unexpectedly accepts {c}"); } diff --git a/crates/aprender-contracts/src/schema/validator.rs b/crates/aprender-contracts/src/schema/validator.rs index aea0df1829..4468b410d7 100644 --- a/crates/aprender-contracts/src/schema/validator.rs +++ b/crates/aprender-contracts/src/schema/validator.rs @@ -90,8 +90,18 @@ pub fn validate_contract(contract: &Contract) -> Vec { /// exercised by at least one contract in `contracts/`; adding a competitor is a /// deliberate one-line edit here plus a test, which is the point — an open /// domain is what let `THIS-COMPETITOR-DOES-NOT-EXIST` validate. -pub(crate) const CRUX_COMPETITORS: [&str; 14] = [ +pub(crate) const CRUX_COMPETITORS: [&str; 15] = [ "apr-qa-playbook", + // AutoGluon (autogluon/autogluon) — the AutoML library, 1.6.3 at admission + // (../autogluon @ 77946149). Added 2026-09-16 with 24 category-O stories + // extracted from its three predictors: TabularPredictor (fit(label), + // presets, leaderboard, feature pipeline, bagging/stacking/weighted + // ensemble, budgets, deployment) and TimeSeriesPredictor (panel data, + // quantile metrics, backtesting, local baselines, Chronos-2 class + // pretrained forecasters). NOT a BEAT pillar — aprender claims no pinned + // benchmark win over AutoGluon; this is a capability/UX source. Epic + // aprender#3370. + "autogluon", // Burn (tracel-ai/burn) — the Rust deep-learning framework, 0.21.0 / 15.9k // stars / 312 reverse-dependencies at admission. Added 2026-09-12 with 7 // category-N stories extracted from its crate surface: burn-linalg (SVD), diff --git a/docs/roadmaps/entries/PMAT-3370.yaml b/docs/roadmaps/entries/PMAT-3370.yaml new file mode 100644 index 0000000000..d750e8c331 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3370.yaml @@ -0,0 +1,22 @@ +- id: PMAT-3370 + github_issue: 3370 + item_type: task + title: 'EPIC: CRUX category O — AutoML parity vs AutoGluon 1.6.3 — 9 P0, 9 P1, 6 P2 (24 stories)' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: docs/specifications/crux-competitive-research-ux-workflows.md + acceptance_criteria: + - 'All 24 children (#3371-#3394) closed or explicitly re-cut; apr automl fit/predict on iris and apr forecast fit on a 5-item panel run end to end; each child lands with a contract and mutation-RED evidence.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - epic + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3371.yaml b/docs/roadmaps/entries/PMAT-3371.yaml new file mode 100644 index 0000000000..d7c032a3b9 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3371.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3371 + github_issue: 3371 + item_type: task + title: 'P0: One-call tabular AutoML: fit(label) -> predict on a CSV — CRUX-O-01' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-01, competitor autogluon, demand 5. No predictor-level AutoML entry point exists. crates/aprender-core/src/automl/ is a hyperparameter TUNER (AutoTuner, TPE, GridSearch, RandomSearch, DESearch, TimeBudget) that tunes ONE estimator the caller already chose; nothing takes a labelled table and returns a fitted model. `apr train` is causal-LM pre-training only (crates/apr-cli/src/commands/train.rs:1-5) and `apr finetune --task classify` is text classification. FALSIFIABLE (contracts/crux-O-01-v1.yaml): fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline — on the iris fixture (crates/aprender-core/src/datasets/iris.csv) accuracy >= 0.90 AND the majority-class baseline is asserted at 0.333 in the same test; the same call works for a regression label without a problem_type argument — a numeric label column yields a regressor whose R^2 on a held-out split exceeds 0.5 AND the mean-predictor baseline is asserted at ~0.0; apr automl fit is reachable from the CLI — `apr automl fit --label class train.csv --out model.apr` exits 0 and writes a loadable artifact; a missing --label exits 2 with a message naming the flag. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3372.yaml b/docs/roadmaps/entries/PMAT-3372.yaml new file mode 100644 index 0000000000..b3aa11f6cf --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3372.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3372 + github_issue: 3372 + item_type: task + title: 'P0: Problem-type inference: binary / multiclass / regression / quantile from the label column — CRUX-O-02' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-02, competitor autogluon, demand 5. No function infers a task from a label column. Estimators are chosen by type name (LogisticRegression vs LinearRegression); DataFrame in crates/aprender-core/src/data/mod.rs carries ColumnStats but no label-kind classifier. FALSIFIABLE (contracts/crux-O-02-v1.yaml): two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression — a table of (label column, expected kind) fixtures including the ambiguous cases {0,1} as int, {0.0,1.0} as float, and 30 unique floats over 1000 rows all classify as documented; the override wins and a contradictory override is rejected — problem_type=regression on a string label column is an Err naming the column, not a silent cast. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3373.yaml b/docs/roadmaps/entries/PMAT-3373.yaml new file mode 100644 index 0000000000..11466513ff --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3373.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3373 + github_issue: 3373 + item_type: task + title: 'P0: Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget — CRUX-O-03' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-03, competitor autogluon, demand 5. No preset vocabulary. AutoTuner takes a SearchSpace the caller hand-builds (crates/aprender-core/src/automl/params.rs); there is no named bundle of {models, bagging, stacking, time_limit}. FALSIFIABLE (contracts/crux-O-03-v1.yaml): every preset name resolves to a portfolio and unknown names are rejected with the valid list — the five quality presets each yield a non-empty ordered model list; `presets="bestest"` is an Err whose message contains all five valid names; presets are ordered: a higher preset never fits FEWER model families than the one below it — for medium < good < high < best the family count is monotone non-decreasing, asserted pairwise. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3374.yaml b/docs/roadmaps/entries/PMAT-3374.yaml new file mode 100644 index 0000000000..2780f35fa1 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3374.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3374 + github_issue: 3374 + item_type: task + title: 'P0: Leaderboard: per-model validation/test score, fit time, predict time and stack level — CRUX-O-04' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-04, competitor autogluon, demand 5. No leaderboard type. GridSearchCVResult in model_selection/ ranks parameter settings of one estimator; TuneResult in automl/tuner.rs is a single best trial. Nothing tabulates several fitted models with timings. FALSIFIABLE (contracts/crux-O-04-v1.yaml): the leaderboard is sorted by validation score descending and its columns are fixed — columns == [model, score_val, score_test?, pred_time_val, fit_time, stack_level, fit_order] and score_val is non-increasing row to row; timings are measured, not defaulted — every fit_time and pred_time_val is > 0 after a real fit; a leaderboard built with no fit has zero rows, not zero timings; apr automl leaderboard prints the same table from a saved artifact — `apr automl leaderboard model.apr --json` emits the rows byte-equal to the in-process leaderboard. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3375.yaml b/docs/roadmaps/entries/PMAT-3375.yaml new file mode 100644 index 0000000000..b6d14f5cf0 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3375.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3375 + github_issue: 3375 + item_type: task + title: 'P0: Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) — CRUX-O-05' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-05, competitor autogluon, demand 5. Encoders exist (LabelEncoder, OneHotEncoder, OrdinalEncoder, StandardScaler, PolynomialFeatures in crates/aprender-core/src/preprocessing/) but every one is applied by hand to a column the caller already typed. There is no pass that reads a raw DataFrame, infers each column''s kind, and emits a fitted transform. Datetime expansion and text n-gram features do not exist. TfidfVectorizer exists in text/ but is not wired to a tabular pipeline. FALSIFIABLE (contracts/crux-O-05-v1.yaml): column kinds are inferred from raw values — a fixture CSV with int, float, low-cardinality string, high-cardinality string, ISO datetime and free-text columns is typed as {numeric, numeric, categorical, text, datetime, text} exactly; constant and duplicate columns are dropped and the drop is reported — a column with one unique value and an exact duplicate of another column are both absent from the transformed output AND named in the fit report; the fitted pipeline is deterministic under transform — transform(train) then transform(train) are byte-identical and transform(test) never sees a category unseen at fit as anything but the reserved unknown code. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3376.yaml b/docs/roadmaps/entries/PMAT-3376.yaml new file mode 100644 index 0000000000..948fd5b023 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3376.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3376 + github_issue: 3376 + item_type: task + title: 'P1: K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) — CRUX-O-06' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-06, competitor autogluon, demand 4. KFold and StratifiedKFold exist (model_selection/) and cross_validate scores them, but no wrapper trains one child per fold, keeps all children, averages them at predict time and exposes the out-of-fold matrix. RandomForest bags trees internally and is not reusable for other estimators. FALSIFIABLE (contracts/crux-O-06-v1.yaml): OOF predictions cover every training row exactly once — for n rows and k folds the OOF matrix has n rows, no NaN, and each row was predicted by the one child that did not see it (asserted through a fold-id trace); bagged prediction is the mean of the children — predict_proba of the bag equals the elementwise mean of the k children''s predict_proba within 1e-12. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3377.yaml b/docs/roadmaps/entries/PMAT-3377.yaml new file mode 100644 index 0000000000..0298e6a705 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3377.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3377 + github_issue: 3377 + item_type: task + title: 'P1: Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) — CRUX-O-07' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-07, competitor autogluon, demand 4. No stacking. crates/aprender-core/src/stack/ is a deployment-health module (StackHealth, InferenceConfig), not a model stacker. ensemble/ holds MixtureOfExperts gating only. FALSIFIABLE (contracts/crux-O-07-v1.yaml): level-2 features are OOF, never in-sample — a mutation that feeds in-sample level-1 predictions to level 2 is detected by the leak test: the L2 holdout score on a pure-noise label rises above chance (asserted RED) while the OOF path stays at chance; dynamic stacking falls back when stacking hurts — on a fixture where L2 holdout score < L1 holdout score, the final model is the L1 ensemble and the decision is recorded in the fit summary. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3378.yaml b/docs/roadmaps/entries/PMAT-3378.yaml new file mode 100644 index 0000000000..762d7bcec4 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3378.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3378 + github_issue: 3378 + item_type: task + title: 'P0: Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) — CRUX-O-08' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-08, competitor autogluon, demand 5. No ensemble-selection algorithm. MixtureOfExperts learns a gating network (ensemble/moe.rs); nothing performs the forward greedy selection with replacement over base-model validation predictions that yields non-negative weights summing to 1. FALSIFIABLE (contracts/crux-O-08-v1.yaml): weights are a probability vector and the ensemble never scores below its best member — sum(w)=1, all w>=0, and validation metric(ensemble) >= max over members within 1e-9, on 3 fixtures; selection is greedy with replacement and reproducible — with ensemble_size=25 the weight of a member equals its selection count / 25; two runs on the same inputs produce identical weights. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3379.yaml b/docs/roadmaps/entries/PMAT-3379.yaml new file mode 100644 index 0000000000..b2139bc2fc --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3379.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3379 + github_issue: 3379 + item_type: task + title: 'P0: Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share — CRUX-O-09' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-09, competitor autogluon, demand 5. TimeBudget and EarlyStopping exist in automl/tuner.rs but budget ONE tuner. There is no allocation of a global limit across an ordered portfolio, no per-model time share, and no ''skip the rest'' when the budget is exhausted. FALSIFIABLE (contracts/crux-O-09-v1.yaml): the wall-clock of fit never exceeds time_limit by more than the tolerance — with time_limit=5s on a portfolio that would take >60s unconstrained, elapsed <= 5s + 1s AND at least one model was skipped with reason=budget in the summary; the budget is redistributed when a model finishes early — a model that uses 10% of its share returns the remainder to the pool; the next model''s share is asserted larger than the naive equal split. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3380.yaml b/docs/roadmaps/entries/PMAT-3380.yaml new file mode 100644 index 0000000000..f141de9e6b --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3380.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3380 + github_issue: 3380 + item_type: task + title: 'P1: Predictor-level permutation feature importance with p-values and confidence intervals — CRUX-O-10' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-10, competitor autogluon, demand 4. PermutationImportance exists in crates/aprender-core/src/interpret/ for a single estimator. Missing: the predictor-level call on raw (pre-pipeline) columns, num_shuffle_sets repeats, stddev / p-value / p99 columns, and subsampling. AutoGluon 1.6 also cut this call''s memory 25x (#5645). FALSIFIABLE (contracts/crux-O-10-v1.yaml): importance is reported per raw input column — on a fixture whose datetime column expands to 4 features, the importance table has one row for the datetime column, not four; a pure-noise column has importance statistically indistinguishable from zero — with num_shuffle_sets=10 the noise column''s p-value > 0.05 AND the signal column''s p-value < 0.01 on the same run. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3381.yaml b/docs/roadmaps/entries/PMAT-3381.yaml new file mode 100644 index 0000000000..ea5cded554 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3381.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3381 + github_issue: 3381 + item_type: task + title: 'P1: Decision-threshold calibration for binary metrics (calibrate_decision_threshold) — CRUX-O-11' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-11, competitor autogluon, demand 4. Probability calibration exists (PlattScaling, IsotonicRegression, TemperatureScaling in calibration.rs) but nothing searches the decision threshold that maximises f1 / balanced_accuracy / mcc on validation data and stores it on the predictor (`decision_threshold`, `set_decision_threshold`). FALSIFIABLE (contracts/crux-O-11-v1.yaml): the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search — on an imbalanced fixture (5% positives) f1 at the calibrated threshold > f1 at 0.5 by >= 0.05 absolute; the threshold is persisted with the model — save then load reproduces predict() bit-identically including the threshold; a mutation that resets the threshold to 0.5 on load turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3382.yaml b/docs/roadmaps/entries/PMAT-3382.yaml new file mode 100644 index 0000000000..9a08ae24a7 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3382.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3382 + github_issue: 3382 + item_type: task + title: 'P2: refit_full: retrain the selected models on train+validation after model selection — CRUX-O-12' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-12, competitor autogluon, demand 3. No refit-on-full-data step. cross_validate and grid_search return scores; the model returned is the one fitted on a fold, not on all rows. FALSIFIABLE (contracts/crux-O-12-v1.yaml): the refit model saw every row — the refit estimator''s training-row count equals n_train + n_val, asserted through the fitted row count it reports; refit keeps the selected hyperparameters and drops the bag children — the artifact after refit has one child per selected model and its hyperparameters are byte-equal to the pre-refit winner''s. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3383.yaml b/docs/roadmaps/entries/PMAT-3383.yaml new file mode 100644 index 0000000000..82060a6508 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3383.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3383 + github_issue: 3383 + item_type: task + title: 'P2: Model distillation: compress the ensemble into one fast student — CRUX-O-13' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-13, competitor autogluon, demand 3. Distillation exists only for LLMs in the training crate (knowledge distillation of transformers); nothing trains a single tabular student on the teacher ensemble''s soft labels with data augmentation. FALSIFIABLE (contracts/crux-O-13-v1.yaml): the student is faster and within tolerance of the teacher — student pred_time < 0.25 x teacher pred_time AND student validation score >= teacher score - 0.02 on a fixture; augmentation produces rows the training set does not contain — with augment_method=spunge the student''s training set has > n_train rows and the added rows are not row-equal to any original. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3384.yaml b/docs/roadmaps/entries/PMAT-3384.yaml new file mode 100644 index 0000000000..3de567d492 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3384.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3384 + github_issue: 3384 + item_type: task + title: 'P1: Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file — CRUX-O-14' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-14, competitor autogluon, demand 4. Single estimators serialize to .apr (bundle/, serialization/). No artifact holds a fitted feature pipeline (O-05) plus several models plus ensemble weights plus a decision threshold and loads them as one predictor. Nothing prunes non-selected models from disk. FALSIFIABLE (contracts/crux-O-14-v1.yaml): a deployment clone is smaller and predicts identically — clone_for_deployment artifact bytes < 0.5 x full artifact bytes AND predict() on the clone is bit-identical to predict() on the original for the test fixture; the artifact is self-describing and refuses a schema drift — load() on an artifact whose feature schema disagrees with the input columns is an Err naming the first mismatched column, not a wrong prediction. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3385.yaml b/docs/roadmaps/entries/PMAT-3385.yaml new file mode 100644 index 0000000000..9220aad0e4 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3385.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3385 + github_issue: 3385 + item_type: task + title: 'P2: Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size) — CRUX-O-15' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-15, competitor autogluon, demand 3. No model or ensemble is ever excluded for being slow at predict time; leaderboard timings (O-04) do not exist to compare against. FALSIFIABLE (contracts/crux-O-15-v1.yaml): the selected ensemble respects the per-row latency limit — with infer_limit=L the measured pred_time per row of the final model <= L AND at least one faster-but-worse model was preferred over a slower-but-better one (asserted on a fixture built to force the trade). Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3386.yaml b/docs/roadmaps/entries/PMAT-3386.yaml new file mode 100644 index 0000000000..6bf9ce886c --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3386.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3386 + github_issue: 3386 + item_type: task + title: 'P2: Fit diagnostics: fit_summary, model_failures and learning curves — CRUX-O-16' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-16, competitor autogluon, demand 3. ProgressCallback in automl/tuner.rs streams trial results; there is no post-fit summary object listing models trained, models failed with their error, per-model hyperparameters and per-iteration validation curves. FALSIFIABLE (contracts/crux-O-16-v1.yaml): a model that raises during fit is recorded, not swallowed and not fatal — a portfolio containing a deliberately failing model finishes; model_failures() has exactly one row naming the model and the error string; the leaderboard omits it; learning curves have one point per boosting/epoch iteration — for a GBM with n_estimators=50 and learning_curves=true the curve has 50 validation points, monotone non-increasing after early-stopping''s best iteration is asserted absent. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3387.yaml b/docs/roadmaps/entries/PMAT-3387.yaml new file mode 100644 index 0000000000..32e0d5c86f --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3387.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3387 + github_issue: 3387 + item_type: task + title: 'P1: Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) — CRUX-O-17' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-17, competitor autogluon, demand 4. No tabular in-context learner. nn/ and the inference crates run causal LMs; nothing consumes (X_train, y_train, X_test) as one context and predicts without gradient steps. AutoGluon 1.4-1.6 added TabPFNv2/2.5/2.6/3, TabICL/v2, Mitra, TabDPT, Nori and made them the extreme preset. FALSIFIABLE (contracts/crux-O-17-v1.yaml): zero-gradient prediction matches a pinned reference within tolerance — loading a pinned small checkpoint via apr pull and predicting a 100-row fixture matches the committed reference probabilities within 1e-4; the context limit is enforced, not silently truncated — a context of rows > the model''s documented max is an Err naming the limit; a mutation that truncates instead turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3388.yaml b/docs/roadmaps/entries/PMAT-3388.yaml new file mode 100644 index 0000000000..6b6c698bcd --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3388.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3388 + github_issue: 3388 + item_type: task + title: 'P2: Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit — CRUX-O-18' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-18, competitor autogluon, demand 3. No estimator reports an expected peak memory before fitting and nothing checks a limit. AutoGluon 1.6 spent 4 PRs on this (#5757, #5768, #5791, GPU budgeting for parallel folds). FALSIFIABLE (contracts/crux-O-18-v1.yaml): a model whose estimate exceeds the limit is skipped with reason=memory — with memory_limit=64MiB on a fixture where the estimate for the largest model is > 64MiB, that model is absent from the leaderboard and present in the skip list with its estimate; the estimate is not a constant — the estimate for a 10x larger fixture is asserted larger than for the base fixture; a mutation returning a constant turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3389.yaml b/docs/roadmaps/entries/PMAT-3389.yaml new file mode 100644 index 0000000000..acb5bc1a8f --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3389.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3389 + github_issue: 3389 + item_type: task + title: 'P0: Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq — CRUX-O-19' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-19, competitor autogluon, demand 5. crates/aprender-core/src/time_series/mod.rs is one struct, ARIMA, on one f32 series (fit/forecast/order). There is no panel container keyed by item and timestamp, no frequency, no horizon-first API and no per-item forecast. FALSIFIABLE (contracts/crux-O-19-v1.yaml): a panel of N items forecasts N x prediction_length rows — on a fixture of 5 items with 200 hourly points each and prediction_length=24 the output has exactly 120 rows, each (item_id, timestamp) unique, timestamps continuing each item''s last stamp at freq=h; irregular timestamps are rejected or regularised, never silently misaligned — an item with a missing hour is an Err naming the item and gap unless fill=forward is passed, in which case the filled row is flagged. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3390.yaml b/docs/roadmaps/entries/PMAT-3390.yaml new file mode 100644 index 0000000000..80cffcbefa --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3390.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3390 + github_issue: 3390 + item_type: task + title: 'P0: Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) — CRUX-O-20' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-20, competitor autogluon, demand 5. ARIMA.forecast returns a point path; metrics/probabilistic.rs holds classification-probability metrics, and metrics/regression.rs has no seasonal-scaled (MASE/RMSSE) or quantile (WQL/MQL) losses. FALSIFIABLE (contracts/crux-O-20-v1.yaml): quantile forecasts are monotone in the quantile level — for every (item, step) q0.1 <= q0.5 <= q0.9, asserted over the whole fixture; a mutation that shuffles the quantile columns turns this RED; each metric matches the AutoGluon reference value on a pinned fixture — WQL, MASE, SMAPE, RMSSE, WAPE and MQL computed on a committed (y_true, y_pred, quantiles) fixture equal the reference values produced by autogluon.timeseries.metrics within 1e-9. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3391.yaml b/docs/roadmaps/entries/PMAT-3391.yaml new file mode 100644 index 0000000000..c70550ddc6 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3391.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3391 + github_issue: 3391 + item_type: task + title: 'P1: Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions — CRUX-O-21' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-21, competitor autogluon, demand 4. No time-aware validation. KFold shuffles rows; nothing cuts the last k horizons of each item as expanding-window validation sets, and there is no API that returns the validation forecasts for inspection. FALSIFIABLE (contracts/crux-O-21-v1.yaml): validation windows never contain a timestamp later than the training cut for that window — for num_val_windows=3 and val_step_size=h, each window''s max training timestamp < min validation timestamp per item, asserted for every item and window; backtest_predictions rows align with backtest_targets — the two frames have identical (window, item_id, timestamp) index sets and no NaN targets. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3392.yaml b/docs/roadmaps/entries/PMAT-3392.yaml new file mode 100644 index 0000000000..8738ebd8e1 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3392.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3392 + github_issue: 3392 + item_type: task + title: 'P2: Known covariates, past covariates and static features in forecasting — CRUX-O-22' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-22, competitor autogluon, demand 3. ARIMA is univariate. No container carries per-item static features or time-varying covariates, and no model consumes them. FALSIFIABLE (contracts/crux-O-22-v1.yaml): a known covariate that fully determines the target is used — on a fixture where y = 10*holiday + noise, a model given known_covariates has MASE < 0.5 x the same model without them; future covariates missing for the horizon is an error — predict() without known_covariates for all prediction_length steps is an Err naming the first missing timestamp. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3393.yaml b/docs/roadmaps/entries/PMAT-3393.yaml new file mode 100644 index 0000000000..a8a4d54254 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3393.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3393 + github_issue: 3393 + item_type: task + title: 'P1: Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston — CRUX-O-23' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-23, competitor autogluon, demand 4. ARIMA exists (fixed order; no auto-order search). SeasonalNaive, ETS, Theta and the intermittent-demand family (Croston, ADIDA, IMAPA) are absent. AutoGluon runs these as its ''local'' tier (timeseries/models/local/) and they anchor every leaderboard. FALSIFIABLE (contracts/crux-O-23-v1.yaml): SeasonalNaive is exactly the lag-m copy — forecast[t] == y[t - m] for the whole horizon on a fixture with m=24, byte-equal; AutoARIMA selects the planted order — on a fixture generated from ARIMA(2,1,1) the selected (p,d,q) equals (2,1,1) in >= 9 of 10 seeds; ETS and Theta match the statsforecast reference — point forecasts on the AirPassengers fixture equal the statsforecast reference within 1e-6. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/entries/PMAT-3394.yaml b/docs/roadmaps/entries/PMAT-3394.yaml new file mode 100644 index 0000000000..446d1c9168 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3394.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3394 + github_issue: 3394 + item_type: task + title: 'P1: Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning — CRUX-O-24' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-24, competitor autogluon, demand 4. apr pull fetches LLM checkpoints (Category A) and the inference crates run decoder transformers, but no forecasting head exists: nothing tokenises a numeric series into the model''s input, samples a horizon, and maps it back to quantiles. FALSIFIABLE (contracts/crux-O-24-v1.yaml): zero-shot output matches a pinned reference — a pinned small checkpoint pulled by `apr pull` forecasts the committed fixture within 1e-3 of the reference quantiles; fine-tuning changes the weights and improves the in-domain metric — after fine_tune on the fixture the WQL improves by >= 5% relative AND at least one weight tensor differs from the pulled checkpoint. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index a1513390ae..993652c9ef 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18011,3 +18011,529 @@ roadmap: estimated_effort: null labels: [] notes: null +- id: PMAT-3370 + github_issue: 3370 + item_type: task + title: 'EPIC: CRUX category O — AutoML parity vs AutoGluon 1.6.3 — 9 P0, 9 P1, 6 P2 (24 stories)' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: docs/specifications/crux-competitive-research-ux-workflows.md + acceptance_criteria: + - 'All 24 children (#3371-#3394) closed or explicitly re-cut; apr automl fit/predict on iris and apr forecast fit on a 5-item panel run end to end; each child lands with a contract and mutation-RED evidence.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - epic + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3371 + github_issue: 3371 + item_type: task + title: 'P0: One-call tabular AutoML: fit(label) -> predict on a CSV — CRUX-O-01' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-01, competitor autogluon, demand 5. No predictor-level AutoML entry point exists. crates/aprender-core/src/automl/ is a hyperparameter TUNER (AutoTuner, TPE, GridSearch, RandomSearch, DESearch, TimeBudget) that tunes ONE estimator the caller already chose; nothing takes a labelled table and returns a fitted model. `apr train` is causal-LM pre-training only (crates/apr-cli/src/commands/train.rs:1-5) and `apr finetune --task classify` is text classification. FALSIFIABLE (contracts/crux-O-01-v1.yaml): fit on a labelled CSV returns a predictor whose predictions score above the majority-class baseline — on the iris fixture (crates/aprender-core/src/datasets/iris.csv) accuracy >= 0.90 AND the majority-class baseline is asserted at 0.333 in the same test; the same call works for a regression label without a problem_type argument — a numeric label column yields a regressor whose R^2 on a held-out split exceeds 0.5 AND the mean-predictor baseline is asserted at ~0.0; apr automl fit is reachable from the CLI — `apr automl fit --label class train.csv --out model.apr` exits 0 and writes a loadable artifact; a missing --label exits 2 with a message naming the flag. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3372 + github_issue: 3372 + item_type: task + title: 'P0: Problem-type inference: binary / multiclass / regression / quantile from the label column — CRUX-O-02' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-02, competitor autogluon, demand 5. No function infers a task from a label column. Estimators are chosen by type name (LogisticRegression vs LinearRegression); DataFrame in crates/aprender-core/src/data/mod.rs carries ColumnStats but no label-kind classifier. FALSIFIABLE (contracts/crux-O-02-v1.yaml): two unique label values infer binary, 3..=N small-cardinality infer multiclass, many-unique numeric infers regression — a table of (label column, expected kind) fixtures including the ambiguous cases {0,1} as int, {0.0,1.0} as float, and 30 unique floats over 1000 rows all classify as documented; the override wins and a contradictory override is rejected — problem_type=regression on a string label column is an Err naming the column, not a silent cast. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3373 + github_issue: 3373 + item_type: task + title: 'P0: Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget — CRUX-O-03' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-03, competitor autogluon, demand 5. No preset vocabulary. AutoTuner takes a SearchSpace the caller hand-builds (crates/aprender-core/src/automl/params.rs); there is no named bundle of {models, bagging, stacking, time_limit}. FALSIFIABLE (contracts/crux-O-03-v1.yaml): every preset name resolves to a portfolio and unknown names are rejected with the valid list — the five quality presets each yield a non-empty ordered model list; `presets="bestest"` is an Err whose message contains all five valid names; presets are ordered: a higher preset never fits FEWER model families than the one below it — for medium < good < high < best the family count is monotone non-decreasing, asserted pairwise. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3374 + github_issue: 3374 + item_type: task + title: 'P0: Leaderboard: per-model validation/test score, fit time, predict time and stack level — CRUX-O-04' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-04, competitor autogluon, demand 5. No leaderboard type. GridSearchCVResult in model_selection/ ranks parameter settings of one estimator; TuneResult in automl/tuner.rs is a single best trial. Nothing tabulates several fitted models with timings. FALSIFIABLE (contracts/crux-O-04-v1.yaml): the leaderboard is sorted by validation score descending and its columns are fixed — columns == [model, score_val, score_test?, pred_time_val, fit_time, stack_level, fit_order] and score_val is non-increasing row to row; timings are measured, not defaulted — every fit_time and pred_time_val is > 0 after a real fit; a leaderboard built with no fit has zero rows, not zero timings; apr automl leaderboard prints the same table from a saved artifact — `apr automl leaderboard model.apr --json` emits the rows byte-equal to the in-process leaderboard. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3375 + github_issue: 3375 + item_type: task + title: 'P0: Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) — CRUX-O-05' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-05, competitor autogluon, demand 5. Encoders exist (LabelEncoder, OneHotEncoder, OrdinalEncoder, StandardScaler, PolynomialFeatures in crates/aprender-core/src/preprocessing/) but every one is applied by hand to a column the caller already typed. There is no pass that reads a raw DataFrame, infers each column''s kind, and emits a fitted transform. Datetime expansion and text n-gram features do not exist. TfidfVectorizer exists in text/ but is not wired to a tabular pipeline. FALSIFIABLE (contracts/crux-O-05-v1.yaml): column kinds are inferred from raw values — a fixture CSV with int, float, low-cardinality string, high-cardinality string, ISO datetime and free-text columns is typed as {numeric, numeric, categorical, text, datetime, text} exactly; constant and duplicate columns are dropped and the drop is reported — a column with one unique value and an exact duplicate of another column are both absent from the transformed output AND named in the fit report; the fitted pipeline is deterministic under transform — transform(train) then transform(train) are byte-identical and transform(test) never sees a category unseen at fit as anything but the reserved unknown code. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3376 + github_issue: 3376 + item_type: task + title: 'P1: K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) — CRUX-O-06' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-06, competitor autogluon, demand 4. KFold and StratifiedKFold exist (model_selection/) and cross_validate scores them, but no wrapper trains one child per fold, keeps all children, averages them at predict time and exposes the out-of-fold matrix. RandomForest bags trees internally and is not reusable for other estimators. FALSIFIABLE (contracts/crux-O-06-v1.yaml): OOF predictions cover every training row exactly once — for n rows and k folds the OOF matrix has n rows, no NaN, and each row was predicted by the one child that did not see it (asserted through a fold-id trace); bagged prediction is the mean of the children — predict_proba of the bag equals the elementwise mean of the k children''s predict_proba within 1e-12. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3377 + github_issue: 3377 + item_type: task + title: 'P1: Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) — CRUX-O-07' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-07, competitor autogluon, demand 4. No stacking. crates/aprender-core/src/stack/ is a deployment-health module (StackHealth, InferenceConfig), not a model stacker. ensemble/ holds MixtureOfExperts gating only. FALSIFIABLE (contracts/crux-O-07-v1.yaml): level-2 features are OOF, never in-sample — a mutation that feeds in-sample level-1 predictions to level 2 is detected by the leak test: the L2 holdout score on a pure-noise label rises above chance (asserted RED) while the OOF path stays at chance; dynamic stacking falls back when stacking hurts — on a fixture where L2 holdout score < L1 holdout score, the final model is the L1 ensemble and the decision is recorded in the fit summary. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3378 + github_issue: 3378 + item_type: task + title: 'P0: Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) — CRUX-O-08' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-08, competitor autogluon, demand 5. No ensemble-selection algorithm. MixtureOfExperts learns a gating network (ensemble/moe.rs); nothing performs the forward greedy selection with replacement over base-model validation predictions that yields non-negative weights summing to 1. FALSIFIABLE (contracts/crux-O-08-v1.yaml): weights are a probability vector and the ensemble never scores below its best member — sum(w)=1, all w>=0, and validation metric(ensemble) >= max over members within 1e-9, on 3 fixtures; selection is greedy with replacement and reproducible — with ensemble_size=25 the weight of a member equals its selection count / 25; two runs on the same inputs produce identical weights. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3379 + github_issue: 3379 + item_type: task + title: 'P0: Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share — CRUX-O-09' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-09, competitor autogluon, demand 5. TimeBudget and EarlyStopping exist in automl/tuner.rs but budget ONE tuner. There is no allocation of a global limit across an ordered portfolio, no per-model time share, and no ''skip the rest'' when the budget is exhausted. FALSIFIABLE (contracts/crux-O-09-v1.yaml): the wall-clock of fit never exceeds time_limit by more than the tolerance — with time_limit=5s on a portfolio that would take >60s unconstrained, elapsed <= 5s + 1s AND at least one model was skipped with reason=budget in the summary; the budget is redistributed when a model finishes early — a model that uses 10% of its share returns the remainder to the pool; the next model''s share is asserted larger than the naive equal split. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3380 + github_issue: 3380 + item_type: task + title: 'P1: Predictor-level permutation feature importance with p-values and confidence intervals — CRUX-O-10' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-10, competitor autogluon, demand 4. PermutationImportance exists in crates/aprender-core/src/interpret/ for a single estimator. Missing: the predictor-level call on raw (pre-pipeline) columns, num_shuffle_sets repeats, stddev / p-value / p99 columns, and subsampling. AutoGluon 1.6 also cut this call''s memory 25x (#5645). FALSIFIABLE (contracts/crux-O-10-v1.yaml): importance is reported per raw input column — on a fixture whose datetime column expands to 4 features, the importance table has one row for the datetime column, not four; a pure-noise column has importance statistically indistinguishable from zero — with num_shuffle_sets=10 the noise column''s p-value > 0.05 AND the signal column''s p-value < 0.01 on the same run. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3381 + github_issue: 3381 + item_type: task + title: 'P1: Decision-threshold calibration for binary metrics (calibrate_decision_threshold) — CRUX-O-11' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-11, competitor autogluon, demand 4. Probability calibration exists (PlattScaling, IsotonicRegression, TemperatureScaling in calibration.rs) but nothing searches the decision threshold that maximises f1 / balanced_accuracy / mcc on validation data and stores it on the predictor (`decision_threshold`, `set_decision_threshold`). FALSIFIABLE (contracts/crux-O-11-v1.yaml): the calibrated threshold beats 0.5 on the calibration metric and the gain is asserted, not the search — on an imbalanced fixture (5% positives) f1 at the calibrated threshold > f1 at 0.5 by >= 0.05 absolute; the threshold is persisted with the model — save then load reproduces predict() bit-identically including the threshold; a mutation that resets the threshold to 0.5 on load turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3382 + github_issue: 3382 + item_type: task + title: 'P2: refit_full: retrain the selected models on train+validation after model selection — CRUX-O-12' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-12, competitor autogluon, demand 3. No refit-on-full-data step. cross_validate and grid_search return scores; the model returned is the one fitted on a fold, not on all rows. FALSIFIABLE (contracts/crux-O-12-v1.yaml): the refit model saw every row — the refit estimator''s training-row count equals n_train + n_val, asserted through the fitted row count it reports; refit keeps the selected hyperparameters and drops the bag children — the artifact after refit has one child per selected model and its hyperparameters are byte-equal to the pre-refit winner''s. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3383 + github_issue: 3383 + item_type: task + title: 'P2: Model distillation: compress the ensemble into one fast student — CRUX-O-13' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-13, competitor autogluon, demand 3. Distillation exists only for LLMs in the training crate (knowledge distillation of transformers); nothing trains a single tabular student on the teacher ensemble''s soft labels with data augmentation. FALSIFIABLE (contracts/crux-O-13-v1.yaml): the student is faster and within tolerance of the teacher — student pred_time < 0.25 x teacher pred_time AND student validation score >= teacher score - 0.02 on a fixture; augmentation produces rows the training set does not contain — with augment_method=spunge the student''s training set has > n_train rows and the added rows are not row-equal to any original. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3384 + github_issue: 3384 + item_type: task + title: 'P1: Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file — CRUX-O-14' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-14, competitor autogluon, demand 4. Single estimators serialize to .apr (bundle/, serialization/). No artifact holds a fitted feature pipeline (O-05) plus several models plus ensemble weights plus a decision threshold and loads them as one predictor. Nothing prunes non-selected models from disk. FALSIFIABLE (contracts/crux-O-14-v1.yaml): a deployment clone is smaller and predicts identically — clone_for_deployment artifact bytes < 0.5 x full artifact bytes AND predict() on the clone is bit-identical to predict() on the original for the test fixture; the artifact is self-describing and refuses a schema drift — load() on an artifact whose feature schema disagrees with the input columns is an Err naming the first mismatched column, not a wrong prediction. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3385 + github_issue: 3385 + item_type: task + title: 'P2: Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size) — CRUX-O-15' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-15, competitor autogluon, demand 3. No model or ensemble is ever excluded for being slow at predict time; leaderboard timings (O-04) do not exist to compare against. FALSIFIABLE (contracts/crux-O-15-v1.yaml): the selected ensemble respects the per-row latency limit — with infer_limit=L the measured pred_time per row of the final model <= L AND at least one faster-but-worse model was preferred over a slower-but-better one (asserted on a fixture built to force the trade). Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3386 + github_issue: 3386 + item_type: task + title: 'P2: Fit diagnostics: fit_summary, model_failures and learning curves — CRUX-O-16' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-16, competitor autogluon, demand 3. ProgressCallback in automl/tuner.rs streams trial results; there is no post-fit summary object listing models trained, models failed with their error, per-model hyperparameters and per-iteration validation curves. FALSIFIABLE (contracts/crux-O-16-v1.yaml): a model that raises during fit is recorded, not swallowed and not fatal — a portfolio containing a deliberately failing model finishes; model_failures() has exactly one row naming the model and the error string; the leaderboard omits it; learning curves have one point per boosting/epoch iteration — for a GBM with n_estimators=50 and learning_curves=true the curve has 50 validation points, monotone non-increasing after early-stopping''s best iteration is asserted absent. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3387 + github_issue: 3387 + item_type: task + title: 'P1: Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) — CRUX-O-17' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-17, competitor autogluon, demand 4. No tabular in-context learner. nn/ and the inference crates run causal LMs; nothing consumes (X_train, y_train, X_test) as one context and predicts without gradient steps. AutoGluon 1.4-1.6 added TabPFNv2/2.5/2.6/3, TabICL/v2, Mitra, TabDPT, Nori and made them the extreme preset. FALSIFIABLE (contracts/crux-O-17-v1.yaml): zero-gradient prediction matches a pinned reference within tolerance — loading a pinned small checkpoint via apr pull and predicting a 100-row fixture matches the committed reference probabilities within 1e-4; the context limit is enforced, not silently truncated — a context of rows > the model''s documented max is an Err naming the limit; a mutation that truncates instead turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3388 + github_issue: 3388 + item_type: task + title: 'P2: Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit — CRUX-O-18' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-18, competitor autogluon, demand 3. No estimator reports an expected peak memory before fitting and nothing checks a limit. AutoGluon 1.6 spent 4 PRs on this (#5757, #5768, #5791, GPU budgeting for parallel folds). FALSIFIABLE (contracts/crux-O-18-v1.yaml): a model whose estimate exceeds the limit is skipped with reason=memory — with memory_limit=64MiB on a fixture where the estimate for the largest model is > 64MiB, that model is absent from the leaderboard and present in the skip list with its estimate; the estimate is not a constant — the estimate for a 10x larger fixture is asserted larger than for the base fixture; a mutation returning a constant turns this RED. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3389 + github_issue: 3389 + item_type: task + title: 'P0: Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq — CRUX-O-19' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-19, competitor autogluon, demand 5. crates/aprender-core/src/time_series/mod.rs is one struct, ARIMA, on one f32 series (fit/forecast/order). There is no panel container keyed by item and timestamp, no frequency, no horizon-first API and no per-item forecast. FALSIFIABLE (contracts/crux-O-19-v1.yaml): a panel of N items forecasts N x prediction_length rows — on a fixture of 5 items with 200 hourly points each and prediction_length=24 the output has exactly 120 rows, each (item_id, timestamp) unique, timestamps continuing each item''s last stamp at freq=h; irregular timestamps are rejected or regularised, never silently misaligned — an item with a missing hour is an Err naming the item and gap unless fill=forward is passed, in which case the filled row is flagged. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3390 + github_issue: 3390 + item_type: task + title: 'P0: Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) — CRUX-O-20' + status: planned + priority: critical + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-20, competitor autogluon, demand 5. ARIMA.forecast returns a point path; metrics/probabilistic.rs holds classification-probability metrics, and metrics/regression.rs has no seasonal-scaled (MASE/RMSSE) or quantile (WQL/MQL) losses. FALSIFIABLE (contracts/crux-O-20-v1.yaml): quantile forecasts are monotone in the quantile level — for every (item, step) q0.1 <= q0.5 <= q0.9, asserted over the whole fixture; a mutation that shuffles the quantile columns turns this RED; each metric matches the AutoGluon reference value on a pinned fixture — WQL, MASE, SMAPE, RMSSE, WAPE and MQL computed on a committed (y_true, y_pred, quantiles) fixture equal the reference values produced by autogluon.timeseries.metrics within 1e-9. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3391 + github_issue: 3391 + item_type: task + title: 'P1: Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions — CRUX-O-21' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-21, competitor autogluon, demand 4. No time-aware validation. KFold shuffles rows; nothing cuts the last k horizons of each item as expanding-window validation sets, and there is no API that returns the validation forecasts for inspection. FALSIFIABLE (contracts/crux-O-21-v1.yaml): validation windows never contain a timestamp later than the training cut for that window — for num_val_windows=3 and val_step_size=h, each window''s max training timestamp < min validation timestamp per item, asserted for every item and window; backtest_predictions rows align with backtest_targets — the two frames have identical (window, item_id, timestamp) index sets and no NaN targets. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3392 + github_issue: 3392 + item_type: task + title: 'P2: Known covariates, past covariates and static features in forecasting — CRUX-O-22' + status: planned + priority: medium + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-22, competitor autogluon, demand 3. ARIMA is univariate. No container carries per-item static features or time-varying covariates, and no model consumes them. FALSIFIABLE (contracts/crux-O-22-v1.yaml): a known covariate that fully determines the target is used — on a fixture where y = 10*holiday + noise, a model given known_covariates has MASE < 0.5 x the same model without them; future covariates missing for the horizon is an error — predict() without known_covariates for all prediction_length steps is an Err naming the first missing timestamp. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P2 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3393 + github_issue: 3393 + item_type: task + title: 'P1: Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston — CRUX-O-23' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-23, competitor autogluon, demand 4. ARIMA exists (fixed order; no auto-order search). SeasonalNaive, ETS, Theta and the intermittent-demand family (Croston, ADIDA, IMAPA) are absent. AutoGluon runs these as its ''local'' tier (timeseries/models/local/) and they anchor every leaderboard. FALSIFIABLE (contracts/crux-O-23-v1.yaml): SeasonalNaive is exactly the lag-m copy — forecast[t] == y[t - m] for the whole horizon on a fixture with m=24, byte-equal; AutoARIMA selects the planted order — on a fixture generated from ARIMA(2,1,1) the selected (p,d,q) equals (2,1,1) in >= 9 of 10 seeds; ETS and Theta match the statsforecast reference — point forecasts on the AirPassengers fixture equal the statsforecast reference within 1e-6. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null +- id: PMAT-3394 + github_issue: 3394 + item_type: task + title: 'P1: Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning — CRUX-O-24' + status: planned + priority: high + assigned_to: null + created: 2026-09-16T12:00:00Z + updated: 2026-09-16T12:00:00Z + spec: null + acceptance_criteria: + - 'CRUX-O-24, competitor autogluon, demand 4. apr pull fetches LLM checkpoints (Category A) and the inference crates run decoder transformers, but no forecasting head exists: nothing tokenises a numeric series into the model''s input, samples a horizon, and maps it back to quantiles. FALSIFIABLE (contracts/crux-O-24-v1.yaml): zero-shot output matches a pinned reference — a pinned small checkpoint pulled by `apr pull` forecasts the committed fixture within 1e-3 of the reference quantiles; fine-tuning changes the weights and improves the in-domain metric — after fine_tune on the fixture the WQL improves by >= 5% relative AND at least one weight tensor differs from the pulled checkpoint. Epic #3370.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P1 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/specifications/crux-competitive-research-ux-workflows.md b/docs/specifications/crux-competitive-research-ux-workflows.md index b556c0d96f..5ec6ca6045 100644 --- a/docs/specifications/crux-competitive-research-ux-workflows.md +++ b/docs/specifications/crux-competitive-research-ux-workflows.md @@ -2,8 +2,8 @@ **Subspec ID**: `CRUX-001` **Status**: DRAFT -**Version**: 2.2 (2026-04-21 — Category L [HF kernels-community, 15 stories] and Category M [APR-QA Playbook canonicalization, 10 stories] added; §13 chain-of-thought derivation appended; §3 matrix and §6 coverage recomputed; story total 250 → 275) -**Date**: 2026-04-21 +**Version**: 2.3 (2026-09-16 — Category O [AutoGluon AutoML parity, 24 stories] added, epic aprender#3370; Category N [linfa + burn, 17 stories, aprender#3146, 2026-09-12] exists in the master contract and is recorded here in §3 for the first time; story total in the master registry 267 → 291) +**Date**: 2026-09-16 **Author**: PAIML Engineering **Parent**: [aprender-spec.md](aprender-spec.md), [aprender-monorepo-consolidation.md](aprender-monorepo-consolidation.md) **Master contract**: [`contracts/crux-competitive-research-ux-v1.yaml`](../../contracts/crux-competitive-research-ux-v1.yaml) @@ -93,8 +93,10 @@ larger workflow surface area (HF Transformers covers training + data + hub). | 7 | **Ecosystem interop** | — | 30 | SDKs, MCP, observability, deployment | | 8 | **HF kernels-community** | `get_kernel("kernels-community/")` | 15 | optimized GPU kernels as drop-in `.so` packages (v2.2) | | 9 | **APR-QA Playbook** | `apr qa --gate=` / `apr-model-qa-playbook` | 10 | Popperian falsification framework for model qualification (v2.2) | +| 10 | **linfa + Burn** | `linfa::traits::Fit` / `burn::module::Module` | 17 | Rust-native ML frameworks; substrates (SVD, spatial index, rank-typed tensor) + breadth (Category N, 2026-09-12, aprender#3146) | +| 11 | **AutoGluon** | `TabularPredictor(label).fit()` / `TimeSeriesPredictor.fit()` | 24 | AutoML: one-call tabular fit, presets, leaderboard, bagging/stacking/weighted ensemble, panel forecasting with quantiles (Category O, v2.3, aprender#3370) | -Total = 275 stories. See §5 for the full registry. +Total = 275 stories in this document's §5 as of v2.2; the master registry carries 291 (Categories L and M are contract-only, see §6 note). See §5 for the full registry. (Counts derived from `yq '[.stories[] | .competitor] | ...'` on master contract; drift between this table and the YAML is falsified by FALSIFY-CRUX-010.) @@ -501,7 +503,38 @@ always `contracts/crux-{ID}-v1.yaml` unless noted. | CRUX-M-09 | Property-based falsifier ≥ 1000 fuzz cases per gate | `apr-qa fuzz --cases 1000` | ❌ | 4 | | CRUX-M-10 | Upstream-fix enforcement (reject workarounds; route to aprender/trueno/realizar) | playbook "no-workarounds" rule | 🔨 | 4 | -**Total: 275 stories** across 13 categories; 5 ID gaps (`C-14`, `F-10`, `H-04`, `I-05`, `K-06`) intentional and documented. +### Category O — AutoML Parity, AutoGluon (24 stories) + +> Added v2.3 (2026-09-16). Competitor source: [autogluon/autogluon](https://github.com/autogluon/autogluon) 1.6.3, surveyed from `../autogluon` @ 77946149; evidence in `evidence/crux/autogluon/`. Canonical verb: `TabularPredictor(label="class").fit("train.csv", presets="best")` — the README's only code block. Aprender target surface: `apr automl fit|predict|leaderboard` over a predictor-level AutoML in `aprender-core::automl` (today a single-estimator tuner) and `apr forecast` over a panel `TimeSeriesPredictor` (today one univariate `ARIMA`). Epic aprender#3370; one GitHub issue per row (#3371–#3394). `MultiModalPredictor`, `autogluon.cloud`, MLZero and Ray-parallel fits are CUT on the epic. Not a BEAT pillar. + +| ID | Story | Competitor verb | S | D | +|----|-------|----------------|---|---| +| CRUX-O-01 | One-call tabular AutoML: fit(label) -> predict on a CSV | `TabularPredictor(label="class").fit("train.csv"); predictor.predict("test.csv")` | ❌ | 5 | +| CRUX-O-02 | Problem-type inference: binary / multiclass / regression / quantile from the label column | `predictor.problem_type (inferred in fit unless problem_type= given)` | ❌ | 5 | +| CRUX-O-03 | Quality presets (medium / good / high / best / extreme) that name a model portfolio and a time budget | `fit(..., presets="best_quality")` | ❌ | 5 | +| CRUX-O-04 | Leaderboard: per-model validation/test score, fit time, predict time and stack level | `predictor.leaderboard(test_data, extra_info=True)` | ❌ | 5 | +| CRUX-O-05 | Automatic feature-type inference and the AutoML feature pipeline (numeric, categorical, datetime, text n-gram, drop-unique, drop-duplicate) | `AutoMLPipelineFeatureGenerator` | 🔨 | 5 | +| CRUX-O-06 | K-fold bagging with out-of-fold predictions (num_bag_folds, predict_oof) | `fit(..., num_bag_folds=8); predictor.predict_proba_oof()` | ❌ | 4 | +| CRUX-O-07 | Multi-layer stack ensembling with a leakage guard (num_stack_levels, auto_stack, dynamic_stacking) | `fit(..., num_stack_levels=1, dynamic_stacking="auto")` | ❌ | 4 | +| CRUX-O-08 | Greedy weighted-ensemble selection over fitted models (Caruana ensemble selection) | `fit_weighted_ensemble=True (default)` | ❌ | 5 | +| CRUX-O-09 | Time-budgeted portfolio fit: time_limit split across models, each model early-stopped on its share | `fit(..., time_limit=3600)` | 🔨 | 5 | +| CRUX-O-10 | Predictor-level permutation feature importance with p-values and confidence intervals | `predictor.feature_importance(test_data, num_shuffle_sets=10)` | 🔨 | 4 | +| CRUX-O-11 | Decision-threshold calibration for binary metrics (calibrate_decision_threshold) | `fit(..., calibrate_decision_threshold="auto"); predictor.calibrate_decision_threshold(metric="f1")` | 🔨 | 4 | +| CRUX-O-12 | refit_full: retrain the selected models on train+validation after model selection | `fit(..., refit_full=True, set_best_to_refit_full=True); predictor.refit_full()` | ❌ | 3 | +| CRUX-O-13 | Model distillation: compress the ensemble into one fast student | `predictor.distill(time_limit=..., augment_method="spunge")` | ❌ | 3 | +| CRUX-O-14 | Deployment artifact: clone_for_deployment / keep_only_best / save_space / persist into one loadable file | `predictor.clone_for_deployment(path); predictor.persist()` | ❌ | 4 | +| CRUX-O-15 | Inference-latency constraint during model selection (infer_limit, infer_limit_batch_size) | `fit(..., infer_limit=0.001, infer_limit_batch_size=10000)` | ❌ | 3 | +| CRUX-O-16 | Fit diagnostics: fit_summary, model_failures and learning curves | `predictor.fit_summary(); predictor.model_failures(); fit(..., learning_curves=True)` | ❌ | 3 | +| CRUX-O-17 | Tabular foundation model: in-context prediction with a pretrained transformer (TabPFN / TabICL / Mitra class) | `hyperparameters={"TABPFNV2": {}, "TABICL": {}, "MITRA": {}}` | ❌ | 4 | +| CRUX-O-18 | Memory-aware fit: per-model memory estimate and a memory_limit that skips models that would not fit | `fit(..., memory_limit="auto")` | ❌ | 3 | +| CRUX-O-19 | Multi-series forecasting predictor: (item_id, timestamp) panel data, prediction_length, freq | `TimeSeriesPredictor(prediction_length=48, freq="h").fit(TimeSeriesDataFrame)` | ❌ | 5 | +| CRUX-O-20 | Probabilistic forecasts: quantile_levels and the forecasting metric family (WQL, MQL, MASE, SMAPE, RMSSE, WAPE) | `TimeSeriesPredictor(eval_metric="WQL", quantile_levels=[0.1,0.5,0.9])` | ❌ | 5 | +| CRUX-O-21 | Rolling-window backtesting: num_val_windows, refit_every_n_windows, backtest_predictions | `fit(..., num_val_windows="auto", refit_every_n_windows="auto"); predictor.backtest_predictions()` | ❌ | 4 | +| CRUX-O-22 | Known covariates, past covariates and static features in forecasting | `TimeSeriesPredictor(known_covariates_names=["holiday"]); train_data.static_features = df` | ❌ | 3 | +| CRUX-O-23 | Local statistical baselines: SeasonalNaive, ETS, Theta, AutoARIMA, Croston | `hyperparameters={"SeasonalNaive": {}, "AutoETS": {}, "Theta": {}, "AutoARIMA": {}, "Croston": {}}` | 🔨 | 4 | +| CRUX-O-24 | Zero-shot pretrained forecaster (Chronos-2 / Toto-2 class) loaded by apr pull, with optional fine-tuning | `hyperparameters={"Chronos2": {"fine_tune": True}}` | ❌ | 4 | + +**Total: 275 stories in §5 (v2.2) + 24 in Category O = 299 documented rows; the master registry holds 291** (Categories L and M have contracts but no registry rows — pre-existing drift recorded on aprender#3146) across 15 categories; 5 ID gaps (`C-14`, `F-10`, `H-04`, `I-05`, `K-06`) intentional and documented. --- @@ -517,6 +550,8 @@ Counts verified from §5 table (via `awk` emoji extraction). Δ columns show v2. | 🤔 unclear | 0 | 0 | 0.0 % | — | | **total** | **275** | +25 | 100 % | | +> **v2.3 note (2026-09-16):** the master contract's `coverage_intake` is now asserted from `stories[]` by the aprender-contracts test-suite and reads supported 43 / partial 77 / missing 171 / total 291 after Category O (+5 partial, +19 missing). This table is the v2.2 intake and is left as history; the YAML is the source of truth. + Demand-weighted view — **high-demand (D≥4)** stories still ❌ missing are the fast path to adoption parity and become the first `pmat work` items (see §12). Exact D-tier counts are regenerated by the falsification harness diff --git a/evidence/crux/autogluon/README.md b/evidence/crux/autogluon/README.md new file mode 100644 index 0000000000..81e97763b9 --- /dev/null +++ b/evidence/crux/autogluon/README.md @@ -0,0 +1,3 @@ +# evidence/crux/autogluon + +CRUX Category O evidence: AutoGluon 1.6.3 (`../autogluon` @ 77946149), surveyed 2026-09-16. See `api-surface.md` (the survey), `readme-verbs.txt` (fold-ranked verbs), `hello.sh` (canonical flow). Epic aprender#3370. diff --git a/evidence/crux/autogluon/api-surface.md b/evidence/crux/autogluon/api-surface.md new file mode 100644 index 0000000000..2e74a185c8 --- /dev/null +++ b/evidence/crux/autogluon/api-surface.md @@ -0,0 +1,46 @@ +# AutoGluon 1.6.3 — user-facing surface (evidence for CRUX Category O) + +Surveyed 2026-09-16 from `../autogluon` @ 77946149 (`VERSION` = 1.6.3). Paths are repo-relative to the autogluon checkout. + +## TabularPredictor (`tabular/src/autogluon/tabular/predictor/predictor.py`) + +Public methods (65): fit, fit_extra, fit_pseudolabel, predict, predict_proba, predict_from_proba, evaluate, evaluate_predictions, leaderboard, learning_curves, model_failures, predict_multi, predict_proba_multi, fit_summary, transform_features, transform_labels, feature_importance, compile, persist, unpersist, refit_full, model_best, set_model_best, model_refit_map, info, model_info, model_hyperparameters, fit_weighted_ensemble, calibrate_decision_threshold, set_decision_threshold, predict_oof, predict_proba_oof, save_space, delete_models, disk_usage, model_names, distill, plot_ensemble_model, save, load, load_log, clone, clone_for_deployment, simulation_artifact, confusion_matrix, plus properties (problem_type, eval_metric, decision_threshold, feature_metadata, class_labels, positive_class, quantile_levels). + +`fit` kwargs that carry a story: presets, time_limit, hyperparameters, num_bag_folds, num_bag_sets, num_stack_levels, auto_stack, dynamic_stacking, fit_weighted_ensemble, refit_full, set_best_to_refit_full, save_bag_folds, keep_only_best, holdout_frac, use_bag_holdout, infer_limit, infer_limit_batch_size, calibrate_decision_threshold, learning_curves, memory_limit, num_cpus, num_gpus, fit_strategy, feature_generator, excluded_model_types, included_model_types, raise_on_no_models_fitted, callbacks, core_kwargs/aux_kwargs (1.6). + +Presets (`tabular/src/autogluon/tabular/configs/presets_configs.py`): extreme_quality (zeroshot portfolio, 8 bag folds, foundation models), best_quality (auto_stack + dynamic_stacking), high_quality (+ refit_full, no bag folds saved), good_quality (light portfolio), medium_quality (no bagging), optimize_for_deployment (keep_only_best + save_space), ignore_text, ignore_text_ngrams, interpretable, noncommercial, tabarena. Portfolios: `configs/zeroshot/zeroshot_portfolio_{2023,2025,cpu_2025_12_18,gpu_2025_12_18,commercial_2026_08_05,noncommercial_2026_08_05}.py`. + +Model families (`tabular/src/autogluon/tabular/models/`): catboost, ebm, fastainn, imodels, knn, lgb, lr, mitra, nori, realmlp, rf, tabdpt, tabicl, tabm, tabpfnmix, tabpfnv2, tabprep, tabular_nn, xgboost, xt (+ automm, image_prediction, text_prediction wrappers). The 2026 commercial portfolio names CAT, GBM, XGB, MITRA, TABICL, TABM. + +Feature pipeline (`features/src/autogluon/features/generators/`): auto_ml_pipeline (enable_numeric/categorical/datetime/text_special/text_ngram/raw_text/vision features), astype, binned, category, cat_int, datetime, drop_duplicates, drop_unique, fillna, frequency, groupby, isnan, label_encoder, one_hot_encoder, oof_target_encoder, text_ngram, text_special, memory_minimize, skrub, rsfc, selection. + +## TimeSeriesPredictor (`timeseries/src/autogluon/timeseries/predictor.py`) + +Constructor: target, known_covariates_names, prediction_length, freq, eval_metric, eval_metric_seasonal_period, horizon_weight, quantile_levels, cache_predictions (deprecated 1.6), log_to_file. +fit: train_data (TimeSeriesDataFrame with item_id/timestamp index + static_features), tuning_data, time_limit, presets, hyperparameters, hyperparameter_tune_kwargs, excluded_model_types, ensemble_hyperparameters, num_val_windows ("auto" since 1.5), val_step_size, refit_every_n_windows ("auto"), refit_full, enable_ensemble, skip_model_selection, random_seed. +Methods: predict, backtest_predictions, backtest_targets (1.5), evaluate, feature_importance, leaderboard, fit_summary, refit_full, persist/unpersist, export_model (1.6: standalone checkpoint), update (1.6 experimental: ensemble re-selection), make_future_data_frame, plot. +Metrics (`timeseries/.../metrics/point.py`, `quantile.py`): MQL WQL SQL RMSE MSE MAE MAEB BIAS WAPE WAPEB SMAPE MAPE MASE RMSSE RMSLE WCD (MAEB/WAPEB/BIAS/MQL new in 1.6). +Models: local (Naive, SeasonalNaive, Average, SeasonalAverage, NPTS, Zero; statsforecast AutoARIMA/ARIMA/AutoETS/ETS/AutoCES/Theta/DynamicOptimizedTheta/Croston/ADIDA/IMAPA), gluonts (DeepAR, SimpleFeedForward, TFT, DLinear, PatchTST, WaveNet, TiDE), pretrained (Chronos, Chronos2 with LoRA/full fine-tune, Toto, Toto2), tabular (per-step, recursive/direct via mlforecast), ensembles (greedy selection, per-item greedy, weighted, array-based, multi-layer since 1.5). + +## MultiModalPredictor (`multimodal/src/autogluon/multimodal/predictor.py`) — CUT on epic #3370 + +Problem types: classification, regression, few_shot_classification, object_detection, ner / named_entity_recognition, image/text/image_text similarity (matching), semantic_segmentation, zero_shot_image_classification, document classification. Methods: fit, predict, predict_proba, evaluate, extract_embedding, export_onnx, optimize_for_inference, dump_model, list_supported_models. + +## Release headlines used for demand scoring + +- 1.4 (2025): extreme preset; TabPFNv2, TabICL, TabM, RealMLP; Mitra; MLZero (AutoGluon Assistant). +- 1.5: Chronos-2 with zero-shot + fine-tuning; item-level and multi-layer forecast ensembles; `num_val_windows="auto"`, `backtest_predictions`; RealTabPFN-2/2.5, TabDPT, TabPrep-LightGBM, EBM; new CPU/GPU portfolios; TabArena SOTA. +- 1.6: Nori, TabPFN-3, TabDPT-Turbo, TabPFN-2.6, TabICLv2; Toto-2; MAEB/WAPEB/BIAS/MQL metrics; `TimeSeriesPredictor.export_model`; `update()`; calibrated CPU/GPU memory estimates; GPU-aware parallel bagging; feature-importance memory cut 25x; params immutable after construction (deprecation). +- Docs index (`docs/tutorials/{tabular,timeseries,multimodal,cloud_fit_deploy}/index.md`): tabular quick start → essentials → in-depth → feature engineering → foundational models → multimodal → FAQ; timeseries quick start → in-depth → Chronos → ensembles → metrics → model zoo → FAQ. + +## What aprender has today (eb262f8eb), for the S column + +- `crates/aprender-core/src/automl/`: AutoTuner, SearchSpace, TPE, GridSearch, RandomSearch, DESearch, ActiveLearningSearch, TimeBudget, EarlyStopping, ProgressCallback — tunes ONE chosen estimator. +- `model_selection/`: KFold, StratifiedKFold, cross_validate, cross_val_score, grid_search, randomized_search, train_test_split. +- `preprocessing/`: LabelEncoder, OneHotEncoder, OrdinalEncoder, Standard/MinMax/MaxAbs/Robust scalers, Normalizer, PolynomialFeatures, PCA, TSNE — all applied by hand, no type inference. +- `tree/`: DecisionTree{Classifier,Regressor}, RandomForest{Classifier,Regressor}, GradientBoostingClassifier (no regressor). +- `calibration.rs`: PlattScaling, IsotonicRegression, TemperatureScaling, ECE/MCE/Brier. No decision-threshold search. +- `interpret/`, `explainable/`: ShapExplainer, LIME, PermutationImportance, IntegratedGradients, CounterfactualExplainer — per estimator. +- `time_series/`: `ARIMA` only (fit/forecast/order), single f32 series. `metrics/`: no MASE/RMSSE/WQL. +- `ensemble/`: MixtureOfExperts + SoftmaxGating. `stack/`: deployment health, not model stacking. +- `apr train` = causal-LM pre-training; `apr finetune --task classify` = text classification. No `apr automl`, no `apr forecast`. diff --git a/evidence/crux/autogluon/hello.sh b/evidence/crux/autogluon/hello.sh new file mode 100644 index 0000000000..7b1662298a --- /dev/null +++ b/evidence/crux/autogluon/hello.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# AutoGluon canonical flow, transcribed from README.md + tabular quick start (1.6.3). +# Run with: uv run --with autogluon.tabular python - <<'PY' +set -euo pipefail +python - <<'PY' +from autogluon.tabular import TabularDataset, TabularPredictor +train = TabularDataset("https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv") +test = TabularDataset("https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv") +predictor = TabularPredictor(label="class").fit(train, presets="medium_quality", time_limit=120) +print(predictor.problem_type) # inferred: binary +print(predictor.leaderboard(test)) # model, score_test, score_val, pred_time_*, fit_time, stack_level +print(predictor.feature_importance(test)) # permutation importance on raw columns +predictor.clone_for_deployment("deploy/") # keep_only_best + save_space +PY diff --git a/evidence/crux/autogluon/readme-verbs.txt b/evidence/crux/autogluon/readme-verbs.txt new file mode 100644 index 0000000000..248b128e2b --- /dev/null +++ b/evidence/crux/autogluon/readme-verbs.txt @@ -0,0 +1,23 @@ +# AutoGluon 1.6.3 README verbs, ranked by fold position (../autogluon/README.md, 2026-09-16) +# fold 1 — the only code block in the README: +pip install autogluon +TabularPredictor(label="class").fit("train.csv", presets="best") +predictor.predict("test.csv") +# fold 2 — docs/tutorials/tabular/tabular-quick-start.ipynb call order: +TabularPredictor(label).fit(train_data, time_limit=...) +predictor.predict(test_data) +predictor.evaluate(test_data) +predictor.leaderboard(test_data) +predictor.feature_importance(test_data) +# fold 3 — timeseries quick start: +TimeSeriesPredictor(prediction_length=48, eval_metric="WQL").fit(TimeSeriesDataFrame, presets="medium_quality") +predictor.predict(train_data) +predictor.leaderboard(test_data) +# tabular presets (tabular/src/autogluon/tabular/configs/presets_configs.py): +extreme_quality best_quality high_quality good_quality medium_quality optimize_for_deployment ignore_text ignore_text_ngrams interpretable noncommercial tabarena +# tabular model families (tabular/src/autogluon/tabular/models/): +catboost ebm fastainn imodels knn lgb lr mitra nori realmlp rf tabdpt tabicl tabm tabpfnmix tabpfnv2 tabprep tabular_nn xgboost xt (+ automm/image/text wrappers) +# timeseries models (timeseries/src/autogluon/timeseries/models/): +Naive SeasonalNaive Average SeasonalAverage NPTS Zero | AutoARIMA ARIMA AutoETS ETS AutoCES DynamicOptimizedTheta Theta Croston ADIDA IMAPA | DeepAR SimpleFeedForward TemporalFusionTransformer DLinear PatchTST WaveNet TiDE | Chronos Chronos2 Toto Toto2 | PerStepTabular RecursiveTabular DirectTabular | ensembles: greedy selection, per-item greedy, weighted, array-based +# timeseries metrics (timeseries/src/autogluon/timeseries/metrics/): +MQL WQL SQL RMSE MSE MAE MAEB BIAS WAPE WAPEB SMAPE MAPE MASE RMSSE RMSLE WCD diff --git a/scripts/crux_scaffold_contracts.py b/scripts/crux_scaffold_contracts.py index c4d66c8c52..f14819e546 100755 --- a/scripts/crux_scaffold_contracts.py +++ b/scripts/crux_scaffold_contracts.py @@ -35,6 +35,7 @@ "L": "HF kernels-community integration", "M": "APR-QA Playbook Canonicalization", "N": "Rust ML Framework Parity", + "O": "AutoML Parity", } STATUS_BADGE = { From 68c9e39bd271d50c0e5bffe924b407dfb402a237 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Wed, 16 Sep 2026 17:30:50 +0200 Subject: [PATCH 2/6] =?UTF-8?q?chore(crux):=20bind=20the=20admission=20PR?= =?UTF-8?q?=20to=20its=20own=20ticket=20PMAT-3401=20=E2=80=94=20the=20epic?= =?UTF-8?q?'s=20acceptance=20criteria=20describe=20the=20programme,=20not?= =?UTF-8?q?=20this=20diff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2 of the quorum read the epic (PMAT-3370) as the ticket and refused the admission for not implementing the 24 stories. The admission is its own unit of work with its own done-when; this fragment says so. Closes #3401 Pmat-Ticket: PMAT-3401 Co-Authored-By: Claude Fable 5.1 --- docs/roadmaps/entries/PMAT-3401.yaml | 21 +++++++++++++++++++++ docs/roadmaps/roadmap.yaml | 21 +++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 docs/roadmaps/entries/PMAT-3401.yaml diff --git a/docs/roadmaps/entries/PMAT-3401.yaml b/docs/roadmaps/entries/PMAT-3401.yaml new file mode 100644 index 0000000000..1f71ff6e25 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3401.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3401 + github_issue: 3401 + item_type: task + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 + intake tests, 24 master-registry rows (intake 267->291), spec v2.3 §3/§5/§6, evidence/crux/autogluon/, 25 roadmap fragments, README count regen — child of #3370' + status: in_progress + priority: critical + assigned_to: null + created: 2026-09-16T16:30:00Z + updated: 2026-09-16T16:30:00Z + spec: docs/specifications/crux-competitive-research-ux-workflows.md + acceptance_criteria: + - 'PR #3395 merges carrying exactly: 24 draft contracts with LIVE-PENDING gates and no implementation; CRUX_COMPETITORS 14->15 with the mutation proof; 24 registry rows and coverage_intake 267->291; spec v2.3; evidence/crux/autogluon/; 25 roadmap fragments; README CONTRACT_COUNT regenerated. Absent on purpose: library code, CLI wiring, test bodies.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 993652c9ef..31c0eb52ae 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18537,3 +18537,24 @@ roadmap: - crux - pareto-autogluon notes: null +- id: PMAT-3401 + github_issue: 3401 + item_type: task + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 + intake tests, 24 master-registry rows (intake 267->291), spec v2.3 §3/§5/§6, evidence/crux/autogluon/, 25 roadmap fragments, README count regen — child of #3370' + status: in_progress + priority: critical + assigned_to: null + created: 2026-09-16T16:30:00Z + updated: 2026-09-16T16:30:00Z + spec: docs/specifications/crux-competitive-research-ux-workflows.md + acceptance_criteria: + - 'PR #3395 merges carrying exactly: 24 draft contracts with LIVE-PENDING gates and no implementation; CRUX_COMPETITORS 14->15 with the mutation proof; 24 registry rows and coverage_intake 267->291; spec v2.3; evidence/crux/autogluon/; 25 roadmap fragments; README CONTRACT_COUNT regenerated. Absent on purpose: library code, CLI wiring, test bodies.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - P0 + - enhancement + - crux + - pareto-autogluon + notes: null From 48c8cf4f825529bd3696cfb89351810bda0d805b Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Wed, 16 Sep 2026 17:40:44 +0200 Subject: [PATCH 3/6] =?UTF-8?q?chore(crux):=20PMAT-3401=20title=20inventor?= =?UTF-8?q?ies=20the=20diff=20exactly=20=E2=80=94=2026=20fragments=20(its?= =?UTF-8?q?=20own=20included)=20and=20the=20scaffold=20CATEGORY=5FNAMES=20?= =?UTF-8?q?edit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quorum round 3 lane 1 refused on two literal mismatches between the ticket and the diff: '25 roadmap fragments' (there are 26 once this ticket's own fragment lands) and an unlisted edit to scripts/crux_scaffold_contracts.py. The title now lists every path. Pmat-Ticket: PMAT-3401 Co-Authored-By: Claude Fable 5.1 --- docs/roadmaps/entries/PMAT-3401.yaml | 4 ++-- docs/roadmaps/roadmap.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/roadmaps/entries/PMAT-3401.yaml b/docs/roadmaps/entries/PMAT-3401.yaml index 1f71ff6e25..d50a28c7d4 100644 --- a/docs/roadmaps/entries/PMAT-3401.yaml +++ b/docs/roadmaps/entries/PMAT-3401.yaml @@ -1,7 +1,7 @@ - id: PMAT-3401 github_issue: 3401 item_type: task - title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 + intake tests, 24 master-registry rows (intake 267->291), spec v2.3 §3/§5/§6, evidence/crux/autogluon/, 25 roadmap fragments, README count regen — child of #3370' + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows (coverage_intake 267->291) + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, README CONTRACT_COUNT 1842->1866 — child of #3370' status: in_progress priority: critical assigned_to: null @@ -9,7 +9,7 @@ updated: 2026-09-16T16:30:00Z spec: docs/specifications/crux-competitive-research-ux-workflows.md acceptance_criteria: - - 'PR #3395 merges carrying exactly: 24 draft contracts with LIVE-PENDING gates and no implementation; CRUX_COMPETITORS 14->15 with the mutation proof; 24 registry rows and coverage_intake 267->291; spec v2.3; evidence/crux/autogluon/; 25 roadmap fragments; README CONTRACT_COUNT regenerated. Absent on purpose: library code, CLI wiring, test bodies.' + - 'Exactly this inventory, nothing more: contracts/crux-O-01..24-v1.yaml (24 files, registry: false, every gate LIVE-PENDING); crates/aprender-contracts/src/schema/validator.rs (CRUX_COMPETITORS 14->15, + autogluon with rationale) and crux_intake_tests.rs (autogluon in the corpus-vocabulary list and in the not-a-BEAT-pillar list); scripts/crux_scaffold_contracts.py (CATEGORY_NAMES O); contracts/crux-competitive-research-ux-v1.yaml (24 stories rows, coverage_intake 43/77/171/291, evidence_sources.autogluon, autogluon reference); docs/specifications/crux-competitive-research-ux-workflows.md (v2.3 header, §3 rows for N and O, §5 Category O table, §6 note); evidence/crux/autogluon/{README.md,api-surface.md,hello.sh,readme-verbs.txt}; docs/roadmaps/entries/PMAT-3370.yaml + PMAT-3371..3394.yaml + PMAT-3401.yaml (26 fragments) and docs/roadmaps/roadmap.yaml regenerated by the aggregator; README.md CONTRACT_COUNT 1842->1866. Absent on purpose: library code, apr CLI wiring, test bodies — those are the 24 children of #3370.' phases: [] subtasks: [] estimated_effort: null diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 31c0eb52ae..49f1799e76 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18540,7 +18540,7 @@ roadmap: - id: PMAT-3401 github_issue: 3401 item_type: task - title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 + intake tests, 24 master-registry rows (intake 267->291), spec v2.3 §3/§5/§6, evidence/crux/autogluon/, 25 roadmap fragments, README count regen — child of #3370' + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows (coverage_intake 267->291) + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, README CONTRACT_COUNT 1842->1866 — child of #3370' status: in_progress priority: critical assigned_to: null @@ -18548,7 +18548,7 @@ roadmap: updated: 2026-09-16T16:30:00Z spec: docs/specifications/crux-competitive-research-ux-workflows.md acceptance_criteria: - - 'PR #3395 merges carrying exactly: 24 draft contracts with LIVE-PENDING gates and no implementation; CRUX_COMPETITORS 14->15 with the mutation proof; 24 registry rows and coverage_intake 267->291; spec v2.3; evidence/crux/autogluon/; 25 roadmap fragments; README CONTRACT_COUNT regenerated. Absent on purpose: library code, CLI wiring, test bodies.' + - 'Exactly this inventory, nothing more: contracts/crux-O-01..24-v1.yaml (24 files, registry: false, every gate LIVE-PENDING); crates/aprender-contracts/src/schema/validator.rs (CRUX_COMPETITORS 14->15, + autogluon with rationale) and crux_intake_tests.rs (autogluon in the corpus-vocabulary list and in the not-a-BEAT-pillar list); scripts/crux_scaffold_contracts.py (CATEGORY_NAMES O); contracts/crux-competitive-research-ux-v1.yaml (24 stories rows, coverage_intake 43/77/171/291, evidence_sources.autogluon, autogluon reference); docs/specifications/crux-competitive-research-ux-workflows.md (v2.3 header, §3 rows for N and O, §5 Category O table, §6 note); evidence/crux/autogluon/{README.md,api-surface.md,hello.sh,readme-verbs.txt}; docs/roadmaps/entries/PMAT-3370.yaml + PMAT-3371..3394.yaml + PMAT-3401.yaml (26 fragments) and docs/roadmaps/roadmap.yaml regenerated by the aggregator; README.md CONTRACT_COUNT 1842->1866. Absent on purpose: library code, apr CLI wiring, test bodies — those are the 24 children of #3370.' phases: [] subtasks: [] estimated_effort: null From de4b4c108ac818c39a230d308497dded16851c4f Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 03:37:00 +0200 Subject: [PATCH 4/6] =?UTF-8?q?PMAT-3401=20(adoption):=20the=20three=20der?= =?UTF-8?q?ivatives=2024=20new=20contracts=20oblige=20=E2=80=94=20census,?= =?UTF-8?q?=20graph,=20README=20=E2=80=94=20regenerated=20together?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR adds 24 contracts and regenerated none of the tracked artifacts derived from the corpus. On #3581 that omission surfaced one per CI round, each masked by the one before it. All three here, at once, with a pv built from this tree under a pinned target dir: contracts/census.json 1800 -> 1824 (+24, the contracts added) contracts/contracts.nt 15,600 -> 15,696 triples (+96 = 24 x 4; GREW — a drop is the tell for a malformed input or a stale binary; binding.yaml still parses, 156 entries) README CONTRACT_COUNT 2 blocks -> 1824 via make readme-sync The merge took main's generated README blocks over the branch's hand-typed 1866, then readme-sync wrote the measured 1824; the branch's number described a tree that never existed on main. Verified: all 24 contracts pv-validate under the pinned binary (control: main's crux-A-01 valid under the same one); lint_passes_on_real_contracts green, so the sigma prose ratchet holds; 1666 engine tests; ont4b shapes gate 11/11; test-binding ratchet; readme-sync-check; FALSIFY-README-002; roadmap additive added=26 deleted=0, aggregate idempotent, 26 fragments present. Refs #3401 Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 4 +- contracts/census.json | 10 ++--- contracts/contracts.nt | 96 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 182bfb024b..a0cd3bbcde 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ publishing — all backed by YAML provable contracts that fail CI on drift. | Metric | Count | Source of truth | |-------:|------:|---| | Workspace crates | **79** workspace crates | `cargo metadata --no-deps` (NOT `ls crates/` — 4 are `exclude`d, 1 has no Cargo.toml) | -| Provable contracts | **1800** provable contracts | `contracts/census.json` `.n_files` — the set `pv lint` walks (`pv census`, ONT-001 ONT-1; regenerated by `make contracts`, written by `make readme-sync`, guarded by `scripts/check_readme_claims.sh`) | +| Provable contracts | **1824** provable contracts | `contracts/census.json` `.n_files` — the set `pv lint` walks (`pv census`, ONT-001 ONT-1; regenerated by `make contracts`, written by `make readme-sync`, guarded by `scripts/check_readme_claims.sh`) | | CLI commands | **110** CLI commands | `apr --help` | | Book CLI chapters | **112** chapters | `ls book/src/cli/*.md` | | Book lib chapters | **71** chapters | `ls book/src/lib/*.md` (parity with `pub mod`) | @@ -262,7 +262,7 @@ falsification_tests: prediction: apr validate bad-model.apr exits non-zero ``` -The tree carries 1800 contracts across inference, training, quantization, attention, FFN, +The tree carries 1824 contracts across inference, training, quantization, attention, FFN, tokenization, model formats, CLI safety — and this README itself. ## Migration from old crates diff --git a/contracts/census.json b/contracts/census.json index d6286d1147..aa1f2e7d3a 100644 --- a/contracts/census.json +++ b/contracts/census.json @@ -1,15 +1,15 @@ { "schema": "ont.paiml.dev/census/v1alpha1", "git_sha": null, - "n_files": 1800, - "n_parsed": 1800, + "n_files": 1824, + "n_parsed": 1824, "n_parse_errors": 0, "parse_errors": [], "quarantined_n": 0, "by_kind": { "beat-benchmark": 24, "corpus-assembly": 1, - "kernel": 363, + "kernel": 387, "model-family": 28, "model-family-variant": 1, "pattern": 86, @@ -25,11 +25,11 @@ "pv-contract": 1 }, "by_anchoring": { - "unanchored": 1797, + "unanchored": 1821, "class": 2, "instance": 1 }, - "id_set_sha256": "51cfc1e6cf27aa5cb6d9345e918a19863f13aa4728b4eba15bfa69bc6002bfb1", + "id_set_sha256": "42f8263b0336e31bce50567710557bd0c2335106c01a7c8b28e147b9a807e18a", "declared_external": [ { "name": "provable-contracts", diff --git a/contracts/contracts.nt b/contracts/contracts.nt index 7990b9e038..cb63ad65e8 100644 --- a/contracts/contracts.nt +++ b/contracts/contracts.nt @@ -7393,6 +7393,102 @@ . "contracts/crux-N-17-v1.yaml"^^ . "crux-N-17-v1"^^ . + . + . + "contracts/crux-O-01-v1.yaml"^^ . + "crux-O-01-v1"^^ . + . + . + "contracts/crux-O-02-v1.yaml"^^ . + "crux-O-02-v1"^^ . + . + . + "contracts/crux-O-03-v1.yaml"^^ . + "crux-O-03-v1"^^ . + . + . + "contracts/crux-O-04-v1.yaml"^^ . + "crux-O-04-v1"^^ . + . + . + "contracts/crux-O-05-v1.yaml"^^ . + "crux-O-05-v1"^^ . + . + . + "contracts/crux-O-06-v1.yaml"^^ . + "crux-O-06-v1"^^ . + . + . + "contracts/crux-O-07-v1.yaml"^^ . + "crux-O-07-v1"^^ . + . + . + "contracts/crux-O-08-v1.yaml"^^ . + "crux-O-08-v1"^^ . + . + . + "contracts/crux-O-09-v1.yaml"^^ . + "crux-O-09-v1"^^ . + . + . + "contracts/crux-O-10-v1.yaml"^^ . + "crux-O-10-v1"^^ . + . + . + "contracts/crux-O-11-v1.yaml"^^ . + "crux-O-11-v1"^^ . + . + . + "contracts/crux-O-12-v1.yaml"^^ . + "crux-O-12-v1"^^ . + . + . + "contracts/crux-O-13-v1.yaml"^^ . + "crux-O-13-v1"^^ . + . + . + "contracts/crux-O-14-v1.yaml"^^ . + "crux-O-14-v1"^^ . + . + . + "contracts/crux-O-15-v1.yaml"^^ . + "crux-O-15-v1"^^ . + . + . + "contracts/crux-O-16-v1.yaml"^^ . + "crux-O-16-v1"^^ . + . + . + "contracts/crux-O-17-v1.yaml"^^ . + "crux-O-17-v1"^^ . + . + . + "contracts/crux-O-18-v1.yaml"^^ . + "crux-O-18-v1"^^ . + . + . + "contracts/crux-O-19-v1.yaml"^^ . + "crux-O-19-v1"^^ . + . + . + "contracts/crux-O-20-v1.yaml"^^ . + "crux-O-20-v1"^^ . + . + . + "contracts/crux-O-21-v1.yaml"^^ . + "crux-O-21-v1"^^ . + . + . + "contracts/crux-O-22-v1.yaml"^^ . + "crux-O-22-v1"^^ . + . + . + "contracts/crux-O-23-v1.yaml"^^ . + "crux-O-23-v1"^^ . + . + . + "contracts/crux-O-24-v1.yaml"^^ . + "crux-O-24-v1"^^ . . . "contracts/crux-competitive-research-ux-v1.yaml"^^ . From 9f59bebb65d51481fef673857209e3e32063a16c Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 03:52:09 +0200 Subject: [PATCH 5/6] PMAT-3401 (adoption): the row states invariants, not the counts of a tree that no longer exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Quorum round 0 on de4b4c108 was 3/3 FAIL, every lane on the same true point: this row's title pinned "README CONTRACT_COUNT 1842->1866" (and "coverage_intake 267->291") from the 2026-09-16 tree; main moved, the merged tree is 1800 -> 1824, and the lanes correctly held the diff to its own ticket's literal numbers. Cop prompt §6 1e: never pin a count the diff may legitimately exceed. The title now names the change without the drifted numbers, and notes: states each acceptance as an invariant against the merged tree — census = main + 24, contracts.nt = fresh extract, README = census — and lists the three derivatives as part of the inventory. Lane 1's second finding is recorded in the notes as known and out of scope: the census files the 24 drafts under kind kernel (no metadata.kind, registry:false), where main's crux contracts carry registry:true and file under registry. A draft/intake bucket is a census question, not this PR's. Refs #3401, #3569 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3401.yaml | 4 ++-- docs/roadmaps/roadmap.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/roadmaps/entries/PMAT-3401.yaml b/docs/roadmaps/entries/PMAT-3401.yaml index d50a28c7d4..3df5483e33 100644 --- a/docs/roadmaps/entries/PMAT-3401.yaml +++ b/docs/roadmaps/entries/PMAT-3401.yaml @@ -1,7 +1,7 @@ - id: PMAT-3401 github_issue: 3401 item_type: task - title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows (coverage_intake 267->291) + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, README CONTRACT_COUNT 1842->1866 — child of #3370' + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, and the three corpus derivatives (census, contracts.nt, README CONTRACT_COUNT) regenerated to the merged tree — child of #3370' status: in_progress priority: critical assigned_to: null @@ -18,4 +18,4 @@ - enhancement - crux - pareto-autogluon - notes: null + notes: 'ACCEPTANCE, stated as invariants and not as counts (the first version of this row pinned "README CONTRACT_COUNT 1842->1866" and "coverage_intake 267->291" from the 2026-09-16 tree; main moved and the numbers described a tree that no longer existed, which is what the AD-04 quorum on #3395 round 0 correctly refused): (1) 24 new contract files contracts/crux-O-01-v1.yaml .. crux-O-24-v1.yaml, each metadata.category O, competitor AutoGluon, status draft, intake_status missing, registry false, and each passing pv validate; (2) CRUX_COMPETITORS grows by exactly one entry (autogluon) in crates/aprender-contracts/src/schema/validator.rs and in both crux_intake_tests.rs lists; (3) CATEGORY_NAMES gains O in scripts/crux_scaffold_contracts.py; (4) the master registry gains one row per new contract and evidence_sources.autogluon; (5) 26 roadmap fragments exist (PMAT-3370, PMAT-3371..PMAT-3394, PMAT-3401) and docs/roadmaps/roadmap.yaml equals aggregate(entries/), additive vs origin/main (deleted=0, reserialised=0); (6) contracts/census.json n_files equals origin/main''s n_files at merge time plus 24, contracts/contracts.nt equals a fresh pv extract of the merged tree, and README''s CONTRACT_COUNT blocks equal census n_files — the three derivatives ride in this PR because adding a contract stales all three (#3569). Known and NOT in scope: the census files these 24 under kind kernel because they carry no metadata.kind and registry:false (main''s crux contracts carry registry:true and file under registry); a draft/intake bucket for the census is a census question (#3640-class), not this PR''s.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index e2ff36ab22..5140b63ae0 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -18717,7 +18717,7 @@ roadmap: - id: PMAT-3401 github_issue: 3401 item_type: task - title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows (coverage_intake 267->291) + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, README CONTRACT_COUNT 1842->1866 — child of #3370' + title: 'Admit AutoGluon 1.6.3 to CRUX as category O: 24 DRAFT contracts crux-O-01..24 with LIVE-PENDING gates (no implementation, no CLI, no tests), CRUX_COMPETITORS 14->15 in validator.rs + both crux_intake_tests.rs lists, CATEGORY_NAMES[O] in scripts/crux_scaffold_contracts.py, 24 master-registry rows + evidence_sources.autogluon, spec v2.3 §3/§5/§6, evidence/crux/autogluon/ (4 files), 26 roadmap fragments (PMAT-3370 epic, PMAT-3371..3394 stories, PMAT-3401 this ticket) + regenerated roadmap.yaml, and the three corpus derivatives (census, contracts.nt, README CONTRACT_COUNT) regenerated to the merged tree — child of #3370' status: in_progress priority: critical assigned_to: null @@ -18734,7 +18734,7 @@ roadmap: - enhancement - crux - pareto-autogluon - notes: null + notes: 'ACCEPTANCE, stated as invariants and not as counts (the first version of this row pinned "README CONTRACT_COUNT 1842->1866" and "coverage_intake 267->291" from the 2026-09-16 tree; main moved and the numbers described a tree that no longer existed, which is what the AD-04 quorum on #3395 round 0 correctly refused): (1) 24 new contract files contracts/crux-O-01-v1.yaml .. crux-O-24-v1.yaml, each metadata.category O, competitor AutoGluon, status draft, intake_status missing, registry false, and each passing pv validate; (2) CRUX_COMPETITORS grows by exactly one entry (autogluon) in crates/aprender-contracts/src/schema/validator.rs and in both crux_intake_tests.rs lists; (3) CATEGORY_NAMES gains O in scripts/crux_scaffold_contracts.py; (4) the master registry gains one row per new contract and evidence_sources.autogluon; (5) 26 roadmap fragments exist (PMAT-3370, PMAT-3371..PMAT-3394, PMAT-3401) and docs/roadmaps/roadmap.yaml equals aggregate(entries/), additive vs origin/main (deleted=0, reserialised=0); (6) contracts/census.json n_files equals origin/main''s n_files at merge time plus 24, contracts/contracts.nt equals a fresh pv extract of the merged tree, and README''s CONTRACT_COUNT blocks equal census n_files — the three derivatives ride in this PR because adding a contract stales all three (#3569). Known and NOT in scope: the census files these 24 under kind kernel because they carry no metadata.kind and registry:false (main''s crux contracts carry registry:true and file under registry); a draft/intake bucket for the census is a census question (#3640-class), not this PR''s.' - id: PMAT-3427 github_issue: 3427 item_type: task From 803037cfdaa03eb27b2f56e22e6cb9956e96cc5c Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 04:02:05 +0200 Subject: [PATCH 6/6] PMAT-3401: quorum verdict 3/3 on 9f59bebb6 (AD-04) Round 0 was 3/3 FAIL on the row's own pinned counts from a 09-16 tree; the row now states invariants. Round 1: 3/3 PASS, gemini-3.1-pro-high / pro-low / 3.6-flash-high, each measured, no dissent. Refs #3401 Co-Authored-By: Claude Opus 5 (1M context) --- docs/audits/quorum-PMAT-3401.json | 160 ++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 docs/audits/quorum-PMAT-3401.json diff --git a/docs/audits/quorum-PMAT-3401.json b/docs/audits/quorum-PMAT-3401.json new file mode 100644 index 0000000000..3378c689aa --- /dev/null +++ b/docs/audits/quorum-PMAT-3401.json @@ -0,0 +1,160 @@ +{ + "ticket": "PMAT-3401", + "base": "main", + "base_resolved": "origin/main", + "base_note": "local main differs from origin/main by 128 commit(s); judged against origin/main", + "head": "9f59bebb65d51481fef673857209e3e32063a16c", + "diff_sha256": "bb44e1f9146e1711a4fd2cc58ef708592dcf2c58dcd76c41174c84b7c9a38144", + "width": 3, + "executor": "agy", + "prompt_mode": "file", + "prompt_bytes": 264947, + "author": { + "model": "claude-opus-5", + "family": "claude", + "source": "flag" + }, + "agreed": true, + "lanes": [ + { + "lane": 1, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "The diff exactly matches the ticket PMAT-3401 requirements. It introduces 24 DRAFT contracts for AutoGluon (crux-O-01 through 24) with LIVE-PENDING gates and no associated code implementations, CLI changes, or test bodies. It correctly increments CRUX_COMPETITORS in validator.rs from 14 to 15, adds autogluon to both lists in crux_intake_tests.rs, and introduces the category O in the crux_scaffold_contracts.py script. The master registry and specifications (v2.3) have been appropriately updated, including evidence_sources.autogluon, 24 new stories, and modifications in §3, §5, and §6 of the CRUX workflows spec. It also correctly supplies the 4 evidence files. Furthermore, the roadmap components are properly configured (26 fragments spanning PMAT-3370, 3371-3394, and 3401) with a regenerated roadmap.yaml. Lastly, the three corpus derivatives (census.json, contracts.nt, README CONTRACT_COUNT) are all correctly updated reflecting the addition of 24 contracts (1800 -> 1824). I found no refutations. PASS.", + "findings": [], + "raw_bytes": 4247, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.1-pro-high", + "model_measured": "gemini-3.1-pro-high", + "model_source": "measured", + "family": "gemini" + }, + { + "lane": 2, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "The diff correctly matches all the requirements from the ticket. It adds exactly 24 DRAFT contracts (crux-O-01 to crux-O-24) with LIVE-PENDING gates, updates CRUX_COMPETITORS to 15 in validator.rs and crux_intake_tests.rs, adds the category O in the python script, adds the master-registry rows and evidence sources, updates the specification document for v2.3, adds the 4 evidence files, adds the 26 roadmap fragments and regenerates roadmap.yaml, and correctly updates the three corpus derivatives to reflect the 24 new contracts. No unintended changes or unrequested implementations were found.", + "findings": [], + "raw_bytes": 3346, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.1-pro-low", + "model_measured": "gemini-3.1-pro-low", + "model_source": "measured", + "family": "gemini" + }, + { + "lane": 3, + "status": "SUCCESS", + "verdict": "PASS", + "summary": "Lane 3 review of PMAT-3401 complete. The diff admits AutoGluon 1.6.3 to CRUX as Category O with 24 draft contracts (crux-O-01..24) featuring LIVE-PENDING gates, CRUX_COMPETITORS 14->15 in validator.rs and crux_intake_tests.rs, CATEGORY_NAMES[O] in crux_scaffold_contracts.py, 24 master-registry rows + evidence_sources.autogluon in crux-competitive-research-ux-v1.yaml, workflow spec updates v2.3, 4 evidence files, 26 roadmap fragments + regenerated roadmap.yaml, and all 3 corpus derivatives (census.json, contracts.nt, README.md). All 1666 tests pass, pv census and extract pass with zero drift, and no refutations were found. Verdict: PASS.", + "findings": [ + { + "claim": "All 1666 tests in aprender-contracts pass cleanly with zero failures.", + "command": "cargo test -p aprender-contracts", + "file": "crates/aprender-contracts/src/schema/validator.rs", + "fix": "None required; all tests pass.", + "grounding": "measured", + "line": 120, + "output": "test result: ok. 1666 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 23.64s" + }, + { + "claim": "contracts/census.json matches the output of fresh pv census for 1824 contracts.", + "command": ". scripts/pv_bin.sh && \"$PV\" census contracts --format json > /tmp/test_census.json && diff -u contracts/census.json /tmp/test_census.json", + "file": "contracts/census.json", + "fix": "None required; census is fully synchronized.", + "grounding": "measured", + "line": 5, + "output": "diff exit code 0, 1824 contracts counted in contracts/census.json" + }, + { + "claim": "README.md CONTRACT_COUNT blocks are synchronized to 1824 contracts.", + "command": "bash scripts/readme_sync.sh --check", + "file": "README.md", + "fix": "None required; README matches measured contract count.", + "grounding": "measured", + "line": 25, + "output": "ok readme_sync: README.md already states the measured count 1824 in every CONTRACT_COUNT block" + }, + { + "claim": "The 24 category O contracts crux-O-01..24 were added with draft status, LIVE-PENDING gates, and no implementation as required by PMAT-3401.", + "file": "contracts/crux-O-01-v1.yaml", + "fix": "None required; contracts meet specification.", + "grounding": "cited", + "line": 202 + } + ], + "raw_bytes": 6493, + "err_bytes": 0, + "envelope_status": "SUCCESS", + "verdict_source": "structured_output", + "grounding_check": "parity", + "model": "gemini-3.6-flash-high", + "model_measured": "gemini-3.6-flash-high", + "model_source": "measured", + "family": "gemini" + } + ], + "dissent": [], + "dedup": [ + { + "file": "README.md", + "line": 25, + "lanes_agreeing": [ + 3 + ], + "claims": [ + "README.md CONTRACT_COUNT blocks are synchronized to 1824 contracts." + ] + }, + { + "file": "contracts/census.json", + "line": 5, + "lanes_agreeing": [ + 3 + ], + "claims": [ + "contracts/census.json matches the output of fresh pv census for 1824 contracts." + ] + }, + { + "file": "contracts/crux-O-01-v1.yaml", + "line": 202, + "lanes_agreeing": [ + 3 + ], + "claims": [ + "The 24 category O contracts crux-O-01..24 were added with draft status, LIVE-PENDING gates, and no implementation as required by PMAT-3401." + ] + }, + { + "file": "crates/aprender-contracts/src/schema/validator.rs", + "line": 120, + "lanes_agreeing": [ + 3 + ], + "claims": [ + "All 1666 tests in aprender-contracts pass cleanly with zero failures." + ] + } + ], + "uncovered": [], + "coverage_source": "lanes", + "partial": false, + "partial_reasons": [], + "auto_merge": { + "checked": true, + "was_armed": false, + "disarmed": false, + "note": "auto-merge not armed" + }, + "lint": { + "ok": true, + "output": "receipt complete: kind=artifact lanes=3 author=claude-opus-5/claude" + } +}