Rules.md 1.3 run/result/submission vocabulary, runs_per_result edition value, shared readiness evaluator (status-and-submit PR 1)

[FileSystemGuy]

PR 1 of the mlpstorage status / mlpstorage submit series (design agreed 2026-09-23; UX note in .planning/status-and-submit-proposal.md). This PR adds no CLI verb. It lays down the three things the verbs will stand on: the vocabulary, the per-result run count as edition data, and the shared readiness evaluator.

1. Rules.md 1.3 runResultSubmission -- vocabulary

Three nouns, pinned once and used everywhere from here on:

noun	is	handle
run	one benchmark invocation: one timestamp directory and its contents	ledger ID (mlpstorage runs)
result	the runs under one workload directory of one system name that share benchmark, workload and emulated accelerator; what §3-§6 reduce to one results.csv row	(division, system, benchmark, model, accelerator)
submission	every result of one submitter in one division hierarchy, plus its systems/ and submission.yaml	the results-dir

The rule is declarative (the lint passes with no allowlist change). Rationale, including why the accelerator is part of a result's identity although it is not a directory level, is in RulesCommentary.md §1.3.

2. runs_per_result in the editions table

editions.yaml 3.0 checker: gains the runs one complete result holds, per family:

runs_per_result:
  training: 6          # 2.1.17: 1 warm-up + 5 measured
  checkpointing: 2     # 4.7.1: write phase + read phase (a combined invocation supplies both)
  vector_database: 5   # 5.3.1
  kv_cache: 1          # 6.3.2.1: one CLOSED sequence run

• editions.py parses and validates it (positive integers; keys must be exactly the edition's workload families), CheckerParameters.runs_per_result, Config.get_runs_per_result(family).
• The validator's two hard-coded counts now read it: 2.1.17 (directory_checks.py) and 5.3.1 (vdb_checks.py). submission_checker/constants.py RUN_TIMESTAMP_COUNT is retired; the three tests that pinned it now pin the table value.
• Rules.md "Rules editions" paragraph, ManPage.md VALIDATOR + FILES, README.md name the new block content (the existing doc-parity tests cover the checker block).

Validator-diff on the frozen v3.0 tree: identical (see below).

3. mlpstorage_py/readiness.py -- the shared evaluator

evaluate(results_dir) -> SubmissionReadiness: per-result rows with RUNS have/required, a SUBMIT token, a one-line NOTE, per-run rows (ok / failed / running / invalid / extra, counted), per-system paperwork, tree-level problems and warnings; to_dict() for --json; evaluate_result(results_dir, leaf) for the post-run recap.

Two inputs, no third:

• the run ledger (IDs, exit_status, the run-time verification);
• the submission checker behind mlpstorage validate, run in-process on the results-dir with its [<id> <name>] <path>: <msg> findings captured instead of printed (collect_findings). Nothing is re-implemented: every finding is attributed -- to a run (path inside a leaf), a result (workload dir), a system's paperwork (systems/<name>.*, rules 2.1.7 / 2.1.8 / 4.7.3 / 4.7.4), a result's code-image paperwork (CHECK-0x / PROV-01 on a leaf) or the tree. Count rules (2.1.17 / 2.1.23 / 5.3.1) fold into RUNS; rollup rules (2.1.16 / 2.1.22 / RPT-01) are dropped because submit regenerates them; anything under whatif/ is dropped.

Severity is stage, not level (design decision 1): paperwork never touches a run's status; failed / INVALID / OPEN-under-closed / extra runs are rerun problems and the NOTE names the mlpstorage runs rm to run.

SUBMIT precedence: short (counted < required) > invalid (complete, but a run must go or a workload rule failed) > paperwork > ready; - for whatif. The design note listed three tokens; invalid is new -- a result with six good runs and a seventh failed one is not short, and calling it so would send the submitter to run more.

Counted set: training / vdb / kvcache take the oldest required ok runs (2.1.17's warm-up is the earliest); checkpointing counts phases -- the first ok run supplying writes and the first supplying reads, a combined run both. A run whose metadata is not written yet (still running) joins its sibling result when there is exactly one.

Tests

tests/unit/test_readiness.py (62 tests): the table field and its validation; Rules.md 1.3 and its commentary; finding parsing; classification against scripted findings (every token, every run status, attribution of each rule class, whatif drop, warnings, JSON shape, evaluate_result); and the real checker on the definition-of-done fixture tree (capture is silent on stdout/stderr, logger state restored, findings carry rule IDs, missing PDF / schema error land as paperwork). The DoD "good" fixture is known-noisy under validate (no DLIO logs, no pool) and the evaluator says exactly that, which is the point.

Realism: evaluate() on a copy of TTA's v3.0 subtree runs in 1.2 s and reports the same things validate reports on it since the post-v3.0 hard-error promotions (#847/#848): unet3d 6/6 invalid (2.1.12 datasize missing), llama3-70b 2/2 invalid (4.3.4 no accelerator recorded), kv_cache 1/1 ready, vdb 0/5 (5.4.2 sidecar), CHECK-02 at tree level.

Not in this PR (PRs 2-4)

mlpstorage status (+ --runs, filters, --json), the per-result recap after every run, the STATUS token in runs list, mlpstorage submit --dry-run / package / ledger, ManPage + --help_all entries.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅