From 42b53af305f3fad0a56bc54014b6435a2e3c5d82 Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Mon, 28 Sep 2026 11:58:12 -0700 Subject: [PATCH] stage-contract, decisions: clarity pass (Codex) Co-Authored-By: Claude Opus 5.5 --- system/decisions.md | 286 ++++++++++++++++++------------------- system/stage-contract.md | 301 ++++++++++++++++++++------------------- 2 files changed, 296 insertions(+), 291 deletions(-) diff --git a/system/decisions.md b/system/decisions.md index 4e94b64..991841d 100644 --- a/system/decisions.md +++ b/system/decisions.md @@ -2,15 +2,14 @@ **Status: DRAFT** -Any team member edits this page directly, adding one dated line per -ruling with the author's name; CI on `main` is the only gate. A ruling is -the team's once it is on this page, and the design pages state the rule -itself and cite this page where the attribution matters. +Any team member can add a ruling directly, with one dated line and the +author's name. CI on `main` is the only gate. Once recorded here, a +ruling belongs to the team. Design pages state the rule and cite this +page where attribution matters. ## Rulings -Ben's rulings of 2026-09-27 on the open questions of the rebuild, as he -gave them. +Ben's rulings of 2026-09-27 on the rebuild's open questions. (decision-trial-database)= **SMDC database**: dedicated to the rebuild. dev's scripts never run @@ -19,22 +18,22 @@ swversions) are a cutover item, not a defect. (Ben, 2026-09-27) (decision-science-flow)= **dev-to-rebuild science flow**: the rebuild side owns it. Each ported -module under `rapidpipe/science` pins the dev file and commit it was -copied from; a CI script on `rebuild` reports dev commits since each pin -that touched the source; porting stays by hand. (Ben, 2026-09-27) +module under `rapidpipe/science` pins its source dev file and commit. +A CI script on `rebuild` reports later dev commits that touched each +pinned source. Porting stays by hand. (Ben, 2026-09-27) (decision-loop-promotion)= -**Loop promotion**: the spec. The loop uses the same auto-promote gate as -`run start`; under a trial policy a date's production run stays a -candidate until a person promotes it or a team-approved policy with -automatic promotion exists. loop.md and the unit test that expects the -opposite change. (Ben, 2026-09-27) +**Loop promotion**: follow the spec. The loop uses the same auto-promote +gate as `run start`. Under a trial policy a date's production run stays +a candidate until a person promotes it or a team-approved policy with +automatic promotion exists. Change loop.md and the unit test that +expects the opposite. (Ben, 2026-09-27) (decision-acceptance)= **Acceptance**: "required checks pass" only. The acceptances table, -`check accept` and the nine derived ancestor states go; the ancestor walk -reduces to "every ancestor is current or superseded"; a false failure is -fixed in the policy file. (Ben, 2026-09-27) +`check accept` and the nine derived ancestor states go. The ancestor +walk reduces to "every ancestor is current or superseded". Fix a false +failure in the policy file. (Ben, 2026-09-27) (decision-legacy-compatibility)= **Legacy compatibility**: remove without a probe. The `logical_key` @@ -43,15 +42,15 @@ selector and `_recorded_inverse` path in promote, go; rapid_rebuild is disposable. (Ben, 2026-09-27) (decision-latency)= -**Fan-out and latency**: rough requirement stated: **within an hour per -detector image, from its delivery to its alert current, typical, with a -dense-field tail accepted; RAPID's time short against the exposure-to-L2 -delay.** Consequences: per-delivery batches with an event or ~30-minute -window trigger (operations.md's own rule); within-date fan-out over -detector images; the difference stage's ~58 min runtime (mostly Photutils -PSF fits) is the binding constraint. Parallel dates and lanes follow the -number. (Ben, 2026-09-27; provisional, the team confirms or replaces the -number) +**Fan-out and latency**: the rough requirement is within an hour per +detector image, from delivery until its alert is current, typically, +with a dense-field tail accepted. RAPID's time is short against the +exposure-to-L2 delay. This requires per-delivery batches with an event +or ~30-minute window trigger (operations.md's own rule) and within-date +fan-out over detector images. The difference stage's ~58 min runtime +(mostly Photutils PSF fits) is the binding constraint. Parallel dates +and lanes follow the number. (Ben, 2026-09-27; provisional, the team +confirms or replaces the number) (decision-step-ledgers)= **Step ledgers**: stay ephemeral. The 632 "supervisor step / R / A" @@ -60,31 +59,31 @@ implement; the 313 dated narration lines leave the pages. (Ben, 2026-09-27) (decision-authority)= -**Authority**: the lead is removed; the team owns the design. A decisions -page in rapid_docs, edited directly by any team member, one dated line -per ruling with the author's name; a ruling is the team's once it is on -the page, no review gate; CI on `main` stays. Every "the lead's" on the -pages becomes the team's. The rulings above go on that page as Ben's, -dated today. Team communication about the rebuild remains Ben's to open. +**Authority**: remove the lead role; the team owns the design. Any team +member can edit the decisions page in rapid_docs directly, adding one +dated line per ruling with its author. Once recorded, a ruling is the +team's, with no review gate; CI on `main` stays. Every "the lead's" on +the pages becomes the team's. Record the rulings above as Ben's, dated +today. Team communication about the rebuild remains Ben's to open. (Ben, 2026-09-27) ## Earlier rulings -Rulings Ben made while he held the lead role. Under the authority ruling -they are the team's; each page states the rule. +Ben made these rulings while he held the lead role. The authority ruling +makes them the team's; each page states the rule. -- 2026-09-21: the `dev` schema is kept: nothing is renamed or dropped; - new columns and tables are added where the vocabulary needs them +- 2026-09-21: keep the `dev` schema: rename or drop nothing; add + columns and tables where the vocabulary needs them ([products](products)). The rebuild's schema coexists with `dev`'s and writes it ([runs](runs)). -- 2026-09-21: nothing about a run's records is ever dropped; deleting a - run's data changes state, not history ([runs](runs)). +- 2026-09-21: retain run records in full; deleting a run's data changes + state, not history ([runs](runs)). - 2026-09-21: current result sets are readable by any run as frozen inputs, by instance id; scratch result sets only within their own run ([products](products)). - 2026-09-22: every ported stage minimises differences from `dev` and - designs in, off by default, anything that could be left unused (every - stage page). The catalog stage's detection member is per-differencer + makes anything that could be left unused optional and off by default + (every stage page). The catalog stage's detection member is per-differencer and recorded with the instance; `vbest` stays the query surface ([products](products)). - 2026-09-22: all promotions take one transaction-scoped advisory lock @@ -92,8 +91,8 @@ they are the team's; each page states the rule. - 2026-09-23: `dev`'s once-per-date CLUSTER and ANALYZE move to their own `maintain` stage ([maintain](maintain), [load](load)). - 2026-09-23: only one producer's frozen inputs bind at a time - ([runs](runs)); everything `dev` wrote is never touched by rebuild - deletion or cleanup ([runs](runs)). + ([runs](runs)); rebuild deletion and cleanup never touch anything + `dev` wrote ([runs](runs)). - 2026-09-23: no per-child UNIQUE on `(pid, id, isdiffpos)`; SFFT loads against its own `pid`; a psf instance's `version` equals the allocated legacy number and `psfs`' key stays global ([load](load)). @@ -103,8 +102,8 @@ they are the team's; each page states the rule. ([difference](difference), [finalize](finalize)). - 2026-09-26: gain matching reads the reference's own `MAGZP` header by default, with `[awaicgen] zprefimg` as an override; SFFT registers as - its own `difference-image` instance by default; which differencer's - instance is current downstream is a promotion choice + its own `difference-image` instance by default; promotion chooses which + differencer's instance is current downstream ([difference](difference), [reference](reference)). - 2026-09-26: each source's field is its own tessellation tile, a stated departure from `dev` ([load](load)). @@ -112,77 +111,37 @@ they are the team's; each page states the rule. ## Carried from the build, pending team review -Rulings made during the rebuild by its build sessions and their reviews. -Each is in force as its page states it until the team confirms, changes -or removes it here. +These rulings come from build sessions and their reviews. Each remains +in force as its page states it until the team confirms, changes or +removes it here. -(decision-slot-identity)= -- **Association-set identity.** The slot-identity table on - [products](products) is the one statement. An association set's identity - adds the field, the crossmatch settings hash, the sorted identities of - its source sets, and a hash of its base's own identity; the proposal's - shorter wording, which omitted the base, is withdrawn with the table - copy it sat in. (2026-09-27) +### Checks -(decision-pending-sharing-rule)= -- **Sharing rule widening.** The specification lets any run read - *current* result sets and keeps scratch within its run; the eligibility - table on [products](products) also lets another run's stage read a - *selected candidate*, whatever its check outcome. The team decides - whether selected candidates are readable across runs and aligns the - specification or the table. (2026-09-26) - **Trial policy.** `rebuild-trial@1`'s `approval: trial` is a trial approval, not the team's sign-off; promotion refuses a policy with `approval: none`, and automatic promotion needs a team approval and `auto_promote: true` ([checks](checks)). (2026-09-24, 2026-09-25) -- **Package layer order.** The subpackage layer order on - [stage-contract](stage-contract) is enforced by a unit test over - every module and top-level import, lazy and relative imports - included. (2026-09-27) -- **`tests/cli` scope.** A `tests/cli` edit may change only the module - path of a monkeypatch target or an import, when the patched code - moves; its assertions, scenarios and fixtures do not change. (2026-09-27) -- **CLI surface on `launch`.** `rapidpipe.cli` takes from - `rapidpipe.launch` only its public functions and the exceptions they - raise; a unit test enforces it. (2026-09-27) -- **Sky-partition geometry.** The pure HEALPix and tessellation - derivations `db` needs live in `rapidpipe.products`, not - `rapidpipe.science`; `products` is no longer standard-library only, - since it imports numpy, healpy and the tessellation code. - ([stage-contract](stage-contract)) (2026-09-27) -- **Running checks against a run.** That code lives in `rapidpipe.runs`, - not `rapidpipe.checks`, since `checks` sits below `runs` in the layer - order. ([stage-contract](stage-contract), [runs](runs)) (2026-09-27) -- **`run --help` grouping.** `rapidpipe run --help` groups its - subcommands under five titled sections, lifecycle, recovery, - promotion, housekeeping and inspection ([tool](tool)); no subcommand - name changes. (2026-09-27) - -### Checks - -- A check always records a row, and a check that raises records - `failed`. (2026-09-24) +- Every check records a row; one that raises records `failed`. (2026-09-24) - `catalog-counts-vs-reference@1` finds its reference by slot; `rebuild-trial@1` marks `difference-image-statistics` required and `catalog-counts-vs-reference` advisory. (2026-09-24, 2026-09-26) - `run_policy_checks` fills every candidate's slot before any check runs. (2026-09-26) -- Known defect, recorded not fixed: the gate can miss a check-result - row committed after its read. (2026-09-27) +- Known, unfixed defect: the gate can miss a check-result row committed + after its read. (2026-09-27) - Checks run outside the pipeline image, on a workstation or the launcher host, reaching the database through the instance role. (2026-09-24) ### Runs and promotion -- `promotion_changes` carries a nullable `slot`; a replacement's kind - and slot must equal the requested selector; `StalePlan` refuses when - the selection or the run's candidates moved since the plan. +- `promotion_changes` carries a nullable `slot`. A replacement's kind + and slot must match the requested selector. `StalePlan` refuses if + the selection or the run's candidates have moved since the plan. (2026-09-26) -- A bare after-instance with no before is accepted only when the slot - has no current occupant; a change into an `association-set` slot with - a non-null expected-before must have the before as an ancestor of the - after. (2026-09-24) +- An after-instance with no before is accepted only if the slot has no + current occupant. In an `association-set` slot with a non-null + expected-before, the before must be an ancestor of the after. (2026-09-24) - Promotion's dependency walk covers every ancestor recursively; rollback skips the check-policy gate, the walk and the association-set chain-direction rule. (2026-09-24, 2026-09-26, @@ -190,15 +149,15 @@ or removes it here. - Released-image validation and check-policy validation are separate. (2026-09-24, 2026-09-26) - A same-request ancestor passes the dependency walk and is validated - on its own, against the same eligibility rule and check policy any - other after-instance answers to. (2026-09-27) + separately against the same eligibility rule and check policy as any + other after-instance. (2026-09-27) - Rollback reverses by slot only; a recorded change with no slot is refused as not reversible. (2026-09-27) - A policy refusal (`PromotionRefused`, `StalePlan`, `CheckPolicyRefused`) exits 1; an argument-shaped refusal or an unknown run, instance or promotion id exits 64. (2026-09-27) -- The check-policy gate's `FOR SHARE` lock on the `checks` rows it - relies on stays; the pipeline role can update `checks` rows, so the +- The check-policy gate keeps its `FOR SHARE` lock on the `checks` + rows it reads. The pipeline role can update `checks` rows, so the table is not append-only. (2026-09-27) - The `approval` value `lead` is renamed `team`. (2026-09-27) - Seeding a replacement run from a failed one is `--seed --only-failed`; @@ -208,8 +167,8 @@ or removes it here. frozen input binding depends on it or a reference row would be orphaned; the science-row cleanup set is fixed on [runs](runs). (2026-09-24) -- Every submission binds its input set before allocating the attempt; a - name that resolves to nothing is logged, not refused; an unreadable +- Every submission binds its input set before allocating the attempt. + A name that resolves to nothing is logged, not refused. An unreadable manifest refuses with exit 65 before anything is written. (2026-09-25) - A retry after a failed commit writes a fresh result set; only the calling attempt's own prior try or a succeeded attempt's set is reused @@ -217,13 +176,13 @@ or removes it here. (2026-09-25) - A scratch run reads the `_SCRATCH`-suffixed job definition and never falls back to the unsuffixed one. (2026-09-24) -- **The launcher retries, not Batch.** The rebuild's job definitions - carry one Batch attempt; the launcher resubmits a unit after exit 75, - a reclaimed host or a container that never started, each time as a - fresh attempt within the run's maximum attempts per unit, which - therefore also counts reclaims. The default maximum stays 1, so a run - that wants retries sets it. Carried from the build, pending team - review. (Claude for Ben, 2026-09-28) +- **Launcher retries.** The rebuild's job definitions carry one Batch + attempt. After exit 75, a reclaimed host or a container that never + started, the launcher resubmits the unit as a fresh attempt. The run's + maximum attempts per unit includes reclaims and defaults to 1. Retries + stay within that limit; a run must set it to allow them. + Carried from the build, pending team review. + (Claude for Ben, 2026-09-28) - Runs written under the first-run prefix stay where they were written. (2026-09-24) @@ -241,15 +200,36 @@ or removes it here. already-current sky continues the existing schedule until a chain switch exists. (2026-09-27) - A finished run takes no new units. (2026-09-25) -- Discovery: identical re-delivery refused, checksum conflict - quarantined, a corrected version deferred ([loop](loop)). (2026-09-26) +- Discovery refuses identical re-delivery, quarantines a checksum + conflict and defers a corrected version ([loop](loop)). (2026-09-26) - Under a check policy without automatic promotion, the loop leaves a - date's run a candidate rather than refusing it; the next date's base - still binds to it, so a person promotes in date order, an earlier - date first. (2026-09-27) + date's run a candidate rather than refusing it. The next date's base + still binds to it, so a person promotes in date order, earlier dates + first. (2026-09-27) ### Stage contract and tool +- **Package layer order.** A unit test enforces the subpackage layer + order on [stage-contract](stage-contract) across every module and + top-level import, including lazy and relative imports. (2026-09-27) +- **`tests/cli` scope.** When patched code moves, a `tests/cli` edit may + change only the module path of a monkeypatch target or an import. + Assertions, scenarios and fixtures stay unchanged. (2026-09-27) +- **CLI surface on `launch`.** `rapidpipe.cli` takes from + `rapidpipe.launch` only its public functions and the exceptions they + raise; a unit test enforces it. (2026-09-27) +- **Sky-partition geometry.** The pure HEALPix and tessellation + derivations `db` needs live in `rapidpipe.products`, not + `rapidpipe.science`; `products` is no longer standard-library only, + since it imports numpy, healpy and the tessellation code. + ([stage-contract](stage-contract)) (2026-09-27) +- **Running checks against a run.** That code lives in `rapidpipe.runs`, + not `rapidpipe.checks`, since `checks` sits below `runs` in the layer + order. ([stage-contract](stage-contract), [runs](runs)) (2026-09-27) +- **`run --help` grouping.** `rapidpipe run --help` groups its + subcommands under five titled sections, lifecycle, recovery, + promotion, housekeeping and inspection ([tool](tool)); no subcommand + name changes. (2026-09-27) - `maintain` is a fifth stage and `detector-date` a fifth unit kind. (2026-09-24) - `alerts` and `export` read named, completed result sets; `export` @@ -265,8 +245,7 @@ or removes it here. - **Photometry stub removed.** The rebuild carries no photometry stage; forced photometry stays `dev`'s `forcedPhotometryForField.py`. Exit code 69 stays reserved in the stage contract, since no stage in this - build returns it now. Carried from the build, pending team review. - (2026-09-27) + build returns it now. Carried from the build, pending team review. (2026-09-27) - `run cancel` from a workstation needs `batch:TerminateJob` on the workstation role, not yet granted. (2026-09-24) - **Custody database access.** A stage's database access is `none`, @@ -274,44 +253,42 @@ or removes it here. read guard. `admit`, `reference`, `difference` and `finalize` declare it, since the guard needs the database whenever their input names a registered instance, and `none` skips the guard ([stage - contract](stage-contract), "Declaration"). Carried from the build, - pending team review. (Claude for Ben, 2026-09-28) -- A stage declaration carries no argument schema and no resource - defaults: nothing read them, and the Batch job definition owns - resources ([stage contract](stage-contract)). `run create` takes no - `--lane`, `--profile` or `--db-target`; the run's lane, resource - profile and database target columns keep their defaults (`local`, - `local`, the connection's database) and nothing reads them to decide - anything. (2026-09-27) + contract](stage-contract), "Declaration"). Carried from the build, pending team review. (Claude for Ben, 2026-09-28) +- A stage declaration carries no argument schema or resource defaults: + nothing read them, and the Batch job definition owns resources + ([stage contract](stage-contract)). `run create` takes no `--lane`, + `--profile` or `--db-target`. The run's lane, resource profile and + database target columns keep their defaults (`local`, `local`, the + connection's database) but control no behaviour. (2026-09-27) ### Releases -- The release command, tag form (`rebuild-v0.`, annotated, remote - authoritative), the `releases` and `release_deployments` tables, - account-specific steps as hooks, migrations applied with recorded - checksums before a cut records, a run submitting only to its release's - job-definition revision, released-image provenance at promotion, one - pin row per consumer per cut, selftests as evidence not a gate - ([releases](releases)). (2026-09-24) +- [Releases](releases) defines the release command, tag form + (`rebuild-v0.`, annotated, remote authoritative), and `releases` + and `release_deployments` tables. Account-specific steps are hooks. + Migrations are applied with recorded checksums before a cut records. + A run submits only to its release's job-definition revision; + promotion requires released-image provenance. Each consumer has one + pin row per cut. Selftests are evidence, not a gate. (2026-09-24) - Migrations are additive while an earlier release's runs are open; `cut` refuses while any release row is incomplete unless resumed. (2026-09-25) ### Stages -- `alerts` is the first registrar of `alert-container` and `alert-set`; - the outbox locator is an Avro block offset and length plus the - record's ordinal; an input naming the wrong base or two pruned sets - for one association set exits 65. (2026-09-24, 2026-09-25) +- `alerts` first registers `alert-container` and `alert-set`. The outbox + locator is an Avro block offset and length plus the record's ordinal. + An input naming the wrong base or two pruned sets for one association + set exits 65. (2026-09-24, 2026-09-25) - Finalize's chain order is `difference -> finalize -> register -> load`, one `register` pass. (2026-09-24) -- `reference` is a transform stage with no database access; filter - spellings are normalised; frame counts are bounded; steps run in - `dev`'s order; fake-source injection is not ported; the catalog key, - the full SHA-256 selection digest, the global `refimages.version` - counter under an advisory lock, `refimages.attempt` recording the - producing attempt, nullable `npucatsources`, and replay checked against - the stored block are as [reference](reference) states. (2026-09-24) +- `reference` is a transform stage with no database access. It + normalises filter spellings, bounds frame counts and runs steps in + `dev`'s order; fake-source injection is not ported. [Reference](reference) + defines the catalog key, full SHA-256 selection digest, global + `refimages.version` counter under an advisory lock, `refimages.attempt` + recording the producing attempt, nullable `npucatsources`, and replay + checked against the stored block. (2026-09-24) - `statistics` reads crossmatch's completion manifest; an association set is its own rows plus its base's, recursively; its source sets already decide "best". (2026-09-24) @@ -324,9 +301,24 @@ or removes it here. ### Products -- Every instance carries a provenance key, an identity key and a slot, - derived in the database; a cycle-guard failure leaves the whole batch - unresolved; identity and slot are derived for every kind. (2026-09-26) +(decision-slot-identity)= +- **Association-set identity.** The slot-identity table on + [products](products) is the sole statement. An association set's + identity adds the field, crossmatch settings hash, sorted source-set + identities, and a hash of its base's own identity. Withdraw the + proposal's shorter wording, which omitted the base, and its table + copy. (2026-09-27) + +(decision-pending-sharing-rule)= +- **Sharing rule widening.** The specification lets any run read + *current* result sets and keeps scratch within its run; the eligibility + table on [products](products) also lets another run's stage read a + *selected candidate*, whatever its check outcome. The team must decide + whether selected candidates are readable across runs, then align the + specification or the table. (2026-09-26) +- The database derives every instance's provenance key, identity key + and slot, with identity and slot derived for every kind. A cycle-guard + failure leaves the whole batch unresolved. (2026-09-26) - The reading and promotion-eligibility table applies identically to file products and result sets. (2026-09-26) - The `refimages` field lists and the `catalog-export` registration diff --git a/system/stage-contract.md b/system/stage-contract.md index 66ff093..0e08bd2 100644 --- a/system/stage-contract.md +++ b/system/stage-contract.md @@ -2,26 +2,28 @@ **Status: DRAFT** -Detail beneath the specification's "The stage contract" section, drawn -from the specification, the `dev` stage inventory and the `smdc` -salvage. A stage implementation is accepted only if it meets this contract. +A stage implementation is accepted only if it meets this contract. +It expands the specification's "The stage contract" section, drawing on +the specification, the `dev` stage inventory and the `smdc` salvage. + ## In plain terms A stage is one program that takes one declared piece of work, reads the inputs a manifest lists, does one job, and writes its outputs plus a manifest saying what it wrote. The same program accepts a local -directory or an S3 location. Stages that transform images use the -database only to check that their run may read the inputs they were -given; stages that build the catalog say what they read and write. Every execution is an attempt with its own output location, so -a retry can never overwrite a finished one. The caller decides about -retries, never the stage. +directory or an S3 location. Image-transform stages use the database +only to check that their run may read the supplied inputs; catalog +stages declare what they read and write. Each execution is an attempt +with its own output location, so a retry cannot overwrite a finished +one. The caller controls retries. ## The package -The repository builds one Python distribution, `rapid-pipeline`, whose import package is `rapidpipe`. The word `rapid` alone keeps meaning the project. Development -workstations use an editable installation; release images install the -distribution built from the recorded source commit. The package has -these subpackages: +The repository builds one Python distribution, `rapid-pipeline`, +with import package `rapidpipe`. The word `rapid` alone means the +project. Development workstations use an editable installation; +release images install the distribution built from the recorded +source commit. | Subpackage | Holds | |---|---| @@ -35,25 +37,24 @@ these subpackages: | `rapidpipe.selftest` | Each stage's packaged fixture and the runner that drives it. | | `rapidpipe.cli` | The command-line tool: parses the command line and prints, with no orchestration of its own. | -Dependency direction is fixed, a layer order enforced by a unit test over -every module, lazy and relative imports included: a unit imports only -units strictly below it. The leaf modules (`exitcodes`, `log`, -`revision`, `seams`: no `rapidpipe` import) come first, then `products`, -then `db` and `science`, which do not import each other, then `checks`, -then `runs`, then `stages`, then `launch`, then `selftest`, then `cli`. -`release` sits beside the stack: it imports only the leaves and `db`, -and only `cli` imports it. Stage modules do not import other stage -modules, or `launch` or `cli`. - -That sky-partition geometry is the HEALPix indexes and tessellation -fields a registered image's row carries, and it lives in `products` -rather than `science` so that `db` can derive it without importing -`science`. +A unit test enforces the dependency order across every module, +including lazy and relative imports: a unit imports only units +strictly below it. From lowest to highest, the layers are the leaf +modules (`exitcodes`, `log`, `revision`, `seams`, with no +`rapidpipe` import), `products`, `db` and `science` (which do not +import each other), `checks`, `runs`, `stages`, `launch`, +`selftest`, then `cli`. `release` sits beside this stack: it +imports only the leaves and `db`, and only `cli` imports it. Stage +modules do not import other stage modules, `launch` or `cli`. + +The sky-partition geometry comprises the HEALPix indexes and +tessellation fields in a registered image's row. It lives in +`products` so `db` can derive it without importing `science`. Stage entrypoints live in `rapidpipe.stages`, reusable algorithms in -`rapidpipe.science`, and database access and migration code in `rapidpipe.db`. -Existing code is retained only where it implements this contract. C tool -versions and build inputs are pinned by the container build. +`rapidpipe.science`, and database access and migration code in +`rapidpipe.db`. Existing code is retained only where it implements +this contract. The container build pins C tool versions and build inputs. ## The stage contract @@ -61,49 +62,52 @@ versions and build inputs are pinned by the container build. Every stage exports a declaration containing its name, unit of work, settings schema, versioned input and output product kinds, database -reads and writes, and supported exit codes. The declaration carries no -resources: the Batch job definition sets them. Importing the -declaration performs no I/O. Dependencies specify the -required product kinds, their mapping to upstream units, and the -condition that makes each input set complete. +reads and writes, and supported exit codes. Importing it performs no +I/O. Resources come from the Batch job definition, not the declaration. +Dependencies specify the required product kinds, their mapping to +upstream units, and the condition that makes each input set complete. +A field-wide stage waits for all its declared field inputs. Stage names are a stable list: `admit`, `reference`, `difference`, -`finalize`, `register`, `load`, `maintain`, `crossmatch`, `statistics`, -`prune`, `alerts`, `export` (`maintain` has its own page, -[maintain](maintain)). Units of work are `exposure`, +`finalize`, `register`, `load`, `maintain`, `crossmatch`, +`statistics`, `prune`, `alerts`, `export` (`maintain` has its +own page, [maintain](maintain)). Units of work are `exposure`, `detector-image`, `field`, `processing-date` and `detector-date` (`detector-date` is `maintain`'s unit; see [maintain](maintain), "Unit"). -A declaration's database access is one of four levels. `none` never -connects, and the shared runner skips the read guard for it. `custody` -connects only for the read guard, which judges the custody of every -registered instance the input manifest names (see "Invocation"), and -reads or writes nothing else. `read` and `read-write` also pass through -the guard. Every shipped stage reads an input manifest, so none -declares `none`. The transform stages (`admit`, `reference`, -`difference`, `finalize`) declare `custody`: they need a reachable -database whenever their input manifest names a registered instance. -`register` records file products from -manifests; `load` loads source rows; `maintain` clusters and analyzes a -`sources` child table, once per observation date and detector, reading -named source sets but writing none of its own. `crossmatch`, `statistics` -and `prune` read named, completed database result sets and write new -run-scoped result sets; their manifests identify those input and output -sets. `alerts` and `export` both read named, completed database result -sets; `export` writes files only, with no database writes of its own. +### Database access + +A declaration specifies one of four access levels. `none` never +connects, and the shared runner skips its read guard. `custody` +connects only for that guard, which judges the custody of every +registered instance named in the input manifest (see "Invocation"); +it reads or writes nothing else. `read` and `read-write` also pass +through the guard. Every shipped stage reads an input manifest, so +none declares `none`. + +The transform stages (`admit`, `reference`, `difference`, +`finalize`) declare `custody` and need a reachable database whenever +their input manifest names a registered instance. `register` records +file products from manifests; `load` loads source rows. `maintain` +clusters and analyzes a `sources` child table once per observation +date and detector, reading named source sets but writing none of its own. + +`crossmatch`, `statistics` and `prune` read named, completed +database result sets and write new run-scoped sets; their manifests +identify both. The run records the catalog versions and epoch range +crossmatch used. Each downstream stage names its exact predecessor +result set, and statistics identify the association set they describe. + +`alerts` and `export` also read named, completed database result +sets. `export` writes files only, with no database writes of its own. No stage changes another run's results or the current selection. -The run records the catalog versions and epoch range crossmatch used. -Each downstream stage names its exact predecessor result set. A -field-wide stage waits for all its declared field inputs. Statistics -identify the association set they describe. - ### Invocation Each stage exposes `main(argv)` and is directly runnable as -`python -m rapidpipe.stages.`; `rapidpipe stage ` calls that same -entrypoint. One invocation form: +`python -m rapidpipe.stages.`; `rapidpipe stage ` calls +the same entrypoint. One invocation form: ``` rapidpipe stage --run --unit --attempt \ @@ -112,28 +116,41 @@ rapidpipe stage --run --unit --attempt \ ``` `--inputs` names a local directory or S3 prefix containing -`manifest.json`; the stage reads only the immutable inputs that manifest -lists. `--outputs` names the attempt's exclusive output location. Every -execution receives a unique attempt ID and an exclusive output location: -the local runner allocates local attempt IDs, the launcher allocates one -per Batch job it submits, and a Batch job runs its container once, so a -retry is always a new submission with a new attempt. The launcher -selects at most one completed attempt per run, stage and unit. Deployment supplies -account-specific locations and connection settings through the -environment; explicit input and output locations may appear on the -command line. Credentials never appear in arguments or committed -settings. - -`--dry-run` validates arguments, settings and the input manifest, then -prints the planned inputs and outputs without executing science code or -writing files or database rows. Exit 0 means validation passed; no -completion manifest is written. The shared runner, `run_stage` in -`rapidpipe/stages/contract.py`, returns on `--dry-run` before a stage's -own body runs, so a stage that needs to validate its kind-specific -inputs before that return passes a `validate_inputs` callable to -`run_stage(..., validate_inputs=)`; the runner calls it with -the built `StageContext` after its own generic validation and before -the `--dry-run` return, not inside the body. +`manifest.json`; the stage reads only the immutable inputs it lists. +`--outputs` names the attempt's exclusive output location. Each +execution receives a unique attempt ID: the local runner allocates +local IDs, and the launcher allocates one per Batch job it submits. +A Batch job runs its container once, so a retry is a new submission +with a new attempt. The launcher selects at most one completed attempt +per run, stage and unit. + +Deployment supplies account-specific locations and connection settings +through the environment; explicit input and output locations may +appear on the command line. Credentials never appear in arguments or +committed settings. + +`--dry-run` validates arguments, settings and the input manifest, +then prints the planned inputs and outputs without executing science +code or writing files or database rows. Exit 0 means validation passed; +no completion manifest is written. + +The shared runner, `run_stage` in `rapidpipe/stages/contract.py`, +returns on `--dry-run` before the stage's body runs. A stage that +needs kind-specific input validation before that return passes a +`validate_inputs` callable to +`run_stage(..., validate_inputs=)`. The runner calls it with +the built `StageContext` after generic validation and before the +`--dry-run` return, outside the body. + +### Settings + +Each stage ships default settings and a settings schema under +`settings/.toml`. `--settings` supplies an overlay: tables merge +recursively, while supplied scalar and array values replace defaults. +Unknown keys and invalid values fail with code 64. Before execution, +the runner retains the resolved settings and their canonical hash. +Algorithm parameters, including random seeds where used, belong in +settings; deployment locations and credentials do not. ### The manifest @@ -148,16 +165,17 @@ The manifest records its schema version; run, unit, stage and attempt IDs; a reference to the execution record; references to the input manifest and any database result sets read; and each output's identity, kind, format version and location, with byte size and SHA-256 for -files. The execution record, kept by `rapidpipe.runs`, retains the source -revision, working-copy changes if any, image digest when applicable, -database schema version, and the resolved settings. Input provenance -identifies exact reference versions and their constituent exposures. +files. The execution record, kept by `rapidpipe.runs`, retains the +source revision, working-copy changes if any, image digest when +applicable, database schema version, and resolved settings. Input +provenance identifies exact reference versions and their constituent +exposures. Each product kind defines the registration metadata its manifest entry must carry. `register` validates that metadata and writes product rows -without reading product contents. It stays independently runnable; -whether it shares a Batch job with a transform changes nothing about -attempt identity, completion or retry safety. +without reading product contents. It remains independently runnable; +sharing a Batch job with a transform changes nothing about attempt +identity, completion or retry safety. ### Exit codes @@ -170,66 +188,61 @@ attempt identity, completion or retry safety. | 70 | Unclassified stage error; stop for investigation | fail, no retry | | 75 | Recognised temporary dependency failure; repeating the same work may succeed | retry within the limit | -These six are the stage subset (`STAGE_EXIT_CODES` in -`rapidpipe/stages/contract.py`) of the one vocabulary in -`rapidpipe/exitcodes.py`, whose full table is on the [tool](tool) page; -a stage never exits 1 or 2. - -Code 69 is `sysexits`' `EX_UNAVAILABLE`, chosen over 64 (which would -misreport a correct invocation as a usage error) and 70 (which calls -for investigation): a stub stage that validates its arguments and -settings and then declines to run is neither of those things. -`disposition_for` maps it to `failed`; the code stays reserved for a -future stage declared but not yet implemented, since no stage in this -build returns it (see [decisions](decisions)). - -The entrypoint maps argument errors to 64 and unhandled exceptions to -70. Forced termination is reported by the launcher, not the stage. -Batch does not retry: the rebuild's job definitions carry one attempt. -The launcher retries, each time as a fresh attempt within the run's -maximum attempts per unit, after exit 75 and after the approved -infrastructure failures, a job with no container exit code whose -status reason starts `Host EC2` (the host was reclaimed) or whose -container reason starts `Cannot` or `DockerTimeoutError` (the container -never started). The launcher also records other terminations without a -stage exit code, and unexpected codes, as not retried. - -### Settings - -Each stage ships default settings and a settings schema under -`settings/.toml`. `--settings` supplies an overlay: tables merge -recursively, supplied scalar and array values replace defaults. Unknown -keys and invalid values fail with code 64. Before execution the runner -retains the resolved settings and their canonical hash. Algorithm -parameters, including random seeds where used, belong in settings; -deployment locations and credentials do not. +These six codes are the stage subset (`STAGE_EXIT_CODES` in +`rapidpipe/stages/contract.py`) of the vocabulary in +`rapidpipe/exitcodes.py`. The [tool](tool) page has the full table; +a stage never exits 1 or 2. The entrypoint maps argument errors to 64 +and unhandled exceptions to 70. + +Code 69 is `sysexits`' `EX_UNAVAILABLE`. A stub stage validates +arguments and settings, then declines to run: 64 would misreport a +correct invocation as a usage error, while 70 calls for investigation. +`disposition_for` maps 69 to `failed`. No stage in this build +returns it; it remains reserved for a future stage declared but not +yet implemented (see [decisions](decisions)). + +The launcher reports forced termination. Batch does not retry: +the rebuild's job definitions carry one attempt. The launcher retries +as a fresh attempt, within the run's maximum attempts per unit, after +exit 75 or an approved infrastructure failure. These are jobs with no +container exit code whose status reason starts `Host EC2` (the host +was reclaimed) or whose container reason starts `Cannot` or +`DockerTimeoutError` (the container never started). It records other +terminations without a stage exit code, and unexpected codes, as not +retried. ## Local execution -Each stage fixture under `tests/fixtures//` includes minimal input -files, settings, required database seed data, and expected products -with documented comparison tolerances, shipped as package data at under -1 MB. `make stage-` prepares an isolated fixture, runs the stage -without account credentials, and checks its products and manifest; -provenance fields are validated for shape, not compared with fixed IDs -or paths. `make db` starts a local PostgreSQL with Q3C and applies the -migrations; fixture setup loads the seed data. CI uses the same -commands. The same fixture also runs through `rapidpipe selftest ---stage ` on Batch, as an ordinary job against the deployed image -and digest; the submitted job's -execution record is the evidence that it ran, distinct from the -real-tool fixture gate below. Each rebuilt science -stage also has an IMSS comparison on fixed inputs, with differences and -tolerances approved by the team before operational use. +Each stage fixture under `tests/fixtures//` includes minimal +input files, settings, required database seed data, and expected +products with documented comparison tolerances. It ships as package +data under 1 MB. + +`make db` starts a local PostgreSQL with Q3C and applies the +migrations; fixture setup loads the seed data. `make stage-` +prepares an isolated fixture, runs the stage without account +credentials, and checks its products and manifest. Provenance fields +are validated for shape, not compared with fixed IDs or paths. CI uses +the same commands. + +The same fixture also runs through `rapidpipe selftest +--stage ` +on Batch as an ordinary job against the deployed image and digest. +The submitted job's execution record is evidence that it ran, distinct +from the real-tool fixture gate below. Each rebuilt science stage also +has an IMSS comparison on fixed inputs, with differences and tolerances +approved by the team before operational use. ## What this replaces -The per-stage scripts on `dev` (`awsBatchSubmitJobs_*`, the launcher and -register scripts, the operator loop) and the `smdc` branch's single -dispatching entrypoint with manifest-driven job types. The launcher -enforces dependencies and controls retries. `rapidpipe.runs` owns attempt -allocation and the execution record. Stages execute the supplied attempt -and report its outputs. +This contract replaces the per-stage scripts on `dev` +(`awsBatchSubmitJobs_*`, the launcher and register scripts, the +operator loop) and the `smdc` branch's single dispatching entrypoint +with manifest-driven job types. + +The launcher enforces dependencies and controls retries. +`rapidpipe.runs` owns attempt allocation and the execution record. +Stages execute the supplied attempt and report its outputs. ## Not decided here