From 4dda28fbb6de985cd197e1c6c0c1965f733958f0 Mon Sep 17 00:00:00 2001 From: Ben Rusholme Date: Mon, 28 Sep 2026 11:41:06 -0700 Subject: [PATCH] releases, tool, operations: clarity pass (Codex) Co-Authored-By: Claude Opus 5.5 --- system/operations.md | 338 +++++++++++++++++++++---------------------- system/releases.md | 241 ++++++++++++++---------------- system/tool.md | 257 ++++++++++++++++---------------- 3 files changed, 397 insertions(+), 439 deletions(-) diff --git a/system/operations.md b/system/operations.md index 85878c2..ee1a95c 100644 --- a/system/operations.md +++ b/system/operations.md @@ -1,25 +1,22 @@ - # Operations: live processing, reprocessing and development together -**Status: DRAFT, a proposal.** This page holds operations proposals the -team has not yet ruled on; what landed is stated on [loop](loop), -[products](products), [runs](runs) and [checks](checks). Nothing on this -page changes code, schema or another page's rules. The -[specification](specification), [runs](runs), [products](products) and -[loop](loop) pages remain the design authority; where this page proposes -a change to them, it says which sentence would change. +**Status: DRAFT, a proposal.** The team has not yet ruled on these +operations proposals. Landed behaviour is on [loop](loop), +[products](products), [runs](runs) and [checks](checks). This page changes +no code, schema or other page's rules. [Specification](specification), +[runs](runs), [products](products) and [loop](loop) remain the design +authority; proposed changes identify the sentence they would change. ## What this page assumes about the mission -Three mission interfaces are not yet defined ([specification](specification), -Edges). This page assumes the following, and the recommendation is -conditional on them. +Three mission interfaces remain undefined ([specification](specification), +Edges). The recommendation depends on the following assumptions. - **Delivery manifest.** Each delivery arrives with a manifest naming, per file, the exposure, detector, delivered version and checksum: the - shape `admit`'s delivery manifest already reads ([products](products), - the l2 image). Where the files land and how RAPID learns of them is a - location the stream lists, not a message it must receive. + shape `admit` already reads ([products](products), the l2 image). + RAPID discovers files by listing their delivery location; it does not + need a message. - **Re-delivery signalling.** A corrected file carries a higher delivered version for the same exposure and detector. An identical re-delivery carries the same version and the same checksum. The same version with @@ -28,7 +25,7 @@ conditional on them. provides one (a per-date manifest closing the day), the stream can use it; the recommendation below does not depend on it. - **Latency.** The provisional requirement is within an hour per - detector image, from delivery to its alert current + detector image, from delivery until its alert is current ({ref}`latency ruling `); the team confirms or replaces the number. A different number changes the batch size, not the shape (Closure and latency, below). @@ -37,131 +34,120 @@ conditional on them. ### The model as written: a run per date, the loop is the continuity -Plain sentence: each processing date is one production run, and the loop -walks the dates a spec names, binding each date's catalog to the -previous date's. +Each processing date is one production run. The loop walks the dates +a spec names, binding each date's catalog to the previous date's. -Nothing changes in code. What strains: a late delivery has no run to -enter; a date must be complete before its run is created, which trades -latency for completeness with no signal to decide on; a corrected frame -makes a new logical key, so its promotion adds a second current product -instead of replacing the first; a reprocessing campaign's promotions add -beside production's for the same reason; and outage catch-up needs a -person to write the dates into a spec. +No code changes, but a late delivery has no run to enter. A date must be +complete before its run is created, trading latency for completeness +without a signal to decide on. A corrected frame makes a new logical +key, so promotion adds a second current product; reprocessing promotions +likewise add beside production's. Outage catch-up needs a person to +write the dates into a spec. ### One long-lived operations run -Plain sentence: all operations processing is one run that grows as -deliveries arrive. - -This breaks more than it fixes. Freeze-at-creation goes: the run's input -set changes after creation, which is exactly the provenance the -specification's Runs section exists to protect. Promotion's default unit, -a whole run, becomes meaningless, so every promotion is piecemeal. A -release change would put two releases inside one run, against -[releases](releases)' "a run never spans a release". `run create --seed ---only-failed` would seed a replacement for the whole of operations -rather than for one date. The per-date association base becomes a -same-run read, which skips the selected-attempt rule the cross-run read -applies. Checks and alert provenance lose their batch: every alert names -the same run, so the run no longer says which code and settings made it. -Scratch expiry is untouched (a production run never expires) and -development is untouched (it still needs its own scratch runs), so the -cost buys nothing for those modes. Rejected. +All operations processing is one run that grows as deliveries arrive. +This shape is rejected. + +Changing the input set after creation loses the frozen provenance +protected by the specification's Runs section. Whole-run promotion +becomes meaningless, so every promotion is piecemeal. A release change +puts two releases in one run, against [releases](releases)' "a run never +spans a release". `run create --seed +--only-failed` seeds a replacement for all operations instead of one +date. The per-date association base becomes a same-run read, skipping +the cross-run selected-attempt rule. + +Checks and alert provenance lose their batch. Every alert names the +same run, which no longer identifies the code and settings that made +it. Neither scratch expiry nor development benefits: production runs +never expire, and development still needs its own scratch runs. ### A named stream whose runs are its batches -Plain sentence: operations is a named stream that turns each batch of new -deliveries into one ordinary frozen run. +Operations is a named stream that turns each batch of new deliveries +into an ordinary frozen run. It uses the loop's existing schedule +identity: the `processing-date-loop` registry row, its spec and its +`loop_dates` rows. -The stream is the schedule identity the loop already has: the -`processing-date-loop` registry row, its spec, and its `loop_dates` -rows. What changes: the stream discovers deliveries it has not yet -admitted instead of reading them from the spec; a date may take more -than one batch (Closure and latency); promotion supersedes by science -slot (Replacement scope); and publication follows the eligibility -table on [products](products), "Reading across runs". What does not -change: runs, units, attempts, custody, the promotion lock, release binding, recovery, the base-catalog -rule, deletion and the scratch path. +The stream discovers unadmitted deliveries instead of reading them +from the spec. A date may have several batches (Closure and latency); +promotion supersedes by science slot (Replacement scope); publication +follows [products](products), "Reading across runs", and its eligibility +table. Runs, units, attempts, custody, the promotion lock, release +binding, recovery, the base-catalog rule, deletion and the scratch path +stay unchanged. ## Recommendation, and where the first cuts differ -The named stream of batch runs. It keeps every boundary the run model -already draws, one release, one frozen input set, one promotion, one -recovery scope per run, and adds only the continuity operations needs. - -Three points settle how the shape behaves: +The recommendation is a named stream of batch runs. Each run keeps one +release, frozen input set, promotion and recovery scope. Continuity uses +the existing schedule identity and rows, with no new service, table or +scheduler. -- On a corrected frame, the repair replays the descendants, and the - replay may be batched. The corrected frame's own products wait for the - same switch rather than being promoted alone: promoting the frame at - once and deferring the catalog publishes both versions in one catalog. -- A date admits several batches when the latency requirement is shorter - than a day, by the same mechanism as a new date. -- The stream is thin configuration: the existing schedule identity and - its rows, with no new service, table or scheduler. +A correction replays the frame's descendants, possibly in batches. +The frame's own products wait for the same switch: promoting the frame +before the catalog would publish both versions in one catalog. When +latency must be shorter than a day, a date admits several batches by +the same mechanism as a new date. ## Replacement scope Promotion supersedes by science slot. Each kind's identity key and slot -are on [products](products), "Identity"; slot replacement, including the -rule that an `association-set` promotion replaces only an ancestor of the -set it promotes, is on [runs](runs), "Rules". What stays open here is -the replacement the chain switch below makes: - -- **A new delivered version** of (E, D): the `l2-image` slot (E, D) and - every slot keyed on (E, D), and each affected field's catalog slots, - all in the one chain-switch promotion a correction run ends in. - -One atomic before and after, for an overlap of live and reprocessing: -the campaign's switch promotion lists, per slot, the expected current -instance (the stream's) and its replacement (the campaign's). If the stream promoted a new -batch between the campaign's plan and its promotion, at least one -expected instance no longer matches, the whole switch is refused, and -the campaign replans against the new selection (stale-plan refusal, as -the promotion lock does today). Rollback is the recorded inverse, refused -if the switch's after-selection is no longer current, as today. +are on [products](products), "Identity". [Runs](runs), "Rules", defines +slot replacement, including the rule that an `association-set` +promotion replaces only an ancestor of the set it promotes. + +The chain switch below remains open. For a new delivered version of +(E, D), one promotion at the end of a correction run replaces the +`l2-image` slot (E, D), every slot keyed on (E, D), and each affected +field's catalog slots. + +When live processing and reprocessing overlap, the campaign's switch +lists each slot's expected current instance (the stream's) and +replacement (the campaign's) in one atomic promotion. If the stream +promotes a batch after the campaign plans its switch, at least one +expected instance no longer matches. The whole switch is refused, and +the campaign replans against the new selection, as today's promotion +lock refuses stale plans. Rollback applies the recorded inverse and, +as today, refuses if the switch's after-selection is no longer current. ### The chain switch -One mechanism serves both a reprocessing campaign and a correction run: -the switch that moves a stream from one catalog chain to another. +A chain switch moves a stream to another catalog chain after either +a reprocessing campaign or a correction run. A reprocessing campaign is a set of ordinary production runs under a new -release or new settings, auto-promote off, on the bulk queue (Capacity, -below). A correction run is an ordinary production run seeded on each -affected field's last association set before a corrected frame's epoch; -it replays the field's batches up to the stream's head with the -corrected version in place, regenerating the association, statistics -and pruned sets and the alert containers of the frames those fields -hold. Corrections may be batched into one correction run on a schedule -the team sets; until the switch, consumers keep the earlier selection. -A frame is *awaiting correction* when the stream has admitted a higher -delivered version for its (exposure, detector) than the version the -current catalog of its field holds, and no chain switch has applied it -yet; only the stream's own admissions count. +release or new settings, with auto-promote off, on the bulk queue +(Capacity, below). A correction run is an ordinary production run +seeded on each affected field's last association set before the +corrected frame's epoch. It replays the field's batches to the stream's +head with the corrected version, regenerating association, statistics +and pruned sets and the alert containers for frames in those fields. + +Corrections may share a run on a schedule the team sets. Consumers keep +the earlier selection until the switch. A frame is *awaiting correction* +when the stream has admitted a higher delivered version for its +(exposure, detector) than its field's current catalog holds, and no +chain switch has applied it. Only the stream's own admissions count. 1. The replacing run (a campaign or a correction run) has processed every delivery up to the stream's most recent complete batch. 2. It takes the stream's schedule lock, the advisory lock `loop run` - already holds for its whole run, so no new batch starts during the - switch; if a batch completed after step 1, it processes that batch's - deliveries first, still under the lock. -3. No batch of the stream may be left open on the old chain, because its - frozen inputs bind the old head and a resumed batch would extend it. - Before the switch, each open, failed or timed-out batch is either - finished (and then covered as in step 2) or cancelled, with its - deliveries recorded as not processed, so the replacing run's catch-up - takes them. The switch refuses while any batch of the schedule is - open. -4. One promotion replaces every slot the replacing run covers, file - products and result sets together, each against its expected - predecessor. The boundary, the stream's last batch covered, is in the - promotion's request context. -5. In the same transaction the stream's records gain a switch row naming - each field's adopted head. The base-catalog rule reads the most recent - complete row or switch row of the schedule, so the stream's next batch - extends the adopted head. + already holds throughout its run. No new batch can start during the + switch. If a batch completed after step 1, the replacing run first + processes its deliveries under the lock. +3. Every open, failed or timed-out batch must finish (and be covered as + in step 2) or be cancelled with its deliveries recorded as not + processed, so the replacing run catches them up. The switch refuses + while any batch of the schedule is open: its frozen inputs bind the + old head, which it would extend on resuming. +4. One promotion replaces every covered slot against its expected + predecessor, file products and result sets together. Its request + context records the boundary: the stream's last batch covered. +5. The same transaction adds a switch row naming each field's adopted + head. The base-catalog rule reads the schedule's most recent complete + row or switch row, so the next batch extends the adopted head. A person makes the switch; it is the one promotion into a catalog slot that is not an extension of the current chain. Its rollback restores the @@ -171,10 +157,10 @@ date can hold several batches) and a row kind (`batch` or `switch`). ## Closure and latency -The arrival outcomes, a later batch for the same date, a refused -identical re-delivery, a quarantined checksum mismatch and a deferred -corrected version, are on the [loop](loop) page, "Discovery and -batches". The batch cadence stays open. The provisional latency, within +[Loop](loop), "Discovery and batches", states the arrival outcomes: +a later batch for the same date, refusal of an identical re-delivery, +quarantine of a checksum mismatch, and deferral of a corrected version. +Batch cadence remains open. The provisional latency, within an hour per detector image ({ref}`latency ruling `), points to per-delivery batches with an event or roughly 30-minute window trigger. @@ -182,18 +168,17 @@ trigger. If the latency requirement is a day or more, one batch per date after a fixed cutoff hour is enough, and the stream is the loop as written plus discovery. If it is minutes, batches become per delivery and the trigger -becomes an event rather than a window. The run model's shape holds; whether -the stages and the queues keep up at that scale is unverified (the -`difference` stage alone runs about 58 minutes per detector image today; -Capacity, below), and a chain switch holds the schedule -lock while it catches up, which pauses live batches for that time. The page does not declare operations ready -until the team confirms the latency and the mission states the delivery -interface. +becomes an event rather than a window. The run model still holds, but +whether stages and queues can keep up is unverified: `difference` +alone takes about 58 minutes per detector image today (Capacity, +below). A chain switch also pauses live batches while it catches up +under the schedule lock. Operations are not declared ready until the +team confirms the latency and the mission states the delivery interface. ## The loop's five calls -Of the loop's five calls, two stay open here; the others are on the -[loop](loop) page. +Two of the loop's five calls remain open here; [loop](loop) covers the +others. - **Cadence** (proposed): a fixed `window:cron(...)` at an interval of half the latency budget; a firing that discovers nothing exits 0 and @@ -203,19 +188,17 @@ Of the loop's five calls, two stay open here; the others are on the ## A production check policy -Proposed as `rebuild-production@1`, with `auto_promote = true` -requested. Approval semantics: a policy is immutable once landed -([checks](checks)), so the file lands once, already carrying the -team's approval, recorded as `approval = "team"` with `approved_by` set -to the approver's login, in a pull request the team approves; until -then its content lives only in the table below, and no file for it -exists to be edited. No other path raises a policy to `team`. -`rebuild-trial@1` stays at trial approval. Team approval of this policy -is pending. +The proposed `rebuild-production@1` requests `auto_promote = true`; +team approval is pending. A policy is immutable once landed +([checks](checks)), so its file must land in a team-approved pull +request carrying `approval = "team"` and `approved_by` set to the +approver's login. Until then, its content exists only in the table below, +with no file to edit. No other path raises a policy to `team`. +`rebuild-trial@1` keeps trial approval. -Bounds from real data: the 15357 difference images `dev` recorded in -production `diffimmeta`, taking each measure's 1st to -99th percentile and widening by half: +Proposed bounds use the 15357 difference images `dev` recorded in +production `diffimmeta`: each measure's 1st to 99th percentile, +widened by half. | Measure | 1st, 50th, 99th percentile | Proposed bound | |---|---|---| @@ -229,58 +212,63 @@ production `diffimmeta`, taking each measure's 1st to `scalefacref` depends on the reference zero point gain matching uses. `dev`'s values come from a fixed `zprefimg` of 17.0; under the ruling that gain matching reads `MAGZP` from the reference, -the factor moves to order one, so its bound is set from the first -production batches under that ruling, not from `dev`. The catalog-count check -stays advisory until earlier production runs have made current source -sets for the same frames to compare against; it compares with another -run's source set, never with a reference catalog ([checks](checks)). +the factor moves to order one. Its bound therefore comes from the first +production batches under that ruling, not from `dev`. + +The catalog-count check stays advisory until earlier production runs +provide current source sets for the same frames. It compares another +run's source set, never a reference catalog ([checks](checks)). ## Reference eligibility and selection -Proposed: a reference image is eligible for a field and filter when it -is current in that slot, built by the recipe the stream's spec names, -from constituents that are all current or superseded. Selection is then -trivial: the one current reference in the (field, filter) slot, which is -`dev`'s rule (the reference with `vbest` set). A new reference promoted -mid-stream is used by later batches; difference images made against the -earlier one keep it, as their identity key says, until a reprocessing -replaces them. Where the reference PSF is resolved from stays open. +Proposed: a reference image is eligible for a field and filter if it is +current in that slot, built by the recipe named in the stream's spec, +and made from constituents that are all current or superseded. Selection +takes the one current reference in the (field, filter) slot, following +`dev`'s rule (the reference with `vbest` set). + +Later batches use a new reference promoted mid-stream. Existing +difference images retain the earlier reference in their identity key +until reprocessing replaces them. Where the reference PSF is resolved +from remains open. ## Staged inputs of a production run -A production run stages its difference input set, a copy of the l2 -image, reference bundle and PSFs plus the manifest, under the scratch -bucket's `rapidpipe/runs//inputs/`, because the launcher -host cannot write the products bucket and [tool](tool) puts every input -set under the scratch root. Proposed run-model rule: a production run's -staged input sets are part of its provenance and are retained for the -run's lifetime; nothing expires or deletes them while the run's row -exists. That holds today without a change: scratch expiry and `run -delete` act only on scratch runs, and the scratch bucket has no -lifecycle rule. The rule is written down so a later -lifecycle rule does not quietly break it. +A production run stages its difference input set under the scratch +bucket's `rapidpipe/runs//inputs/`: copies of the l2 image, +reference bundle and PSFs, plus the manifest. The launcher host cannot +write the products bucket, and [tool](tool) puts every input set under +the scratch root. + +Proposed run-model rule: staged production inputs are provenance, +retained for the run's lifetime. Nothing expires or deletes them while +the run's row exists. This already holds: scratch expiry and `run +delete` affect only scratch runs, and the scratch bucket has no +lifecycle rule. Stating the rule protects it against a later lifecycle +rule. ## Capacity Two queues exist: `rapid-queue-prompt` (priority 10, on-demand `rapid-ce-prompt`, 5000 vCPU) and `rapid-queue-bulk` (priority 1, Spot `rapid-ce-bulk`, 10000 vCPU). Measured rebuild jobs put `difference` at -a median of about 58 minutes of execution per detector image, the only stage over the 30-minute target; -`alerts` ran about 25 minutes on production runs, and every other stage -under a minute. Proposed lanes: the stream on prompt; reprocessing -campaigns and correction runs on bulk; scratch runs on bulk by default, -prompt by request. That is the specification's "scratch runs cannot -consume capacity reserved for regular operations" with the queues that -already exist. +a median of about 58 minutes of execution per detector image, the only +stage over the 30-minute target. `alerts` ran about 25 minutes on +production runs; every other stage took under a minute. + +Proposed lanes use these queues to meet the specification's "scratch +runs cannot consume capacity reserved for regular operations": the +stream uses prompt; reprocessing campaigns and correction runs use +bulk; scratch runs use bulk by default, prompt by request. ## Not decided here - Confirmation of the provisional delivery-to-alert latency ({ref}`latency ruling `), and the mission's delivery manifest and completeness signal. -- How often correction runs run: per correction, or batched on a - schedule; the team's. -- Whether alert history for a late-arriving earlier epoch is ordered by - observation time or by arrival; the team's. +- The team's choice of correction cadence: per correction or batched + on a schedule. +- The team's choice of alert-history order for a late-arriving earlier + epoch: observation time or arrival. - Team approval of `rebuild-production@1` and its `scalefacref` bound. - Where the reference PSF is resolved from. diff --git a/system/releases.md b/system/releases.md index 046b1ca..ba17449 100644 --- a/system/releases.md +++ b/system/releases.md @@ -2,22 +2,13 @@ **Status: DRAFT** -How operations move from a commit to a running, verified version: the -tag, the record that ties a commit to a schema and an image, the -command that cuts one, and what promotion and the launcher require of -it. The [specification](specification)'s Releases section states the -requirement; this page records how the rebuild meets it. - -## In plain terms - -`main` moves every day; operations do not follow it. A release is one -point on that history that has been tagged, migrated, built into an -image and deployed, with a database row recording each step. Cutting a -release is the one operation that performs all of it in order and -refuses to record success until every step's evidence says so. A run -created under a release reads that row, never the branch or the -environment, so what a run used is exactly what the record says it -used. +Operations run a release while `main` moves every day. A release is a +commit that has been tagged, migrated, built into an image and deployed, +with a database row recording each step. Cutting a release performs +those steps in order and records success only when each step has the +required evidence. The [specification](specification)'s Releases +section states the requirement; this page defines the tag, record and +cut, and what the launcher and promotion require of them. ## The command @@ -26,53 +17,48 @@ used. `rapidctl`. The full flag surface for `cut`, `show`, `list` and `verify` is on the [tool](tool) page. The Python interface is `rapidpipe.release.core`: `next_tag`, `cut`, `show` and `verify`, and -the `Release` dataclass the record above maps onto. +the `Release` dataclass that represents the release record. ## The tag -A release tag is annotated, on the `rebuild` branch, named -`rebuild-v0.` where `n` is one more than the highest tag already -pushed; remote tags are authoritative, not a local checkout's. The -scheme becomes `v1.` once the rebuild replaces -`main`, the specification's own sequencing point for that move. - -The tag message is the release's initial manifest: the tag, the source -revision it points at, the schema version the tagged tree carries, and -who cut it and when. Once written it is never amended, moved or -deleted; everything a cut learns afterwards, images built and -consumers deployed, lives in the release record instead, not in the -tag. The image registry tag equals the git tag, and registry tag -immutability then does the same job on the image side: a second build -under the same tag fails outright, which is the immutability the -specification asks for without any check the cut has to write itself. +A release tag is annotated, on the `rebuild` branch, and named +`rebuild-v0.`. Here `n` is one more than the highest tag already +pushed: remote tags are authoritative. The scheme becomes `v1.` +once the rebuild replaces `main`, as sequenced in the specification. + +The tag message is the release's initial manifest: the tag, its source +revision, the tagged tree's schema version, and who cut it and when. +Once written, the tag is never amended, moved or deleted. Later +evidence, including images built and consumers deployed, goes in the +release record. The image registry tag equals the git tag. Registry +tag immutability makes a second build under the same tag fail outright, +meeting the specification's requirement without a separate check in the +cut. ## The record -Two tables carry a release's state. `releases` -holds one row per tag: source revision, schema version, image digest, -image reference, a state that advances `migrated`, `built`, -`deployed`, `complete`, who cut it and when, and when it completed. -`release_deployments` holds one row per consumer a release reaches: -the release, the consumer, the job definition name and revision -deployed, and who deployed it and when. - -A run created under a release copies its source revision and image -digest from this record; it reads nothing from git or the environment -to get them. An attempt's execution record carries the release -identity its job definition was deployed under, read from the -deploy-time environment variable `RAPID_RELEASE_IDENTITY`; a job -definition never deployed under a release reports `unreleased`, and -reconcile stores whichever value it finds. +Two tables carry a release's state. `releases` holds one row per tag: +source revision, schema version, image digest, image reference, who cut +it and when, and when it completed. Its state advances through +`migrated`, `built`, `deployed` and `complete`. +`release_deployments` holds one row per consumer the release reaches: +the release, consumer, deployed job definition name and revision, and +who deployed it and when. + +An attempt's execution record carries the release identity its job +definition was deployed under, read from the deploy-time environment +variable `RAPID_RELEASE_IDENTITY`. A job definition never deployed +under a release reports `unreleased`; reconcile stores whichever value +it finds. ## The order of a cut `cut` runs six steps in order: tag, migrate, record, build, deploy, -pins. The account-specific steps, migrate, build, deploy and pins, are -hooks: executables the systems repository supplies, which `cut` invokes -in order with the release's tag, source revision and schema version in -their environment. The pipeline repository -itself names no account, host or bucket; everything account-specific -lives in the hook, on the other side of that boundary. +pins. The systems repository supplies executables, called hooks, for +the account-specific steps: migrate, build, deploy and pins. `cut` +invokes them in order with the release's tag, source revision and schema +version in their environment. All account-specific details live in the +hooks; the pipeline repository names no account, host or bucket. Each hook reports its result as one JSON object on its last line of output, and `cut` reads only that line: @@ -84,39 +70,62 @@ output, and `cut` reads only that line: | deploy | each consumer's job definition revision | | pins | the number of rows written | -Hooks are idempotent, so `--resume TAG` re-enters a cut at the state -its record already reached without re-tagging; `cut` itself never -retries a hook; a hook that fails leaves the record at its last -completed step for a person to fix and resume. `--skip HOOK` omits one -step for a cut whose account side was already done by hand; `--dry-run` -prints the planned order and each hook's command without running any of -them or writing the record. +Hooks are idempotent. `--resume TAG` re-enters a cut at its recorded +state without re-tagging. `cut` never retries a hook: a failure leaves +the record at its last completed step for a person to fix and resume. +`--skip HOOK` omits one step whose account side was already done by +hand. `--dry-run` prints the planned order and each hook's command +without running them or writing the record. + +The final pins step appends one row per rebuild consumer to the +operations pin table. It reads each consumer's live job definition and +records the release identity beside it, so the database states which +release is deployed. The existing daily pin sweep over the legacy +pipeline's consumer set is unchanged; the cut's pin step covers the +rebuild's consumers alongside it. + +CI on `rebuild` gates the source before tagging. The cut stops at +deploy and pins. Selftests run afterwards on Batch under the newly +deployed revision as evidence that the image behaves; the cut does not +wait for them. ## Migrations at release time -A release's schema version is the greatest migration filename present -in its tagged tree. The migrate hook applies that tree's migrations to -the release's database target, and `cut` refuses to record the release -until every one of those files is applied with the checksum the -migration applier recorded for it. Migration -therefore always precedes the image deploy in a cut, so a job -definition revision never runs code ahead of the schema it expects. A -run started without a release still submits by the unversioned job -definition name and may pick up a revision a later deploy repoints it -to, the specification's "scratch runs may use any commit" carried -through to the job definition itself. +A release's schema version is the greatest migration filename in its +tagged tree. The migrate hook applies that tree's migrations to the +release's database target. `cut` refuses to record the release until +every file is applied with the checksum the migration applier recorded +for it. Migration always precedes image deployment, so a job definition +revision never runs code ahead of its expected schema. + +Migrations must stay additive (new tables and nullable columns; no drop +or rename while an earlier release's runs are open) and compatible with +the readers and writers of every release whose runs remain open. This +is a constraint of the migration rule. Together with the fixed release +binding below, it makes cutting a release while other runs remain open +safe. ## The launcher reads the release -A run created under a release submits every unit to the job definition -revision that release's `release_deployments` row records for the -run's kind, verified `ACTIVE` before submission; there is no fallback -to whatever revision is latest. That makes a -job definition revision something operations must keep alive past its -own release: a later cut's deploy step repoints the job definition to a -new revision, and a run still reading the earlier release needs the -earlier revision to still exist. Retention of superseded revisions is a -systems-repository concern, not a pipeline one. +A run's release is fixed at creation by `run create --release`. It +copies the source revision and image digest from the release record, +without reading git or the environment. Every unit submits to the job +definition revision that the release's `release_deployments` row +records for the run's kind, verified `ACTIVE` before submission. There +is no fallback to the latest revision. + +A later cut repoints the job definition to a new revision, so operations +must keep earlier revisions alive for runs still using them. +`SkipDeregisterOnUpdate` keeps a revision `ACTIVE` past its own +release. Retention of superseded revisions belongs to the systems +repository. Every attempt reads the same release's recorded revision: +a run never picks up a later cut mid-flight, and its execution records +and `schema_version` all carry the release it was created under. + +A run started without a release submits by the unversioned job +definition name and may pick up a revision a later deploy repoints it +to. This carries the specification's "scratch runs may use any commit" +through to the job definition. The processing-date loop's binding to a release is resolved on the [loop](loop) page: the loop's spec names the release, and the scheduled @@ -124,62 +133,28 @@ operation checks that tag out before running each date. ## Promotion eligibility -Promotion checks executed provenance, not a tag on a commit: every -deliverable's selected producing attempt must have an execution record -whose image digest matches a complete release's digest, and whose -recorded release, when it has one, is that release's tag. A deliverable -that fails this refuses promotion, unless the operator passes the -explicit unreleased exception, which the promotion's request context -then carries. This closes the trial exception -the [runs](runs) page recorded earlier: promotion no longer treats a -missing released-image check as passing by default, it treats it as -refused unless waived. - -## Deployed pins - -The last step of a cut appends one row per rebuild consumer to the -operations pin table, reading each consumer's live job definition and -writing the release identity beside it, so which release is deployed -has a database answer rather than a person's memory of the last deploy. -The daily pin sweep that already runs against -the legacy pipeline's own consumer set is unchanged; a cut's pin step -covers the rebuild's consumers alongside it. - -## Selftests are not part of a cut - -CI on `rebuild` gates the source before any tag is cut; a cut's own -steps stop at deploy and pins. Selftests run on Batch under the newly -deployed revision afterwards, as evidence that the deployed image -behaves, not as a gate the cut itself waits on. - -## A run never spans a release - -A run's release is fixed at its creation, by `run create --release`, -and every attempt it submits reads that same release's recorded -job-definition revision -- kept `ACTIVE` past its own release by -`SkipDeregisterOnUpdate` -- so a run never picks up a later cut's -revision mid-flight, and its execution records and `schema_version` -all carry the one release it was created under. This is what makes -cutting a release while other runs are still open safe, provided -migrations stay additive (new tables and nullable columns; no drop or -rename while an earlier release's runs are open) and compatible with -the readers and writers of every release whose runs are still open. -That additivity and compatibility are now a stated constraint of the -migration rule above, not an assumption a concurrent cut could quietly -violate. +Promotion checks executed provenance. Every deliverable's selected +producing attempt must have an execution record whose image digest +matches a complete release's digest. Its recorded release, when present, +must be that release's tag; a tag on a commit alone does not suffice. +Failure refuses promotion unless the operator passes the explicit +unreleased exception, which is then recorded in the promotion's request +context. This closes the trial exception recorded earlier on the +[runs](runs) page: a missing released-image check refuses promotion +unless waived. ## Concurrent cuts are serialised -`cut` refuses to start, before any fetch or tag, while any `releases` -row is in a state other than `complete`, unless `--resume` names that -row; the message names the tag and its state, exit 1 (a refusal; the -[tool](tool) page's table). The row itself is -still written only after the tag is pushed, so two cuts started in the -same instant can both pass the check before either has a row to be -refused by: this rule serialises through the record once it exists, it -is not a lock, and closing that window is recorded open. A `--resume` -of the row already in flight is the way through a cut that failed -partway, not a second `cut`. +Before any fetch or tag, `cut` refuses to start while any `releases` +row has a state other than `complete`, unless `--resume` names that +row. The refusal names the tag and state and exits 1 (the +[tool](tool) page's table). To continue a cut that failed partway, use +`--resume` for the row already in flight. + +The row is written only after the tag is pushed. Two cuts started at +the same instant can therefore both pass the check before either has a +row. The rule serialises cuts once a record exists; it is not a lock. +Closing that window remains open. ## Not decided here diff --git a/system/tool.md b/system/tool.md index 9af8539..c868eeb 100644 --- a/system/tool.md +++ b/system/tool.md @@ -2,27 +2,20 @@ **Status: DRAFT** -The operations the team runs through `rapidpipe`, mapped from the -specification's Tools section onto its subcommands; the input-set -composer; the tool's own exit codes; personal submission from a -workstation; and where the tool runs. The -[specification](specification)'s Tools section states the requirement; -this page records how the rebuild meets it. - -## In plain terms - -The team touches one command-line tool: `rapidpipe`, shipped in the -pipeline repository and already the image's entrypoint for a stage -invocation. There is no second binary: a `rapidctl` would contradict the -specification's own sentence naming one tool, and would split an -entrypoint the container already carries. Everything below is `rapidpipe`'s command surface; nothing -here is a separate program. +`rapidpipe` is the team's single command-line tool, shipped in the +pipeline repository and already the image's entrypoint for stage +invocations. It meets the [specification](specification)'s Tools +requirement. A second binary, `rapidctl`, would contradict that +requirement and split the container's existing entrypoint. + +This page maps operations to subcommands and describes input-set +composition, exit codes, workstation submission and where the tool runs. +Every command below belongs to `rapidpipe`. ## Operations -The specification's Tools section names a small set of operations. Each -maps onto one or more subcommands, all existing arguments keeping their -names: +The specification's Tools operations map to these subcommands. Existing +arguments keep their names. | Operation | Subcommand | |---|---| @@ -60,88 +53,94 @@ The check, release and loop commands take these arguments: | `loop plan --spec ` | Prints what `loop run` would do: the dates it would process and the runs it would create, without creating them; given an inbox spec, it discovers and classifies without writing any row | | `loop show ` | Prints the schedule's `loop_dates` rows, then its `loop_deliveries` rows | -The check commands run launcher-side, on a workstation or the launcher -host, reaching the database through the instance role; none of them -touches the pipeline image. +The check commands run launcher-side, on a workstation or launcher host. +They reach the database through the instance role and do not touch the +pipeline image. + +## Starting stages `run start --unit [--stage ]` walks the run's selected -stages in order, or the one named stage: a complete unit is skipped; a -unit already terminal `failed` or `cancelled` is not re-attempted: -`start` prints its state and exits 1, and recovering it is -`run create --seed --only-failed`; a unit with -an attempt still running is attached to and polled, never resubmitted; -and a unit reconcile returns to `ready` after a transient result (exit -75, or a lost job) gets another attempt within its allowance. `run -cancel` followed by `run start` is how a person restarts one attempt by -hand. A run created with `--release ` submits to that release's -recorded job-definition revisions rather than whatever a job definition -currently pins, and its outputs are promotable without the -unreleased-image exception ([releases](releases) page). Each stage's -inputs resolve in this order: an explicit `--inputs` on the command -line, then a `--template` composed through `run inputs` (below), then -the nearest preceding non-`register` stage's selected output (`register` -produces database rows, not files, so a stage that follows it reads -`difference`'s output instead). `--no-wait` submits the first runnable -stage and prints the command that continues the walk; otherwise `start` -polls reconcile until the unit is terminal, printing one line per -attempt (stage, unit, attempt, job, disposition, outputs). - -`stage run …` and `stage …` take the one frozen invocation -form the [stage contract](stage-contract) page describes -(`--run --unit --attempt --inputs --outputs [--settings] [--dry-run]`); -the launcher's own submission uses that exact form. +stages in order, or just the named stage. It skips complete units and +attaches to running attempts to poll them without resubmitting. For a +unit already terminal `failed` or `cancelled`, `start` prints the state +and exits 1 without another attempt. Recovery uses +`run create --seed --only-failed`. A unit that reconcile returns +to `ready` after a transient result (exit 75 or a lost job) gets another +attempt within its allowance. `run cancel` followed by `run start` +restarts one attempt by hand. + +A run created with `--release ` submits to that release's recorded +job-definition revisions rather than the current pins. Its outputs are +promotable without the unreleased-image exception +([releases](releases) page). + +Each stage resolves inputs in this order: explicit command-line +`--inputs`, a `--template` composed through `run inputs` (below), then +the nearest preceding non-`register` stage's selected output. +`register` produces database rows, not files, so a stage following it +reads `difference`'s output instead. + +`--no-wait` submits the first runnable stage and prints the command that +continues the walk. Otherwise, `start` polls reconcile until the unit +is terminal, printing one line per attempt: stage, unit, attempt, job, +disposition, outputs. + +`stage run …` and `stage …` take the frozen invocation +form described on the [stage contract](stage-contract) page +(`--run --unit --attempt --inputs --outputs [--settings] [--dry-run]`). +The launcher's submission uses that exact form. ## Input sets `run inputs --unit --from-stage --template [--dest ]` -is the input-set composer. It reads the template's manifest, replaces or -adds the entry of the producer's output kind with that producer's -selected output, copies that member and the template's other members to -``, admits and binds the unit through `bind_input_set` before -writing the composed manifest there last, and prints the location. It -refuses to overwrite a manifest already at ``. - -An input set is a staged working copy, not a product, so `` -defaults to, and `--dest` is refused outside, -`/runs//inputs///`, the scratch outputs -root, for every run kind, since the launcher host has no write access to -the products bucket. Before copying, the composer checks the run's -admission fence: a run that is finished, deleting or already deleted -admits no new input set. `run delete` removes a run's inputs prefix -along with its attempt outputs. - -Every submission binds its input set, not only one composed by this -command. Before any write, `run submit`, `run start` and `run local` -all read `manifest.json` at `--inputs`, whether it came from `run -inputs`, a producing stage's own completion manifest, or a hand-composed -one; collect every output entry's instance and every -`inputs.result_sets` entry (not `inputs.products`, which name what the -*upstream* attempt read, not this unit's own binding); create the unit; -bind the collected names that are registered product instances through -`bind_unit_inputs`; commit both together; and only then allocate the -attempt. A name that is not a registered instance (a delivery manifest, -a dev-era template entry) binds nothing and is logged, not refused. A -manifest that is absent, invalid or unreadable for a non-network reason -refuses the submission before anything is written, exit 65; a network -error is exit 75 instead. Binding is idempotent per (unit, instance), so -a retry or a seeded `--only-failed` re-run binds nothing new. This is -what makes a unit a live consumer of its declared inputs from -submission, not only once it has produced an output of its own to -depend on ([runs](runs) page, "Units", "Deletion"). - -This is the explicit, whole-input-set form of resolution: which -reference among several eligible ones a field should use is not decided -here (below). +composes an input set from a template manifest. It replaces or adds the +entry for the producer's output kind with that producer's selected +output, then copies that member and the template's other members to +``. It admits and binds the unit through `bind_input_set`, writes +the composed manifest last and prints its location. It refuses to +overwrite a manifest already at ``. + +An input set is a staged working copy, not a product. For every run +kind, `` defaults to +`/runs//inputs///` in the scratch +outputs root, and `--dest` is refused outside it: the launcher host +cannot write to the products bucket. Before copying, the composer +checks the run's admission fence. Finished, deleting and deleted runs +admit no new input sets. `run delete` removes the run's inputs prefix +and attempt outputs. + +Every submission binds its input set. Before writing anything, +`run submit`, `run start` and `run local` read `manifest.json` at +`--inputs`, whether composed by `run inputs`, supplied as a producing +stage's completion manifest or composed by hand. They collect every +output entry's instance and every `inputs.result_sets` entry. +`inputs.products` names what the *upstream* attempt read and does not +contribute to this unit's binding. + +The commands create the unit, bind the collected registered product +instances through `bind_unit_inputs`, commit both together and only +then allocate the attempt. Unregistered names, such as a delivery +manifest or dev-era template entry, bind nothing and are logged without +refusal. An absent, invalid or otherwise unreadable manifest refuses +submission before any write: exit 65 for a non-network reason, exit 75 +for a network error. + +Binding is idempotent per (unit, instance), so a retry or seeded +`--only-failed` re-run binds nothing new. The unit becomes a live +consumer of its declared inputs at submission, before producing an +output that depends on them ([runs](runs) page, "Units", "Deletion"). + +This resolves an explicit whole input set. Which reference a field +should use among several eligible ones remains open (below). ## Exit codes -One vocabulary covers every `rapidpipe` process, in the module -`rapidpipe/exitcodes.py` (`ExitCode`). The [stage contract](stage-contract) -page's Exit codes table carries the six of these a stage itself reports -(0, 64, 65, 69, 70, 75), 69 reserved: no stage in this build returns it. -A stage never exits 1 or 2. The table below is -the full eight, with which command family returns each one folded into -the third column. +Every `rapidpipe` process uses `rapidpipe/exitcodes.py` (`ExitCode`). +The table below lists all eight codes and the command families that +return them. The [stage contract](stage-contract) page's Exit codes +table lists the six stage codes (0, 64, 65, 69, 70, 75), including the +reserved 69, which no stage in this build returns. A stage never exits +1 or 2. | Code | Meaning | Returned by | |---|---|---| @@ -159,34 +158,45 @@ check fails or the stage under test exited 0 where the fixture expected a non-zero code, 64 when the work directory already exists, and otherwise the stage's own unexpected code. -Three invariants hold across the vocabulary. A parse failure exits 64 from every family: every -`rapidpipe` parser, nested subparsers included, is -`rapidpipe.exitcodes.ArgumentParser`, whose usage error exits 64 -directly, and help still exits 0; the stage runner also translates a -parse failure to 64 for `stage run `, as a safeguard. So 2 means -only "still running", never "could not parse". And `release` spells its -own usage and precondition refusals 64, the same as every other family. +Three invariants hold across the vocabulary: parse failures exit 64; +2 means only "still running", never "could not parse"; and `release` +uses 64 for usage and precondition refusals, like every other family. +Every `rapidpipe` parser, including nested subparsers, is +`rapidpipe.exitcodes.ArgumentParser`: usage errors exit 64 directly, +while help exits 0. As a safeguard, the stage runner also translates +parse failures to 64 for `stage run `. + +## Where it runs + +The tool runs launcher-side on a fleet host under that host's instance +role. Named environment variables supply deployment-specific locations, +connection settings and credentials: for example, +`RAPIDPIPE_BATCH_JOB_QUEUE`, +`RAPIDPIPE_BATCH_JOB_DEFINITION_SCRATCH` and `_PRODUCTION`, +`RAPIDPIPE_OUTPUTS_ROOT_SCRATCH` and `_PRODUCTION`, +`RAPIDPIPE_SCRATCH_BUCKET`, `RAPIDPIPE_CLEANUP_ROLE_ARN`, and the +database connection variables (`PG*`, `RAPID_DB_SECRET_ID`). The +pipeline repository's README, "Running on Batch", has the full list. +A launcher-side change, including everything on this page, needs no +image rebuild or job-definition deployment: the container's +`stage ` invocation is unchanged. ## Personal submission from a workstation -A person can run the same commands from a workstation that the launcher -runs from a fleet host. The identities involved are documented in the -systems repository, but the shape reaches this page: the -workstation's own submission role is set as one parameter of the -systems repository's scratch identities, rather than three hand edits -across the policies that name it, so a further per-owner workstation -role is one parameter value. Scratch jobs submitted this way still run -under the scratch job role, exactly as they do from the launcher; only -the submitting identity differs. - -Cleanup (`run delete` and `run expire`) has its own principal, -separate from the general workstation role. When the environment -variable `RAPIDPIPE_CLEANUP_ROLE_ARN` is set, the tool assumes that role -for the deletion; when it is unset, the tool deletes under whatever -credentials are ambient. During the transition, the workstation role -also carries the cleanup permission directly, so a failure of the -assumed path cannot strand a deletion; removing that direct attachment -is open (below). +A person can run the same commands from a workstation. The systems +repository documents the identities and sets the workstation submission +role through one scratch-identities parameter. Adding a per-owner +workstation role takes one parameter value rather than three hand edits +across the policies that name it. Scratch jobs submitted this way still +run under the scratch job role, exactly as they do from the launcher; +only the submitting identity differs. + +Cleanup (`run delete` and `run expire`) uses a separate principal from +the general workstation role. If `RAPIDPIPE_CLEANUP_ROLE_ARN` is set, +the tool assumes that role for deletion; otherwise it uses ambient +credentials. During the transition, the workstation role also carries +cleanup permission directly, so failure of the assumed path cannot +strand a deletion. Removing that attachment remains open (below). `run cancel` needs `batch:TerminateJob` on whichever identity issues it. The workstation's instance role does not carry that permission, so @@ -194,21 +204,6 @@ a cancel issued from a workstation fails `AccessDenied` until the systems repository grants it; the launcher's own fleet-host role is unaffected. -## Where it runs - -The tool runs launcher-side: on a fleet host, under that host's instance -role. Deployment-specific locations, -connection settings and credentials reach it through named environment -variables, for example `RAPIDPIPE_BATCH_JOB_QUEUE`, -`RAPIDPIPE_BATCH_JOB_DEFINITION_SCRATCH` and `_PRODUCTION`, -`RAPIDPIPE_OUTPUTS_ROOT_SCRATCH` and `_PRODUCTION`, -`RAPIDPIPE_SCRATCH_BUCKET`, `RAPIDPIPE_CLEANUP_ROLE_ARN`, and the -database connection variables (`PG*`, `RAPID_DB_SECRET_ID`): the -pipeline repository's README, "Running on Batch", carries the full -list. A launcher-side change, including everything on this page, needs no image -rebuild and no job-definition deployment, since the container's own -`stage ` invocation does not change. - ## Not decided here - The field-level input resolver (which reference a field should use