P1 pilot: add lingcod_msy_recovery.csv with its manifest#12
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds the first “P1 pilot” dataset to the new published lectures/ tree—lingcod_msy_recovery.csv—together with its sidecar manifest, and updates manifest-schema.yml based on lessons from this first real manifest.
Changes:
- Extend
manifest-schema.ymlto split integrity intomigrationvsupstream, addlicense.verified, addschema.known_nulls, and introducebuilder_status. - Add
lectures/lingcod_msy_recovery.csvand its manifestlectures/lingcod_msy_recovery.csv.yml. - Record the pilot dataset’s provenance, schema, consumers, integrity assessment, and known nulls in the new manifest.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
manifest-schema.yml |
Updates the manifest schema template to reflect pilot-driven requirements (integrity split, known nulls, builder status, license verification date). |
lectures/lingcod_msy_recovery.csv.yml |
Adds a full per-dataset manifest (source/license/integrity/schema/consumers/builder fields). |
lectures/lingcod_msy_recovery.csv |
Adds the lingcod MSY recovery time series data used by the consuming lecture. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Jul 16, 2026
The first dataset to land in the published tree, and the first manifest written against the schema sketch. Consumed by lecture-python-intro's msy_fishery lecture, which currently reads it from a relative path — so the downloaded notebook cannot resolve it. The repoint is the paired lecture PR. Licence established before adding, per AGENTS.md: RAM Legacy is CC BY 4.0, attribution-only, so rehosting this extract is permitted. Worth recording how that was found — ramlegacy.org states no licence at all; the authority is the Zenodo record behind its DOI (license id "cc-by-4.0", access_right "open"). Meeting a real file changed the schema in three ways, all folded back into manifest-schema.yml: - integrity splits into `migration` and `upstream`. Lingcod is a byte-perfect copy of what the lecture consumes, yet its upstream provenance cannot be re-derived at all. One "verified" field would have to overstate or understate it; two fields tell the truth. - `builder_status` records `unrecovered`. This file is constructed but ships no builder — the bug AGENTS.md names. The tempting fix is to relabel it verbatim, which buries the gap; instead it is now visible in the catalog. - `known_nulls` declares nulls that are correct. F/Fmsy is null in the terminal assessment year, which a blanket no-nulls validation rule would wrongly reject. Part of QuantEcon/meta#338 See QuantEcon/meta#336 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The P1 lingcod manifest declares `retrieved: null` and `class: constructed` with `builder: null` / `builder_status: unrecovered`. Copilot flagged both as out of compliance with AGENTS.md — correctly, because the rules had no wording for inherited files. Rather than patch the manifest to dodge the rules (which would mean inventing a retrieval date or misclassifying a constructed file as verbatim), bless both states explicitly so the gap stays visible in the generated catalog and the rules and the manifest stop contradicting. - AGENTS.md: add "Two inherited-file states" under the manifest rules — `retrieved: null` permitted for inherited-undated bytes (paired with an integrity.upstream reason; never reconstructed from git history), and `builder_status: unrecovered` for inherited constructed files (Phase 9), inherited-only so a new constructed file still must ship its builder. - manifest-schema.yml: document the `retrieved` null policy and scope `unrecovered` to inherited files at the two field sites. Provisional P1 decisions, to be folded into the convention. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108.
mmcky
force-pushed
the
datasets/p1-lingcod
branch
from
July 16, 2026 22:48
06b1dad to
8d9a167
Compare
Two small doc corrections surfaced reviewing the pilot: - AGENTS.md's manifest field list was written for the #10 schema sketch and never gained the fields P1 added: license.verified, the integrity migration/upstream split, schema.known_nulls, and builder_status. Bring it current and point to manifest-schema.yml as the authoritative field reference so the two stop drifting. - manifest-schema.yml: note that row_count_floor is the exact row count for a frozen historical extract by design (P1 lingcod = 82), so it is not later "relaxed" as if it were a mistake; a loose floor is for files that grow. See QuantEcon/meta#338 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ilable Two tone corrections to the manifest convention, from review discussion — the fields are unchanged, only the surrounding policy language: - Reframe the repo's identity as a stability CACHE for lecture data (snapshots upstream sources so a build doesn't depend on a live API or third-party host), not a content-distribution host. The licence question follows: "is this source OK to cache and serve publicly, with attribution?", not "may we republish it as our own?". Attribution (the manifest `source` block) is the primary obligation; redistribution becomes a cheap binary gate (permitted | restricted) that is a fast yes for public data sources and blocks only genuinely restricted ones (e.g. FRED third-party series). Softens "promoted public host" / "blocker". - State the capture-when-available principle explicitly: rich provenance is always welcome and worth recording when the source provides it, but a missing field is recorded as a reasoned gap, never a bar to landing a useful dataset. Effort scales with the source — a clean one gets a short manifest. Does not touch the integrity method/upstream field structure — that simplification is still under discussion. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applies the simplification agreed in review discussion. The P1 finding stands — migration and upstream are independent questions with different answers — but only one of them belongs in the manifest: - Drop the `migration` sub-block. Migration safety is a ONE-TIME check that matters at the repoint and never again, so it lives in the repoint PR as a Phase 8 gate rather than as a permanent manifest field. The manifest keeps `sha256`, which makes the check reproducible at any later date. Lingcod's verified migration (byte-compare against intro's copy, 2026-07-16) is preserved as a comment pointing at data-lectures#12 / lecture-python-intro#792. - Replace free-text `method` with an optional one-line `note`, populated only when status is not a clean verified — the valuable prose was always the REASON (why lingcod is unverifiable), not a re-typed procedure. - Move the generic how-to-verify-by-class recipe to AGENTS.md, stated once, instead of being repeated in every manifest. - Add `against`: what a verified check compared to (URL, DOI, or repo path), so "verified" is auditable rather than a bare assertion. Net effect: a clean verbatim file now carries a three-line integrity block with zero prose; only genuinely messy provenance earns a note. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AGENTS.md's own rule: when a decision settles, update PLAN.md in the same PR that acts on it. This PR revised the manifest schema in ways four PLAN passages still described in their pre-pilot form: - "Where we're going": the manifest field list predated the pilot — point to manifest-schema.yml as authoritative and name the fields P1 added (integrity, builder_status, known_nulls, license.verified). - Phase 5: link both the invariant-tests bullet and the business_cycle.py retrofit to #14, which carries the builder template and the schema decisions the tests will force (column patterns, known_nulls semantics, dtype vocabulary). - Phase 6: the licence check is per SOURCE, not per file — a cheap binary cache-and-serve-with-attribution gate, per the cache-not-host reframe. RAM Legacy (P1) joins World Bank as the second source already established. - Phase 7: the byte-compare is recorded in the repoint PR as a one-time gate (not a manifest field); verification results land in integrity.upstream as sha256/status/against/date, replacing the retired "method" field; and unverifiable-with-note is the honest known status. - Open-decisions table: licensing strawman restated in cache-not-host terms. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pilot P1 — local-path static (QuantEcon/meta#338). The first dataset to land in the published tree, and the first manifest written against the schema. Paired with QuantEcon/lecture-python-intro#792, which repoints the lecture.
Licence — cleared before adding
RAM Legacy is CC BY 4.0, attribution-only, so rehosting this extract is permitted. How that was established is itself a finding: ramlegacy.org states no licence at all. The authority is the Zenodo record behind its DOI (
license: cc-by-4.0,access_right: open), confirmed two ways. "Check the homepage" is not a sufficient licence procedure — so the manifest records when and against what a licence was established. An undated licence claim is not evidence.What meeting a real file did to the schema
The sketch in #10 was written from a clean example. One actual dataset broke it in three places, each folded back into
manifest-schema.yml:verifiedfield would overstate or understate. Resolution (after review discussion): the manifest carriesupstreamonly (status + optional one-linenote, plusagainstso a verified check is auditable); migration safety is a one-time repoint-PR gate, reproducible later from the manifest'ssha256AGENTS.mdnames. The tempting fix is to relabel itverbatim, which buries the gapbuilder_status: unrecovered, so it stays visible in the generated catalog. One of the constructed-but-unscripted set PLAN Phase 9 tracksF_over_Fmsyis null in the terminal assessment year — correct, and normalknown_nulls, so Phase 5's invariant tests can enforce "no unexpected nulls" without a blanket rule rejecting this fileIntegrity
sha256 fc2f873f…byte-identical to intro's copy, so the repoint cannot change lecture outputRecording
unverifiablewith a reason is deliberate: PLAN Phase 7 says not to promote a file whose integrity status is unknown, and "we checked, and it cannot be checked" is a known status. Silence would not be.Where each is recorded: upstream in the manifest (
status: unverifiablewith itsnote); migration here and in the paired repoint PR — it is a one-time gate, not a manifest field, and stays reproducible from the manifest'ssha256.Data
82 rows, 1928–2009,
year/B_over_Bmsy/F_over_Fmsy. Verified readable over https and renders the lecture's figure unchanged.Part of QuantEcon/meta#338
See QuantEcon/meta#336