Add Python port of the SAS curation pipeline

[pendingintent]

Summary

Full, faithful port of utilities/*.sas (converters, cross-workbook validators, reference-data
refresh utilities, and the JSON round-trip/delta-regeneration helpers) to Python under scripts/
and scripts/cosmoslib/. The SAS files are untouched and still work — both toolchains read the
same curation spreadsheets, and this port was verified to produce byte-for-byte-equivalent YAML
output against already-published data.

• scripts/cosmoslib/ — shared library: Excel/YAML I/O, release manifests, structured issue
  reporting, LinkML enum + NCI EVS + CDISC Library codelist/relations caches, the BC/SDTM/CRF
  converters, cross-workbook validators, subset-codelist and hierarchy helpers, and JSON
  round-trip flattening.
• scripts/convert_{bc,sdtm,crf}_xlsx2yaml.py + convert_latest_xlsx2yaml.py —
  curation-to-YAML converters, driven by utilities/manifests/*.yaml release manifests
  (replacing SAS's hand-edited, commented-out per-release blocks).
• scripts/validate_spreadsheet_{sdtm,crf}.py — cross-workbook referential checks
  (unresolved BC/DEC/SDTM links, duplicates, retired-BC pointers, character-coding issues).
• scripts/refresh_{enums,codelists,sdtm_relations}.py — reference-data cache builders,
  with added resilience so one failure doesn't abort the whole run.
• scripts/dump_{bc,sdtm,crf}_from_json.py — ad hoc live-API round-trip diagnostics.
• tests/ — golden-file regression tests (diffed against already-published YAML) plus unit
  tests for every converter/validator rule.
• how/python-conversion-pipeline.md — usage guide: setup, pipeline order, manifests,
  caveats, best practices, testing.
• plans/port-sas-utilities-to-python.md — the implementation plan this port followed.

Notes

• Two approved fixes applied: the CRF completionIinstructions typo, and the &folder.2
  output-path bug (now yaml/<folder>/crf).
• Every other SAS quirk (comparator auto-clear, group-boundary list-header printing, asymmetric
  field quoting, validator asymmetries between SDTM and CRF) is deliberately preserved for
  fidelity, not "fixed" — see grep -rn "SAS-QUIRK" scripts/ for the full audit trail.
• Two general bugs were found and fixed in excel_reader.py along the way (case-sensitive sheet
  lookups; duplicate blank-header columns breaking pd.concat) — both are real fixes, not
  quirks, and are covered by tests.
• Emitted YAML never depends on a live API lookup; only the issues/findings logs do. Golden-file
  tests run with stub NCI EVS/codelist/relations clients and no network access.
• No SAS installation is available anywhere in this project's dev or CI story; fidelity was
  established by diffing against already-published YAML and, where practical, against live API
  data — see how/ for details.

Test plan

• pytest — full suite (unit + golden-file regression tests) passes
• flake8 — clean across scripts/ and tests/
• Golden-file regression tests diff generated BC/SDTM/CRF YAML against already-published
  output under yaml/20260714_r18/ and yaml/20260630_draft/crf/ with zero mismatches
• BC and SDTM converters additionally run end-to-end against the live NCI EVS API for a
  real fixture, confirmed byte-identical to published YAML
• Cross-workbook validators run against the real, full r18/dht_test/draft corpus; findings
  traced back to genuine pre-existing curation data issues (not port bugs)
• Reviewer: skim how/python-conversion-pipeline.md for the usage flow and caveats before
  approving

🤖 Generated with Claude Code