Infrastructure: Observational data pipeline and preprocessing

[duncanwp]

Description

Set up standardized observational datasets for all Tier II evaluation.

Datasets needed

• Temperature: NASA GISS, Berkeley Earth, HadCRUT, NOAA GlobalTemp, HadSST, HadISST, CRU TS
• Precipitation: GPCP, IMERG, MSWEP
• Radiation: CERES EBAF, ERBE, CERES SYN
• Sea ice: NSIDC Sea Ice Index, OSI SAF
• Reanalysis: ERA5, ECCO (for MHT validation)
• Extended: Argo, MODIS, CloudSat/CALIPSO, ISCCP, RSS

Notes

• All data should be regridded to a common grid
• Post-2015 test period extraction
• Handle observational uncertainty estimates for CRPS calculation

[duncanwp]

Proposed solution: adopt ICONEval's observational bounds and multi-obs pattern

ICONEval provides two patterns directly relevant to the observational data pipeline:

1. Precomputed observational bounds — recipe_sanity_checks.yml contains hardcoded min/max global-mean ranges derived from multiple reference datasets (ERA5, MERRA2, CERES-EBAF, ESACCI, etc.). Extract these into a shared observations/reference_bounds.yaml that both ESMValTool recipes and the Python pipeline can consume as quick-reject validation thresholds.

2. Multi-obs mean references — ICONEval's cloud recipes use ESMValTool's multi_model_statistics preprocessor with alias: MultiObsMean to aggregate 4-6 observational products per variable into a single reference field. This approach should be adopted for all variables where multiple observational products exist (tas: HadCRUT5 + NASA GISS, tos: NOAA OISST, etc.).

This enables a two-stage validation: (1) quick bounds check against the YAML file, (2) detailed scoring against multi-obs mean.

Phase: 1 (Foundation) in the integration plan
ICONEval reference: recipe_sanity_checks.yml (bounds), recipe_clouds_maps.yml (multi-obs pattern)