Merge dev/ufs-weather-model into dev/unified#40
Open
alperaltuntas wants to merge 196 commits into
Open
Conversation
…ange UST scale factor (NOAA-EMC#1050)
…ile instead of inp file (NOAA-EMC#1070)
…n binary and ascii format using switch ASCII. (NOAA-EMC#1089)
…efined in w3odatmd (size=15). Also, defined unit numbers for NDS(14) and NDS(15). (NOAA-EMC#1098)
… DIA in NL1 or NL2. (NOAA-EMC#1083)
…ST4 code (NOAA-EMC#1124) Co-authored-by: Fabrice Ardhuin <fabrice.ardhuin@ifremer.fr>
…A-EMC#1564) w3iosfmd, w3profsmd, w3snl1md, w3src4md, w3srcemd, w3triamd and w3uqckmd
…OAA-EMC#1566) w3strkmd, w3triamd, w3wavemd, and w3wdatmd
…riamd, and w3wavemd (NOAA-EMC#1563)
Brings the UFS weather-model line (develop as of 2026-02-12 + UFS work) into the CESM unified branch. Conflict resolution follows the experience from the merge_develop branch, plus a new reconciliation for the UFS netCDF/PIO I/O rework that develop did not contain. Restart handling (CESM initfile/use_user_restname vs UFS restart_from_binary/use_restartnc) is reconciled to COEXIST: - Re-added use_user_restname/use_user_histname to w3odatmd (UFS branch had dropped them); runtype/initfile were already retained. - w3iorsmd: kept the CESM use_user_restname restart-open logic (incl. the IOSTYP>0 restart-gather deadlock fix) and grafted UFS's optional `filename` netCDF-from-binary branch as `else if (present(filename))`. - UFS netCDF I/O subsystem (wav_pio_mod/wav_restart_mod/wav_history_mod in pio_src, plus w3initmd's read_restart path) remains gated behind #ifdef W3_PIO, as upstream intends. The CESM CIME build (CESM CMakeLists/src_list) builds the non-PIO configuration; binary restart + CESM mediator history paths are active. CESM-side files taken from merge_develop (CESM line + the 13 fixes): w3iorsmd, w3wavemd, w3src4md (ST4/Charnock/lambda science reverts + EC2/ES2/ESC imports), CMakeLists.txt (Always build ww3_strt), namelists.nml, and the NUOPC cap (wav_comp_nuopc/wav_shel_inp/wav_shr_mod/wav_import_export with the w3nmlshel/MPI_COMM signature fixes). Upstream-shared files (w3gridmd/w3iogomd/w3iogrmd/w3sic4md/w3srcemd/w3gdatmd) were 3-way merged to combine the merge_develop resolution with UFS deltas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The UFS branch had deleted CESM's netCDF gridded-output modules (w3iogoncdmd.F90, wav_grdout.F90) and reorganized src_list.cmake around its own PIO I/O. Since the merge keeps the CESM I/O line (merge_develop's w3wavemd uses w3iogoncd/user_netcdf_grdout), restore: - w3iogoncdmd.F90 and wav_grdout.F90 (deleted by UFS, needed by w3wavemd) - model/src/cmake/src_list.cmake from merge_develop (matches the CESM CIME CMakeLists, which appends nuopc_mesh_cap_src; the UFS src_list had moved the cap files into pio_src, which the CIME build does not compile) - user_netcdf_grdout flag in w3odatmd UFS PIO I/O modules (wav_pio_mod/wav_restart_mod/wav_history_mod) remain in the tree but are not compiled by the CIME build (W3_PIO off), as intended. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-applies '14e6bedf Functional form of Charnock calculation in ST4 (NOAA-EMC#1225)' that merge_develop's 44c9362 had reverted, restoring the CAPCHNK parameters in w3src4md/w3gdatmd/w3gridmd/w3iogrmd. Kept SEPARATE from the bundled '3952826f Remove ST4 lookup table' revert (lookup table stays present). The feature is off by default (CAPCHA=0 => CAPCHNK(1)=0); in that path CHATH=AALPHA and the code reduces bit-for-bit to the prior ST4 ustar calculation. Restores parity with dev/ufs-weather-model for this feature. Answer-neutrality to be confirmed by ERS cprnc vs merge_develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
While minimizing the diff against dev/ufs-weather-model (without changing answers, breaking exact-restart, or removing CESM-specific code), three latent bugs were found where the CESM-side code had diverged from the correct UFS version. Each fix both realigns with UFS and corrects a real defect; all three are answer-neutral for CESM configs (dormant or no-op on the active code paths). 1. w3iogrmd.F90: restore the IF(IERR.NE.0) CALL EXTIOF check after the main mod_def CAPCHNK read. The Functional Charnock cherry-pick (of the older 14e6bed) had reverted this region to a state lacking the I/O error check that dev/ufs-weather-model has. No-op on a successful read. 2. wav_shr_mod.F90 (timeInit): read the dummy argument ymd instead of an uninitialized local 'date'. A dev/unified refactor had added an unset local integer 'date' and changed tdate=abs(ymd) -> tdate=abs(date) and if(ymd<0) -> if(date<0), so the routine read an undefined variable. Only reachable via the optDate alarm path (interval alarms never call it), so the validated interval-alarm config is unaffected. 3. wav_comp_nuopc.F90 (InitializeRealize): guard the diagnose_mesh size argument by mesh type. In the unstructured-mesh branch gindex is never allocated (gindex_sea is used and deallocated), so size(gindex) read an unallocated array. Use nseal_cpl for the unstructured path, matching UFS. Only fires for unstructured mesh with dbug_flag>5; structured grids (e.g. wgx3v7) never reach it. Verified: full coupled build succeeds; ERS_Ld3.TL319_t232_wg37.GW_JRA passes exact-restart (COMPARE_base_rest) and reproduces merge_develop bit-for-bit across WW3, coupler, MOM6, and CICE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second hunk-by-hunk pass minimizing the diff against dev/ufs-weather-model without changing answers, breaking exact-restart, or removing CESM-specific code. All changes are provably inactive or no-ops for CESM configurations. Files now identical to dev/ufs-weather-model (whitespace-only divergence): - w3sic4md.F90, w3srcemd.F90: taken verbatim from dev/ufs-weather-model. w3wavemd.F90: - ODAT dummy dimension 35 -> 40 (matching UFS). ODAT is unused in the body in both branches and every call site passes a 40-element array (wav_shel_inp odat(40) via the cap; wmwavemd DUMMY2(40)). - Re-add UFS guards that are no-ops under CESM defaults (use_restartnc and logfile_is_assigned default .false., verboselog defaults .true., none set by CESM code): "if (.not. use_restartnc)" around the FLOUT(4)/FLOUT(8) MPI restart prep and on the J=8 second-restart-stream output; ".and. verboselog" on three NAPLOG log writes; "if (.not. logfile_is_assigned)" around the NDSO BACKSPACE. - Restore W3_T-only ILEN declaration/use (W3_T never set in CESM builds), UFS FLOUTG/FLOUTG2 = .false. declaration initializers, and several comment/blank-line/trailing-space differences. w3iogomd.F90 (W3OUTG): restore UFS USE block exactly (the extra imports IAPROC/NAPROC/NAPFLD/NSEALM were unused; NDST is used only under W3_T) and remove the unused SWW declaration. wav_shr_mod.F90: re-add the wav_loginit subroutine present in UFS (not called by CESM, harmless); remove unused locals (ierr, len, i, j x2, sec); align comment spacing. w3gridmd.F90: restore upstream changelog comment order and remove a trailing space. w3iogrmd.F90: align a continuation-line spacing. w3iorsmd.F90: restore two blank lines and the upstream TIMETAG format string (identical formatted output). wav_import_export.F90 / wav_shel_inp.F90: remove unused merge-side locals (num, fvalue, ofile). Deliberately NOT aligned (kept merge-branch side): the 4961 FORMAT missing-comma fix and the duplicate FLAG0 declaration removal (both fix real UFS bugs), the IDOUT(6,14) label, STK_TAIL documentation, validated newer-develop physics, and all CESM build/cap/output/restart architecture. Verified: full coupled build succeeds; ERS_Ld3.TL319_t232_wg37.GW_JRA passes exact-restart (COMPARE_base_rest) and remains bit-identical to merge_develop across WW3, coupler, MOM6, and CICE history output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same fix as e08e4f9 on the merge_develop branch (which never landed on dev/unified): the gnu/intel/regtest_gnu workflows run a standalone WW3 build (cmake .) that descends into model/src/CMakeLists.txt. On this CESM fork that file is CESM-only — it requires CASEROOT/Macros.cmake and ESMFMKFILE — so cmake configure fails immediately for every matrix entry when those are unset (as they are in CI). The dev/ufs-weather-model merge re-introduced these workflows as active (upstream re-enabled them in NOAA-EMC NOAA-EMC#1509 after fixing them for the upstream CMake, which this fork does not use). Rename them back to .tempdisable so PR checks stop failing. The CESM build is exercised via CIME, not these workflows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dabail10
approved these changes
Jun 11, 2026
dabail10
left a comment
dabail10
left a comment
Collaborator
There was a problem hiding this comment.
Amazing that this is all bfb.
Replace the CESM-only serial netCDF gridded-output subsystem
(w3iogoncdmd.F90 + wav_grdout.F90, -833 lines) with the UFS PIO-based
history writer (wav_history_mod), which is its direct descendant and was
designed to be CESM-compatible (the UFS cap's cesmcoupled branch already
sets use_historync with CESM filename conventions).
Build:
- Add the PIO switch to the CESM CMakeLists (W3_PIO). The parallelio
include path and -DCESMCOUPLED were already present in the wav compile
flags; shr_pio_mod comes from csm_share via LIBROOT/include.
- Compile wav_pio_mod, wav_history_mod and wav_restart_mod (the latter is
required by w3initmd's W3_PIO restart block; it stays dormant at runtime
since CESM keeps use_restartnc=.false. and the binary initfile path).
wav_pio_mod: make the CESMCOUPLED branch compile and work (it had never
been built): import inst_name from wav_shr_mod (set to "WAV" by the cap)
and assign pio_ioformat instead of the undeclared leftover nmode0. The
PIO subsystem/iotype/ioformat now come from the CESM driver via
shr_pio_getiosys/getiotype/getioformat.
wav_comp_nuopc: set use_historync=.true. for CESM (replacing
user_netcdf_grdout); call wav_pio_init before w3init in waveinit_cesm;
call wav_history_init (after w3init, since it needs mod_def) replacing
wavinit_grdout; rename the waveinit_ufs attribute to use_historync.
w3wavemd: the J=1 gridded-output slot now calls write_history(time) from
all ranks (PIO is collective) instead of the root-only W3IOGONCD; the
NRQGO/NRQGO2 gather machinery is no longer used for history (w3initmd
already skips posting it under use_historync) and the CESM-only FLGMPI(1)
wait in section 9 is removed, aligning with UFS.
wav_history_mod: fix the units attribute for THM and THP0 ("rad" ->
"degree"); the writer converts these direction fields to degrees in
meteorological convention (mod(630-rade*theta,360)) but labeled them
"rad". Candidate upstream fix - UFS files are currently mislabeled too.
Known ww3.hi file changes relative to the old writer (user-approved):
- THM is now written in degrees, meteorological convention (the WW3
standard used by ww3_ounf), instead of raw radians.
- The frequency dimension is renamed freq -> nf_ef and a freq_ef
coordinate variable is added.
All other fields are bit-identical (verified with cprnc), and the file
naming convention (casename.ww3.hi.YYYY-MM-DD-SSSSS.nc) is unchanged.
Verified on ERS_Ld3.TL319_t232_wg37.GW_JRA.derecho_intel.mom-tx3deg:
build, RUN, exact-restart (COMPARE_base_rest) and short-term archiver all
pass; coupler, MOM6 and CICE history remain bit-identical to the
merge_develop reference, confirming no answer changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mvertens
approved these changes
Jun 11, 2026
mvertens
left a comment
mvertens
left a comment
Collaborator
There was a problem hiding this comment.
@alperaltuntas - this is great. I'll work on merging this into NorESM next.
In multi-instance cases buildnml stages wav_in_NNNN (no plain wav_in), so read_shel_config failed to find the nml config, fell back to the nonexistent ww3_shel.inp, and crashed reading an undefined unit. Append inst_suffix to the wav_in filename (empty for single instance and UFS, so behavior there is unchanged) and abort with a clear error if the legacy ww3_shel.inp fallback cannot be opened. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CESM driver registers PIO settings under the ESMF gridcomp name
("WAV"), not the instance name ("WAV_0001"), so multi-instance runs
aborted with 'shr_pio_getindex:: compid out of allowed range'. Query
the name from the gridcomp instead of using inst_name. Single-instance
behavior is unchanged (both resolve to "WAV").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
|
@alperaltuntas - it looks like wav_comp_nuopc.F90 in this merge differs significantly from the emc version. In particular, a lot of the functionality in InitializeRealize has been moved to InitializeAdvertise. I think it would be good to stay aligned with the updated emc version. Is there a reason why this was not included in the merge? |
Replace the dev/unified-derived wav_comp_nuopc.F90 with UFS's current version plus a small explicit CESM patch, shrinking the diff vs dev/ufs-weather-model from 921 changed lines to ~55. This adopts UFS's phase structure (model init in InitializeAdvertise, mesh-only InitializeRealize), mpi_f08, UFS_TRACING hooks, flexible-restart (is_restart_fh) support, and drops the multigrid cap code UFS removed (dead under CESM; multigrid defaults false). CESM deltas kept on top of the UFS version: - use_user_histname/use_user_restname = .true. and use_historync only: CESM keeps binary restarts read via initfile (UFS sets use_restartnc) - wav_pio_init deferred to InitializeRealize: CMEPS sets up component PIO at PostChildrenAdvertise, between the advertise and realize phases - use_cmeps stays .false. for CESM: it selects UFS's direct-forcing core path (size-1 w3idatmd/w3adatmd forcing arrays, no time interpolation in w3updtmd) which the CESM import code does not use - time_origin from runtype-aware esmfTime (restart-run metadata) - Sw_Hs/t01/t0m1/thm zero-init in DataInitialize (CESM diag fields) - dtime_drv wired to wav_shr_mod's variable used by alarmInit; the old cap set nuopc_shr_methods' copy, leaving nsteps-type alarm options reading an uninitialized value (latent, unexercised) Side effect carried in from UFS: logfile_is_assigned is now set when the driver provides diro/logfile, so W3WAVE skips its per-step BACKSPACE on the log unit. This fixes a fatal multi-instance crash (three instances sharing wav.log corrupted the file position; gfortran 'Numerical result out of range' in formatted_backspace). wav_shel_inp.F90: read_shel_config now takes the mpi_f08 communicator directly and supports UFS's rstfldlist passthrough (additional restart fields), aligning the cap call sites with UFS verbatim. Validated: ERS_Ld3.TL319_t232_wg37.GW_JRA (intel) exact-restart PASS, cpl.hi bit-identical and ww3.hi identical except the known THM convention delta vs the merge_develop reference; MCC GW_JRA and MCC CW_JRA (gnu) multi-instance all PASS, instances bit-identical to single-instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Continue the cap alignment by replacing the dev/unified-derived wav_import_export.F90 (990 changed lines -> ~100) and wav_shel_inp.F90 (282 -> ~30) with UFS's current versions plus explicit CESM patches. wav_import_export.F90 CESM patch: - advertise set unchanged: So_h and Sw_lasl stay disabled; Sw_Hs/Sw_t01/ Sw_t0m1/Sw_thm diagnostic exports added under cesmcoupled - import fills both interpolation endpoints (X0 and XN): CESM keeps use_cmeps=.false., so the classic time-interpolated forcing path needs both arrays (UFS fills only X0) - UFS's coastal-coupling export blocks are guarded .not. cesmcoupled: their Sw_thm/Sw_t0m1 state lookups collide with the CESM diagnostic field names, and CalcTHM (degrees) silently overwrote the CESM export (radians from w3iogomd) -- caught as a wavImp_Sw_thm diff in cpl.hi - casename import made unconditional (UFS guards it non-CESM but uses it unguarded; never compiled for CESM upstream) wav_shel_inp.F90 CESM patch: - shel config read from wav_in//inst_suffix (multi-instance aware) instead of ww3_shel.nml, with a clear abort when neither config exists - notype=7 kept unconditional: CESM builds do not define W3_COU, and UFS's notype=6 would skip type-7 (coupling) output date processing wav_shr_mod.F90: removed the multigrid module variable, orphaned after the cap and import/export rebases dropped multigrid support (as UFS did). Overall model/src diff vs dev/ufs-weather-model: 996+/1297- before, ~640+/500- after; remaining divergence is the CESM build system, the CESM output/restart core (w3wavemd/w3iorsmd/w3iogomd), validated newer develop physics, and the documented cap deltas. Validated: ERS_Ld3.TL319_t232_wg37.GW_JRA (intel) exact-restart PASS with cpl.hi and cpl.hx.ww3 bit-identical to the merge_develop reference and ww3.hi identical except the known THM convention delta; MCC GW_JRA (gnu) multi-instance all PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
@mvertens I was initially hesitant to make a lot of changes in the cap in case it would break things on your end. But after your comment, I rebased it directly onto the emc version: wav_comp_nuopc.F90 is now emc's file plus a small CESM patch (~920 changed lines down to ~60). Same treatment for wav_import_export.F90 and wav_shel_inp.F90. I also switched CESM over to the netCDF (PIO) restarts. One deliberate exception to the phase alignment: for CESM, model init (w3init/wav_history_init) stays in InitializeRealize rather than InitializeAdvertise. This is required by PIO timing: the netCDF restart read happens in w3init, but CESM's PIO subsystem isn't set up by the driver until PostChildrenAdvertise (after the advertise phase), so w3init can't run there. advertise_fields stays in InitializeAdvertise. The UFS path is unchanged and follows emc exactly. The other deltas are small and commented in the code (e.g. PIO init via shr_pio for consistency with the other CESM components). All answers still b4b |
alperaltuntas
force-pushed
the
merge_dev/ufs-weather-model_20260609
branch
from
8a5c585 to
3ea16a5
Compare
Member
Author
|
On second thought, I'm backing the netCDF restart switch out of this PR. Changing the wave restart format this late in CESM3 release schedule is risky because it breaks hybrid/branch runs starting from existing binary reference restarts. (Reverted commit for reference: 8a5c585) |
Collaborator
|
@alperaltuntas - thanks for these updates. I am wondering if you could do 2 PRs. This current one and a follow up with the netCDF restart switch not backed out. I'd like to have the netCDF restart switch in our NorESM version - and this would enable our code bases to be very similar. If you create another PR with a77fa99 brought back in - then I could do a PR back to ESCOMP with the NorESM changes. Moving forwards on the CESM end - your current PR would be the release branch - and an updated PR with a77fa99 could be the development branch we could collaborate on that had netcdf restarts. Does that sound reasonable? Happy to have a call about this if that would be helpful. |
Member
Author
|
@mvertens Quick clarification on the commits: this PR already includes a77fa99 (wav_comp_nuopc rebase) and 3ea16a5 (wav_import_export/wav_shel_inp rebase). The only thing I backed out is 8a5c585, the netCDF (PIO) restart switch. So as it stands, the NUOPC cap in this PR is still quite closely aligned with dev/ufs-weather-model. As for 8a5c585, I've kept it in a separate branch called |
Collaborator
|
@alperaltuntas - thanks for the clarification. We have the necessary mods in the NorESM version to enable netcdf restarts correctly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR merges dev/ufs-weather-model (NOAA-EMC sync of develop as of 2026-02-12, dcc9da7) into dev/unified (9ea1d89), preserving all CESM functionality, and retires the CESM-only netCDF gridded-output subsystem in favor of UFS's PIO-based history writer (all with huge help from Claude Opus 3.8). After this PR, dev/unified differs from dev/ufs-weather-model in only 18 files, and every remaining difference is CESM-required, a deliberate fix, or validated newer-develop physics.
What's in the merge
Merge commit (9b7f7ba) — adopts the UFS/develop state wholesale (~419 files), including the netCDF restart/history subsystem (wav_pio_mod, wav_restart_mod, wav_history_mod), the new restart format (VERINI 2021-05-28 → 2024-04-26), and all upstream physics through 2026-02-12.
History output: CESM now uses UFS's PIO writer (8e41148 + 3e8c5a5) — the merge initially restored the CESM serial netCDF writer (w3iogoncdmd/wav_grdout) to fix the build; the final commit then deletes it permanently (−833 lines) and wires up wav_history_mod/wav_pio_mod instead: PIO switch in the CESM CMake, use_historync=.true. in the cap, wav_pio_init (PIO subsystem from the CESM driver via shr_pio), write_history called from all ranks (no more root gather). The cap's use_restartnc stays .false. — CESM restarts are unchanged.
Functional Charnock ST4 restored (58482b2) — cherry-pick of upstream 14e6bed (CAPCHNK), which an earlier revert had dropped. Answer-neutral by default (CAPCHA=0).
Three latent bug fixes (fc75073) — dropped IOSTAT check in w3iogrmd; timeInit reading an uninitialized local instead of its ymd argument; diagnose_mesh calling size() on an unallocated array in the unstructured path.
Diff minimization vs UFS (6819b93) — hunk-level realignment of gratuitous divergence (no-op guards re-added, unused locals removed, whitespace-only files taken verbatim).
CI (2094552) — re-disables the standalone gnu/intel/regtest workflows the merge re-introduced (they cmake-configure the CESM-only model/src/CMakeLists.txt and cannot pass on this fork; same fix as the develop-merge branch).
History output changes (ww3.hi)
Field values are bit-identical to the old writer with one exception, plus structural deltas:
Two upstream-able fixes made here: wav_pio_init's CESMCOUPLED branch had never compiled (undeclared inst_name, leftover nmode0); and the THM/THP0 units mislabel exists in UFS files too.
Validation: aux_ww3 test suite. b4b
This PR should be evaluated in conjunction with ESCOMP/WW3_interface#17