ctsm5.4.053 problems encountered with running PLUMBER2 sites

[olyson]

Description of changes

This PR fixes failures in running PLUMBER2 sites. There are four categories of failures.

1. Wetland sites fail at case creation because the usermods are requesting a change to baseflow_scalar in the user_nl_clm namelist, however, this parameter is now on the parameter file. This issue is described in PLUMBER2 wetland sites fail at case creation #4023 . The sites are:

        "CZ-wet",
        "DE-SfN",
        "FI-Kaa",
        "FI-Lom",
        "RU-Che",
        "SE-Deg",
        "US-Los",
        "US-Myb",
        "US-Tw4",
        "PL-wet"

2. Several sites fail with a negative 2-m relatve humidity error in HumanIndexMod. The negative humidity is generated from either BareGroundFluxesMod or CanopyFluxesMod and passed into HumanIndexMod where there is an error check for this condition (and endrun). This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are

        "ES-ES2",
        "ES-LgS",
        "ES-VDA",
        "SE-Deg",
        "US-Cop"

3. Several sites fail with calendar errors. These are due to the fact that each site may require a different case setup configuration for spinup. This issue is described in Several PLUMBER2 sites fail at run time #4081 . The sites are

        "AT-Neu",
        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Hai",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "IT-MBo",
        "IT-SRo",
        "NL-Loo",
        "US-NR1",
        "US-FPa"

4. After fixing 3. above, five of those sites run successfully but nine fail with a different calendar error. This is related to the fact that the time dimension in the atmospheric forcing files is float, not double, consequently, the time of day calculated in CDEPS is off by several seconds both for the lower bound and the upper bound. This can cause problems for leap day transitions. This issue is described in Several PLUMBER2 sites fail at run time #4081. The sites are:

        "AU-How",
        "BE-Vie",
        "CH-Dav",
        "DE-Geb",
        "DE-Tha",
        "DK-Sor",
        "FR-Pue",
        "NL-Loo",
        "US-NR1"

Specific notes

The fixes for these issues are:

1. Here, tower_sites.py is modified so that for wetland sites, the parameter file for the base case is queried (regardless if it is wetland or not), and then the parameter file is replaced with a modified parameter file with baseflow_scalar=0 for any requested wetland sites in the run_tower submission.
   The baseflow_scalar namelist change request was deleted from the wetland sites shell commands by running tools/site_and_regional/plumber2_usermods. These modified shell commands are part of this commit.

2. Per conversation with @wwieder , this PR turns off HumanIndexMod for all the PLUMBER2 sites for now, to avoid the negative relative humidity error check. This is accomplished in cime_config/usermods_dirs/clm/PLUMBER2/defaults/user_nl_clm.

3. For the sites failing with the calendar error during spinup, these are the xml variables that need to be changed on a site by site basis for spinup:

RUN_STARTDATE
CALENDAR
DATM_YR_START
DATM_YR_END
DATM_YR_ALIGN
START_TOD (may already be set correctly though)

My run tower script (/glade/work/oleson/release-clm5.0.34/tools/PTCLM/run_clmtowers_plumber2.csh) handled this complexity, but this was not implemented in the run_tower infrastructure for PLUMBER2.
I've combined the new spinup variables into create_PLUMBER2_sites_csv.ncl -> PLUMBER2_sites.csv and modified the code to read these new variables in and generate the required shell commands.

• We'll also need a change in CDEPS, the addition of a new variable DATM_YR_END_FILENAME, following the existing variable DATM_YR_START_FILENAME that was put in for PLUMBER2. See Add DATM_YR_END_FILENAME xml variable for PLUMBER2 CDEPS#416

4. I have changed the time dimension in all of the forcing files from float to double and regenerated the time data. The nine sites now run successfully.

• The new forcing data is at /glade/work/oleson/PLUMBER2/scripts/MAKE_TIME_DOUBLE/ and will need to be copied (and rimported) to /glade/campaign/cesm/cesmdata/cseg/inputdata/atm/datm7/CLM1PT_data/PLUMBER2.

A one line change to plumber2_usermods.py was also required to get these sites to run. This line had been commented out pending a CDEPS mod, which appears to have been implemented.

• A couple of other CDEPS changes were required to get these sites to run, a change in components/cdeps/datm/cime_config/stream_definition_datm.xml to set dtlimit to 50 for 2000 compsets (spinup), and setting taxmode to extend and dtlimit to 1.e30 for HIST compsets.

Contributors other than yourself, if any: @wwieder (python help)

CTSM issues resolved or otherwise addressed, if any:
Resolves #4023
Resolves #4081
Resolves #4105

If answers are expected to change, describe (delete this line otherwise): The sites listed above will now run to completion. Sites that had run successfully prior to these changes may not have run correctly, so answers will likely change for those sites.

Any user interface changes (namelist or namelist defaults changes)? None to the defaults namelist. For wetlands, we only create a temporary modified parameter file for the requested run(s) and point to it in the run directory. Other changes here are accomplished by the PLUMBER2 shell commands.

Testing planned or performed, if any:
I've tested all of the sites with various submissions of the type:
./run_tower --plumber-sites {XX-XXX} --run-type ad --xmlchange STOP_OPTION=nyears,STOP_N=100,RESUBMIT=2,JOB_WALLCLOCK_TIME=06:00:00,PROJECT=P93300041 --overwrite.

• It would be good to test all of the sites in one submission of run_tower once all of the new forcing files are in place in the proper location, in order to fully test this branch.

I also tested a NEON site (ABBY) to make sure I didn't break that configuration.

Requirements before merge:

• I have followed the CTSM contribution guidelines.
• The code in this PR branch builds with no errors.
• The code in this PR branch runs with no errors. Briefly describe tested configuration(s): See above
• This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?: I guess this is (d) We actually now get answers

• I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe: I plan to incorporate this as part of Documentation for Plumber2 refactoring #2409
• This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?: See Revision of User's Guide section 1.6.2.3 - PLUMBER2 Tower Single Point Simulations #4135 for documentation updates.

[olyson]

I see that there are some python errors in the automated testing, I'll address those next week once I figure out what they are about. Although my code was rated 9.99/10, ha ha.

[olyson]

Ok, I've fixed all of the python-lint-and-black errors.

[wwieder]

@samsrabin can you review this. Keith's done functional tests to check this works, but would like a review on the python. We can also discuss testing (which may be outside the scope of this PR)?

[samsrabin]

Can do, sure. What sort of timeline were y'all hoping for?

[olyson]

Can do, sure. What sort of timeline were y'all hoping for?

Maybe by the end of next week?

[olyson]

Per discussion at CTSM software meeting, I'm going to expand this PR to include other fixes to problems I've encountered with running PLUMBER2 sites.

[olyson]

It looks like the compset I was using for the three tests I added is no longer available (I1PtClm60Bgc). So I'll need to change those tests. The replacement looks like it would be I1PtClm60BgcRsGs.

[olyson]

I've update the compsets in those tests and they run and Pass.

[slevis-lmwg]

@olyson while I run aux_clm on derecho and izumi, could you run the UpdateChangelog.pl script to update the ChangeLog and ChangeSum:
./UpdateChangelog.pl ctsm5.4.053 "Resolve problems with running PLUMBER2 sites"

[olyson]

Ok, I've run it, should I try filling it out, or start by committing the result of that command?

[slevis-lmwg]

Fill out the new sections pertaining to ctsm5.4.053 (one in ChangeLog and one in ChangeSum). When done, commit the updated files and push to this PR. Let me know if anything is unclear in the process.

[slevis-lmwg]

• aux_clm on derecho
• aux_clm on izumi:
  I'm rerunning some failing tests hoping to get past the failures.

[slevis-lmwg]

Resubmitting fixed the failing tests, but need to investigate two tests - also on izumi - that seem stuck in PEND mode. I will check them on Wednesday:

PEND SMS_Ly1_D_Mmpi-serial.CLM_USRDAT.I1PtClm60BgcRsGs.izumi_nag.clm-default--clm-PLUMBER2-ESLgS--clm-ADspinup RUN
PEND SMS_Ly2_D_Mmpi-serial.CLM_USRDAT.I1PtClm60BgcRsGs.izumi_nag.clm-default--clm-PLUMBER2-ARSLu--clm-ADspinup RUN

UPDATE: These two exceeded walltime. I'll see if they finish next time, or whether I need to up the requested walltime.