Skip to content

Fix --resume for CMIP7 compound variable names - #743

Merged
rhaegar325 merged 1 commit into
mainfrom
fix_cmip7_batcprocessing_resume_bug
Oct 7, 2026
Merged

rhaegar325 merged 1 commit into
mainfrom
fix_cmip7_batcprocessing_resume_bug

Conversation

@rhaegar325

@rhaegar325 rhaegar325 commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Fix #742

Problem

With cmip_version: CMIP7, --resume never resumes. find_resume_checkpoint
derives the expected variable_id with a CMIP6-style parse:

expected_variable_id = variable.split(".", 1)[-1].split("_", 1)[0]

For ocean.wmo.tavg-ol-hxy-sea.mon.glb this gives
wmo.tavg-ol-hxy-sea.mon.glb, which never equals the file's
variable_id = 'wmo'. Every existing split is rejected, the function returns
None, and the worker reprocesses from the first input year into a new
version directory, leaving the previous partial version behind.

Observed in production: an esm-piControl wmo run with 107 complete,
marker-carrying splits in v20261006/ returned no checkpoint.

Fix

A new helper, _resume_file_identity(), returns the frequency and the global
attributes that identify a variable's output files.

  • CMIP7 (realm.variable.branding.frequency.region): match on
    variable_id, branding_suffix, frequency and region.
  • CMIP6: unchanged (variable_id from the table-qualified name).

Matching on variable_id alone would not be enough for CMIP7.
atmos.tas.tavg-h2m-hxy-u, atmos.tas.tmaxavg-h2m-hxy-u and
atmos.tas.tminavg-h2m-hxy-u all write variable_id = 'tas', so a tasmax
resume could adopt tas files as its checkpoint.

The same change fixes the split length for CMIP7 daily variables.
_canonical_frequency parses CMIP6 table names and falls back to 'mon' for
CMIP7 names, so split_years: auto resolved to 10 years for daily variables
instead of 5. The frequency is now taken from the compound name.

Attribute comparison is case-insensitive, because regions appear as both glb
and GLB.

@codecov

codecov Bot commented Oct 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.8%. Comparing base (a78c3fc) to head (a364821).

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #743     +/-   ##
=======================================
+ Coverage   79.7%   79.8%   +0.1%     
=======================================
  Files         41      41             
  Lines       9173    9180      +7     
  Branches    1710    1711      +1     
=======================================
+ Hits        7315    7326     +11     
+ Misses      1524    1522      -2     
+ Partials     334     332      -2     
Flag Coverage Δ
unit 79.8% <100.0%> (+0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] --resume never finds a checkpoint for CMIP7 compound variable names

1 participant