Repository navigation
Allow source partitioning for time-local formula mappings - #741
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #741 +/- ##
=====================================
Coverage 79.7% 79.7%
=====================================
Files 41 41
Lines 9173 9173
Branches 1710 1710
=====================================
Hits 7315 7315
Misses 1524 1524
Partials 334 334
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
source_partition_yearscurrently applies only todirectmappings. Formula mappings are always skipped, even when their operation is local in time. For variables with long file lists, the whole input then goes into one Dask graph. In a 1026-year ACCESS-ESM1-6 piControl run,SImon.sisnthicktook 4h55m / 147.6 SU.SImon.siconcreads the same 12312 files with partitioning and took 25m / 12.8 SU.This PR lets a formula mapping opt in with
"partitionable": truein itscalculationblock. The default is unchanged: an unmarked formula mapping is still not partitioned.Changes
cmor_python_script.j2: partition when the mapping isdirect, or whencalculation.partitionable is True. The skip message now says the mapping is not marked partitionable.sisnthick,vmo,hfdsandvolcelloas partitionable. Each operates on single time steps only.zostogais deliberately left unmarked.calc_zostogatakes its reference state from the first time step and averagesdztover the processed period, so partitioning would change its result.configuration.mddescribes the opt-in and when not to use it.zostogamust not be, for both models. The template test checks the new condition.Validation
snd200 years (2400 files, 21 partitions), andhfds,vmoandvolcello20 years each (3 partitions).access_moppy_version.zostogastill logsSource partitioning skipped.sisnthick(same code, same 2400 files, 8 workers): unpartitioned 844 s, partitioned 275 s. The gain grows with run length, because the unpartitioned graph is resubmitted once per output split.