Skip to content

Allow source partitioning for time-local formula mappings - #741

Merged
rbeucher merged 1 commit into
mainfrom
optimise_partitionable_in_formula_calculation
Oct 6, 2026
Merged

rbeucher merged 1 commit into
mainfrom
optimise_partitionable_in_formula_calculation

Conversation

@rhaegar325

Copy link
Copy Markdown
Collaborator

Summary

source_partition_years currently applies only to direct mappings. Formula mappings are always skipped, even when their operation is local in time. For variables with long file lists, the whole input then goes into one Dask graph. In a 1026-year ACCESS-ESM1-6 piControl run, SImon.sisnthick took 4h55m / 147.6 SU. SImon.siconc reads the same 12312 files with partitioning and took 25m / 12.8 SU.

This PR lets a formula mapping opt in with "partitionable": true in its calculation block. The default is unchanged: an unmarked formula mapping is still not partitioned.

Changes

  • cmor_python_script.j2: partition when the mapping is direct, or when calculation.partitionable is True. The skip message now says the mapping is not marked partitionable.
  • ESM1-5 and ESM1-6 mappings: mark sisnthick, vmo, hfds and volcello as partitionable. Each operates on single time steps only.
  • zostoga is deliberately left unmarked. calc_zostoga takes its reference state from the first time step and averages dzt over the processed period, so partitioning would change its result.
  • Docs: configuration.md describes the opt-in and when not to use it.
  • Tests: the 4 mappings must be partitionable and zostoga must not be, for both models. The template test checks the new condition.

Validation

  • Unit tests: 2252 passed.
  • Ran the batch pipeline with this branch on real ACCESS-ESM1-6 piControl data: snd 200 years (2400 files, 21 partitions), and hfds, vmo and volcello 20 years each (3 partitions).
  • Compared every output with the production run made with v1.8.1 without partitioning. All data and coordinates are bit-identical. The only global attribute that differs is access_moppy_version.
  • zostoga still logs Source partitioning skipped.
  • Standalone A/B for sisnthick (same code, same 2400 files, 8 workers): unpartitioned 844 s, partitioned 275 s. The gain grows with run length, because the unpartitioned graph is resubmitted once per output split.

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.7%. Comparing base (6c1d216) to head (28f20dd).

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #741   +/-   ##
=====================================
  Coverage   79.7%   79.7%           
=====================================
  Files         41      41           
  Lines       9173    9173           
  Branches    1710    1710           
=====================================
  Hits        7315    7315           
  Misses      1524    1524           
  Partials     334     334           
Flag Coverage Δ
unit 79.7% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rbeucher
rbeucher merged commit 1e590a6 into main Oct 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants