Skip to content

Allow windowed array to hold a wider window - #3061

Draft
jemmajeffree wants to merge 10 commits into
Parcels-code:mainfrom
jemmajeffree:big_memory_window
Draft

jemmajeffree wants to merge 10 commits into
Parcels-code:mainfrom
jemmajeffree:big_memory_window

Conversation

@jemmajeffree

Copy link
Copy Markdown

Description

Changes class WindowedArray to grab as many timesteps as requested by time_chunk, to allow for dask parallelisation of file reading alongside fast numpy interpolation.
See details in #3060

Checklist

  • This will ideally close Option to speed up #3060, but it will probably need iteration and comments from those more involved with the codebase
  • Tests added — nope, can do so but will need some guidance on what to put here, still relatively new to test driven development
  • This PR targets the correct branch (main for normal development, v3-support for v3 support)

No AI was used in the development/writing of this code

@jemmajeffree

jemmajeffree commented Nov 15, 2026 •

Copy link
Copy Markdown
Author

For what it's worth, this is how I've been defining a fieldset

chunks = {'lat':-1,'lon':-1,'time':24}

ds_fset = parcels.convert.mom6_to_sgrid(
    fields={"U": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
                                                                     )["SSU"].sel(time=slice('2009-01-01','2009-07-05')), 
            "V": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
                                                                     )["SSV"].sel(time=slice('2009-01-01','2009-07-05')),
            # Engine really shouldn't need to be specified, but it was throwing errors without it
           }, 
    coords=xr.load_dataset(coords_file,engine='netcdf4'),
)

fieldset = parcels.FieldSet.from_sgrid_conventions(ds_fset).to_windowed_arrays(time_chunk=24*28)

The rest of my testing setup was pretty much following the MOM6 tutorial

@erikvansebille

Copy link
Copy Markdown
Member

Thanks so much for this contribution, @jemmajeffree! And of course also all the other context and Issues. Really nice to see someone engaging with our code so much!

I see this PR is still in Draft - do you want a review of it? Or would you prefer to keep working on it?

@jemmajeffree

Copy link
Copy Markdown
Author

Please review it! It's working for my use-case, so I'm not planning on making changes without feedback

I just left it as a draft because I think it might be failing a handful of tests and I don't know if it generalises to all use cases, or if it aligns with how you would like the codebase to fit together

@erikvansebille

Copy link
Copy Markdown
Member

OK, I'll then Approve the workflows to run and will aks @wyatt-fluidnumerics or @fluidnumericsJoe to review this code, as they developed the WIndowedArrays() backend so have more insight into how this PR would work there

@wyatt-fluidnumerics wyatt-fluidnumerics left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jemmajeffree , thanks for this PR and the associated issue, its especially cool how close to numpy performance you were able to get out of windowed arrays. A couple thoughts/questions which relate to your specific parcels use case, and then to this PR:

  • Are you using a dask.distributed.Client? I think that the speedup you are experiencing may be coming from the larger chunk reads rather than parallelized file reading. The netCDF backend for xarray has a per-process lock that prevents multiple threads within a process from reading from disk at the same time.
    xarray/backends/netCDF4_.py
        try:
            with self.datastore.lock:
                original_array = self.get_array(needs_lock=False)
                array = getitem(original_array, key)

The dask default for multi-core jobs is just to use multiple threads, because of the lock your reading speed with 1 vs 28 cores would be the same. In order to actually parallelize the file reading you'd need to set up a dask.distributed.Client and prescribe multiple single threaded processes by calling something like dask.distribured.Client(n_workers=28, threads_per_worker=1) before running the code for your parcels simulation.

  • I am also wondering if you have tried changing the max_cache_bytes in .to_chunk_cached_arrays(). With your chunking scheme (i.e. only chunking in time), ChunkCachedArrays should (to my knowledge) perform essentially the same as this modified version of WindowedArrays by pulling in large, multi step chunks of model data all at once. However, the default for max_cache_bytes is 600 MB, which would definitely not be enough to hold multiple model time levels, when max_cache_bytes for a variable is exceeded ChunkCachedArrays doesn't save the data to the cache at all, which would of course be very slow.
  • I am concerned this could introduce a bug in the interaction between self._max/max_levels and time_chunk. With these changes, the caching process loads an entire chunk worth of time levels on every read, however, when running into self._max cap, it still removes individual time levels. I worry that this could lead to weird behavior where not all of the time levels from a chunk are removed from the cache, and then the chunk is reloaded, leading to duplicate time levels in the cache.
  • On the point of generalization, I think it depends on how these changes compare to using ChunkCachedArray with comparably large chunks being loaded at once. If it turns out that this is significantly faster than a ChunkCachedArray with chunking of {'lat':-1,'lon':-1,'time':24*28} then I think it is definitely worth getting into the code base for other users, and I'd be happy to help work on a solution to the max_bytes/time_chunk problem.

@jemmajeffree

jemmajeffree commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

Thanks!

  • Yes I'm using a dask.distributed.Client. Sorry, I overlooked it as part of the setup in my description because I'm always using one. Full code is here https://gist.github.com/jemmajeffree/13620b7188705af58d831e3012718dfe. I'm pretty sure I'm getting reading in parallel, from watching the dask task stream, though the task taking most of the time is called concatenate (as usual for reading a lot of data) so I guess it's possible that the thing taking a while isn't the raw reading per say.
  • I'll experiment with max_cache_bytes in chunked arrays. My understanding from the documentation is that this still only pulls one chunk at a time? In which case you would lose the speed-up from having the dask client. Is this correct? Also, does it have a deletion mechanism? In my use case, I know I'm going to need to read the entire dataset eventually, I just don't want to do it all at once.
  • Agreed on potential for a bug with max_levels. Can you explain to me what max_levels should do? It looked to me like max_levels got overrulled if isel asked for more, and like this line would throw out everything except the two levels requested anyway, so I didn't want to touch anything in here while not understanding. In what situation might Parcels ask for non-sequential levels?

@wyatt-fluidnumerics

Copy link
Copy Markdown
Contributor

Awesome!

Good point that you would loose the speedup from the dask client with ChunkCachedArrays. It does have a deletion mechanism, whenever the cache size hits max_cached_bytes the LRU cache purges the least recently used data, so you'd just want to make sure to set max_cached_bytes to be 2 or 3 times the size of your chunks, and it will act like a rolling buffer of data. That being said I think you are right that this loses the speedup from the dask client, so this wouldn't be faster.

max_levels is supposed to represent the greatest number of time levels that WindowedArray should ever hold in memory. This matters any time particles have non-synchronous clocks, which could happen if particles are released at different times that aren't divisible by outputdt, or you use an advection kernel that uses per particle dt (see AdvectionRK45 or #3016). Originally, WindowedArray was only implemented to support synchronous particle clocks, but since #2846 we've been trying to move towards an implementation that supports non-synchronous clocks, especially as we move towards a more generalized backend API structure (see #3005).

@jemmajeffree

Copy link
Copy Markdown
Author

I tested the same setup (2000 particles distributed randomly through space and time for 100 days) with both chunk cached arrays and my suggested windowed arrays backend. Lists of times are from repeat tests

  • Windowed Arrays: 176, 177, 186 s
  • Chunk Cached Arrays (same chunks): 255, 262, 268 s
  • Chunk Cached Arrays (28 times larger chunks): 218, 224 s
  • Chunk Cached Arrays (no dask client): 170, 177 s
  • Windowed Arrays (no dask client): 236 s

Taking the two setups in bold above, trying instead releasing 2000 particles at the start of the simulation rather than randomly distributed through time, I get:

  • Windowed Arrays 143, 145, 146 s
  • Chunked Cached Arrays (no dask client) 160, 178, 193 s

Slight improvement in some situations, not universal.
Which isn't what I would have expected, but if it works it works. I guess either the overhead of the dask client is not worth the parallel boost, or they're both limited by file reading anyway.

I think it'd be possible to persuade max_levels to coordinate with time_chunks (e.g. enforcing that it needed to be an integer multiple of time_chunks, and throwing out timechunks together), but if there isn't any speed-up to be gained then probably not worth introducing the complication.

Looking back at how I started down this rabbit hole; the "squeezing performance" docs implied that windowed arrays were going to be a more intelligent way of loading only some data when particles covered the whole domain. It looks like you've got the cached chunked arrays backend working well enough that it's actually fine in that scenario, as long as the max_cache_bytes reflects the system memory limitations and there's no dask client running.
Maybe the most useful thing to do is to update the docs to say so, and close this pr?

caveats would be:

  • If there's situations in which a dask client is necessary for other reasons it might be nice to have something working well with it, but I'm having trouble coming up with a specific scenario
  • If you're forcibly cooperating with something else using a dask client, like the copernicus marine toolbox?
  • If there's another reason for keeping windowed arrays around and you'd like the performance of these to keep up with chunked cached arrays
  • If it's worth having the small performance boost in some situations

Let me know your thoughts

@fluidnumericsJoe

Copy link
Copy Markdown
Contributor

I'm curious, what kind of file system is holding the NetCDF files for the benchmarks referenced in the discussion here ? Is this lustre, beegfs, vast, something else ??

@jemmajeffree

Copy link
Copy Markdown
Author

lustre
scratch described here https://nci.org.au/infrastructure/storage-systems

I'd be curious to know if people get different comparisons on different systems

@erikvansebille

Copy link
Copy Markdown
Member

Very interesting insights and discussion - thanks for initiating it, @jemmajeffree. As was also highlighted in #3005, maintaining both the ChunckCachedArrays and WindowedArrays` classes is quite a hassle - so if there's good reasons to drop one over the other then I think we should seriously consider that. Your insights here add valuable information for that discussion!

A quick, perhaps simple question: how did you turn off the dark client when you reported the timing for Chunked Cached Arrays (no dask client)?

I leave the formal review to @wyatt-fluidnumerics and/or @fluidnumericsJoe, but can I ask you to perhaps also update the https://github.com/Parcels-code/Parcels/blob/main/docs/user_guide/simulation_performance.md file to fix some of the confusion you mentioned below (and perhaps reflect some of the findings)?

@jemmajeffree

Copy link
Copy Markdown
Author

how did you turn off the dark client when you reported the timing for Chunked Cached Arrays (no dask client)?

I commented out the lines that said:

# from dask.distributed import Client
# client = Client(threads_per_worker=1, memory_limit=0)

restarted the kernel of my notebook and ran the whole thing again

can I ask you to perhaps also update the https://github.com/Parcels-code/Parcels/blob/main/docs/user_guide/simulation_performance.md file to fix some of the confusion you mentioned below (and perhaps reflect some of the findings)?

Absolutely, just waiting for the dust to settle before I do so :)

@fluidnumericsJoe

Copy link
Copy Markdown
Contributor

Hi @jemmajeffree, thanks for all the work on this. We ran a benchmark of WindowedArray (WA) against ChunkCachedArray (CCA) on one of our datasets:

  • MITgcm North Atlantic at 10 km resolution,
  • daily 3-D output, every 5 days and each NetCDF file holds time 5 records
  • uncompressed NetCDF3 on BeeGFS
  • 2000 particles
  • 60 days
  • RK4 with dt=1h and outputdt=24h.
  • All runs produced identical particle positions, so the differences below are purely about speed and memory.

What we found

We looked at running straight numpy, chunk cached array (CCA), and windowed array (WA) with and without Dask clients.We also looked at the effects of releasing all particles at the same time level and staggering the particle releases.

Config Spread release Release at t0
numpy (fully loaded) 51 s –
CCA, 1-level chunks, no Client 63 s 76 s
CCA, 1-level chunks, 8-worker Client 100 s –
WA, time_chunk=6, no Client 99 s 71 s
WA, time_chunk=6, 8-worker Client 141 s 84 s
WA, time_chunk=1, no Client 127 s 72 s
WA, time_chunk=1, 8-worker Client 384 s –

We profiled each run with py-spy for hotspot profiles which helps us understand where we're spending most of the runtime for each setup. Three things drive the differences:

  1. Mixed particle clocks cause WA to re-read data. Kernel.execute evaluates every particle whose clock is before the segment end in a single call, and that includes particles released partway through the segment. So with continuous release times, one call can mix clocks up to a full outputdt apart. Near a data time level, the first RK4 stage asks for levels {i, i+1} and the later stages ask for {i, i+1, i+2}. WA keeps only the range of the current request, so it evicts level i+2 and reads it back on the next stage. We saw each level read about 8 times. When all particles are released at t0 there are no re-reads, and WA and CCA come out about even.
  2. CCA thrashes if its cache can't hold the whole span of clocks. With file-sized chunks and a cache that fit only one chunk, CCA swapped chunks back and forth at every file boundary: 5,450 s instead of 65 s.
  3. A dask.distributed Client made every configuration slower on our data. Each fetch is a single dask task, so extra workers have nothing to share. The Client only adds the cost of shipping each chunk from the worker back to the main process, about 1.2–1.4 s per 65 MB chunk on our node. Our data is uncompressed, though. We're now rerunning the same benchmark on a zlib-compressed NetCDF4 copy to check whether decompression changes this. That's our best guess for why your results differ.

This PR has gotten us thinking about the future of WindowedArray and CacheChunkedArray in Parcels - I'm happy to take what we learned here and update the docs with concrete recommendations. Since CCA has a bit more flexible cache eviction behavior, and we can effectively implement a WindowedArray with CCA (with better caching), we may end up deprecating WindowedArray.

Takeaways we have from this PR discussion so far

  • Don't use a Client. CCA fetches one chunk at a time, so there's nothing for the workers to share.
  • Chunk time in blocks of several hours to a day, e.g. chunks={"time": 24}, rather than one hourly level per chunk (for your case). Every fetch has a fixed overhead, and with small hourly surface fields that overhead would dominate.
  • Size the cache for the full span of particle clocks, not just two levels (example snippet below)
import math
import numpy as np
import xarray as xr

ds = xr.open_mfdataset("mom6.surface_1hr*.nc", chunks={"time": 24})  # 1 chunk = 24 hourly levels
# ... build `fieldset` from `ds` as in your gist ...

dt = np.timedelta64(1, "h")
outputdt = np.timedelta64(24, "h")
chunk_span = np.timedelta64(24, "h")  # model time covered by one chunk

u = ds["SSU"].data  # dask array
chunk_bytes = math.prod(u.chunksize) * u.dtype.itemsize
n_chunks = math.ceil((outputdt + dt) / chunk_span) + 2  # chunks one kernel call can touch
fieldset.to_chunk_cached_arrays(max_cache_bytes=2 * n_chunks * chunk_bytes)  # 2x margin

# no dask.distributed Client
pset.execute(...)

@fluidnumericsJoe

Copy link
Copy Markdown
Contributor

Got our results in for running with compressed NetCDF data.

Config Uncompressed (NetCDF3) Compressed, 1 depth level per chunk Compressed, full 3-D chunks
WA, time_chunk=1, no Client 127 s 112 s 108 s
WA, time_chunk=1, 8 workers 384 s 176 s 220 s
WA, time_chunk=6, no Client 99 s 146 s 142 s
WA, time_chunk=6, 8 workers 141 s 132 s 117 s
CCA, 1-level chunks, no Client 63 s 88 s 70 s
CCA, 1-level chunks, 8 workers 101 s 147 s 85 s
CCA, file chunks, 1200 MB cache, no Client 65 s 77 s 72 s
CCA, file chunks, 1200 MB cache, 8 workers 112 s 98 s 86 s

Compressed runs use zlib level 1 with shuffle. For reference, numpy with the fields fully loaded into memory takes 51 s.

This gives a slight refinement to our last point :

A dask.distributed Client helps only when we have compressed data and read several chunks at a time. On uncompressed NetCDF3, CCA fetches one chunk per dask task, so workers have nothing to share and the Client only adds the cost of shipping each chunk back. On a zlib-compressed NetCDF4 copy, WA with time_chunk=6 did get faster with 8 workers (117 s vs 142 s), because the 6 chunks in one read are decompressed in parallel instead of one at a time behind the HDF5 lock. Single-chunk reads still got slower with a Client, and CCA without a Client remained fastest (70 s).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Option to speed up

4 participants