Skip to content

Improve stdataset - #78

Draft
SarahAlidoost wants to merge 14 commits into
mainfrom
improve_stdataset
Draft

Improve stdataset#78
SarahAlidoost wants to merge 14 commits into
mainfrom
improve_stdataset

Conversation

@SarahAlidoost

@SarahAlidoost SarahAlidoost commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

closes #75

In this PR:

  • data preparation is moved from STDataset class to util data preparation function
  • util data preparation function has an option to save preprocessed data to zarr
  • STDataset class has an option to keep data lazy in init and instead load it in getitem: save memory but slower
  • Scripts to run data preparation on HPC in scripts folder using dask distributed

I'm still testing the scripts on HPC. But this PR can be reviewed.

@SarahAlidoost
SarahAlidoost marked this pull request as ready for review July 31, 2026 13:46
@SarahAlidoost
SarahAlidoost requested review from meiertgrootes and rogerkuou and removed request for meiertgrootes and rogerkuou July 31, 2026 13:50
@SarahAlidoost
SarahAlidoost marked this pull request as draft July 31, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Keep data lazy in class STDataset

1 participant

@SarahAlidoost