Basic and advanced gap-filling via standalone tclean library - #13
ddahawkins-TUDelft wants to merge 251 commits into
Conversation
…ning the data with config yaml validation.
…ma validation to accept rules, and made the inptus required in clean.smk
…nd opsd data sources to resolve missing data issues. The code is restructured into loading, preparation, and cleaning stages now.
…to redownload when data params change
…ion with improved colour-blind friendliness, and updated pixi env accordingly to include cmap
…allow for different levels of data cleaning (off,basic,advanced). Advanced not yet implemented.
…ntial for file names too long for Windows
|
@jnnr @irm-codebase I think I have addressed all notes/comments from yesterday that I agree with. The only things I didnt change were the folder organisation wrt to source helpers e.g. "sources/ I have confirmed that the validate_config.py script explicitly uses tclean to validate objects that the module and tclean have in common to avoid the creation of files that will conflict downstream in the workflow (snakemake would not be able to identify these conflicts) and the user would have to manually resolve them otherwise). I have updated the local_test.py to deliver a more substantial e2e test against the EUROPE_C34 shape file. Given that this file includes Iceland, I have configured a new Icelandic source based on their TSO's very limited public data. Fortunately, OPSD covers everything pre-2019. The final outstanding capability are the data-quality checks i.e. where demand drops to 0, or flatlines for a few weeks. I think these will require an update to tclean and therefore should appear in the next PR. |
|
This was a fun exercise in 'how easy is it to add a source'... took me like 10mins. Should make future development very convenient... e.g. expanding this module to other continents down the line. |
… unnecessary downloads of large files.
…ency inconsistency.
for more information, see https://pre-commit.ci
|
Closing this one as #15 exists and has everything needed. |
Fixes #
Summary of changes in this pull request
Reviewer checklist
pipdependencies in the module's environment files (workflow/envs/).pathvars(e.g.,<results>) in their inputs and outputs.pre-commit.citests pass.INTERFACE.yamlmentions all relevantpathvarsandwildcards.README.mddescribes how to use the module and has the necessary citations.