Skip to content

Basic and advanced gap-filling via standalone tclean library - #13

Closed
ddahawkins-TUDelft wants to merge 251 commits into
modelblocks-org:mainfrom
ddahawkins-TUDelft:main
Closed

ddahawkins-TUDelft wants to merge 251 commits into
modelblocks-org:mainfrom
ddahawkins-TUDelft:main

Conversation

@ddahawkins-TUDelft

@ddahawkins-TUDelft ddahawkins-TUDelft commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Fixes #

Summary of changes in this pull request

  • Functionality is largely the same as the previous, now closed, PR
  • Substantially cleaning code is now hosted in the standalone tclean library which for this PR is hosted on anaconda pending conda-forge making the library available
  • Addresses comments from previous PR

Reviewer checklist

  • There are no pip dependencies in the module's environment files (workflow/envs/).
  • All rules use pathvars (e.g., <results>) in their inputs and outputs.
  • The integration test-suite is successful, including:
    • pre-commit.ci tests pass.
    • tests pass for all relevant OS configurations (linux, osx, windows).
  • Module documentation is up-to-date, including:
    • INTERFACE.yaml mentions all relevant pathvars and wildcards.
    • README.md describes how to use the module and has the necessary citations.

…ma validation to accept rules, and made the inptus required in clean.smk
…nd opsd data sources to resolve missing data issues. The code is restructured into loading, preparation, and cleaning stages now.
…ion with improved colour-blind friendliness, and updated pixi env accordingly to include cmap
…allow for different levels of data cleaning (off,basic,advanced). Advanced not yet implemented.
@ddahawkins-TUDelft

Copy link
Copy Markdown
Collaborator Author

@jnnr @irm-codebase I think I have addressed all notes/comments from yesterday that I agree with. The only things I didnt change were the folder organisation wrt to source helpers e.g. "sources//download.py" and "sources//prepare.py" -- i prefer this and there is no functional difference.

I have confirmed that the validate_config.py script explicitly uses tclean to validate objects that the module and tclean have in common to avoid the creation of files that will conflict downstream in the workflow (snakemake would not be able to identify these conflicts) and the user would have to manually resolve them otherwise).

I have updated the local_test.py to deliver a more substantial e2e test against the EUROPE_C34 shape file. Given that this file includes Iceland, I have configured a new Icelandic source based on their TSO's very limited public data. Fortunately, OPSD covers everything pre-2019.

The final outstanding capability are the data-quality checks i.e. where demand drops to 0, or flatlines for a few weeks. I think these will require an update to tclean and therefore should appear in the next PR.

@ddahawkins-TUDelft

Copy link
Copy Markdown
Collaborator Author

This was a fun exercise in 'how easy is it to add a source'... took me like 10mins. Should make future development very convenient... e.g. expanding this module to other continents down the line.

@irm-codebase

Copy link
Copy Markdown
Contributor

Closing this one as #15 exists and has everything needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants