Identical to PR 13 but can use ENTSOE secret by ddahawkins-TUDelft · Pull Request #15 · modelblocks-org/module_demand_electricity

ddahawkins-TUDelft · 2026-09-07T09:52:01Z

Fixes #

Summary of changes in this pull request

As per title.

Reviewer checklist

There are no pip dependencies in the module's environment files (workflow/envs/).
All rules use pathvars (e.g., <results>) in their inputs and outputs.
The integration test-suite is successful, including:
- pre-commit.ci tests pass.
- tests pass for all relevant OS configurations (linux, osx, windows).
Module documentation is up-to-date, including:
- INTERFACE.yaml mentions all relevant pathvars and wildcards.
- README.md describes how to use the module and has the necessary citations.

…emantics

…e other minor efficiency gains.

…s and scope. Users can keep a generic config file and not worry about whether each rule is specifically relevant.

…general library of overrides without being concerned with specific case relevance.

…braries.

…e/rules.

…ucted_profile

…der internal/settings.json and limited by the number of countries requested.

for more information, see https://pre-commit.ci

irm-codebase

Only real change is some leftovers in tests/conftest.py.
I'd also consider slimming down repeated text in the README files so future you does not run into maintenance issues.

for more information, see https://pre-commit.ci

ddahawkins-TUDelft · 2026-09-16T10:22:00Z

Updated READMEs as per your suggestion @irm-codebase to avoid overlapping coverage.

Some tweaks to legacy code that you pointed out too.

irm-codebase

One final bit related to mentioning the beatiful tclean somewhere in the docs.
Otherwise, I think we are ready.

jnnr

Many thanks for all the efforts, @ddahawkins-TUDelft. The comprehensive datasources, gapfilling and plotting functionality is just amazing. I left some more detailled comments. The code is nicely separated and sufficiently clean, even if I still think that substantive parts could be simplified/compressed/outsourced to tclean, to make it easier to maintain. That shouldn't stop us from merging though.

jnnr · 2026-09-18T14:13:42Z

Great plot! Some final questions: What does Rank mean here? I understand that you first show datasources, then gaps filled, then tests.

Labels for 9, 10, 11 are not immediately clear.

For the timeseries: consider plotting a longer-term average, e.g. daily, in front of hourly timeseries, at a lighter hue.

Figure was out of date, have updated to reflect new format which omits rank and old labels. I dont have time to do the daily average but feel free to add this.

Nice! Much clearer.

Nitpick: I would write the full [Source], [Basic], [Advanced].

for more information, see https://pre-commit.ci

jnnr · 2026-09-25T14:36:50Z

@ddahawkins-TUDelft, can we merge?

ddahawkins-TUDelft added 30 commits August 10, 2026 11:59

Support sequential advanced overrides and leap-year alignment

6f3419a

Advanced workflow is fully wired into the load_cleaned.parquet now

c139514

Advanced filling now shows in the diagnostic plot

c6516f3

Refactored the basic subdirectory

4c62cab

refactored the basic subdirectory

2310d6d

Refactored the advanced folders

883af25

Reorganised folders and fixed bad imports

e936aed

Centralize final cleaning provenance

716706c

Refactored plotting logic to separate it from snakemake script calls

a8bd3ab

As per previous commit

8b07b6a

Changed plotting to viridis for colourblind friendliness and visual s…

8a57b9c

…emantics

Add energy-matched auxiliary profile scaling

d3ae4f1

Validated end to end

583c7dc

Testing a larger run

bc28a27

Optimised the ENTSOE Downloader to make requests in parallel alongsid…

f67275d

…e other minor efficiency gains.

Advanced overrides are now filtered active/inactive based on countrie…

c73e03c

…s and scope. Users can keep a generic config file and not worry about whether each rule is specifically relevant.

Implemented active vs. inactive rules such that users can maintain a …

f6489bc

…general library of overrides without being concerned with specific case relevance.

Added config test for advanced overrides

456f114

refactor: compile advanced execution manifest

223c781

Stripped python dependencies (local) from advanced.smk

bb22bb9

Implemented manifest helper to reduce local python dependencies

3cde5bd

isolate snakemake dispatch environment

9bd8eff

Updated pixi toml and validated that code does not depend on local li…

421048e

…braries.

Fixed legacy bug in advanced.smk

62dc24a

Finalised the removal of local python dependencies from snakemake cod…

52f5330

…e/rules.

Changed advanced scope enums

fdc1cfe

Rewired overrides code to allow for external_profile alongside constr…

7425dbc

…ucted_profile

Fixed bug and added external profile read tests

7413ff4

Added that aux data accepts external_profile changes

8268be0

Moved the threads assignment for entso-e into snakemake with a cap un…

2655e40

…der internal/settings.json and limited by the number of countries requested.

ddahawkins-TUDelft and others added 7 commits September 15, 2026 13:57

Updated gitignore

a0a5379

Added logging to all rule-called scripts.

b25c068

Better config.yaml examples

c4afd24

Moved 'adding a source' under development in README

8e8a889

Made entsoe-token a temp variable instead of global for conftest

eed0e7e

Removed reference to dead plot

435df43

[pre-commit.ci] auto fixes from pre-commit.com hooks

6cc961b

for more information, see https://pre-commit.ci

ddahawkins-TUDelft requested a review from irm-codebase September 15, 2026 14:57

Relaxed snakemake dependency on entsoe-token.

28fefa6

irm-codebase requested changes Sep 15, 2026

View reviewed changes

Comment thread tests/conftest.py Outdated

Comment thread README.md Outdated

ddahawkins-TUDelft and others added 5 commits September 15, 2026 20:14

Removed .clip() calls which were compromising the output visuals

4f80bfa

Removed legacy path declarations

5fad524

Adjusted plot dimensions.

f38b007

Updated READMEs

5a69f63

[pre-commit.ci] auto fixes from pre-commit.com hooks

49eee49

for more information, see https://pre-commit.ci

ddahawkins-TUDelft requested a review from irm-codebase September 16, 2026 10:21

irm-codebase approved these changes Sep 16, 2026

View reviewed changes

Comment thread config/README.md

Comment thread README.md

jnnr reviewed Sep 18, 2026

View reviewed changes

ddahawkins-TUDelft and others added 6 commits September 25, 2026 14:10

README Updates.

dade858

Removed whitespace from config schema

fc7b50d

Replaced basic_cleaning with basic_gap_filling

2c817a0

Updated language

9f64270

Updated readme figure

aacb93a

[pre-commit.ci] auto fixes from pre-commit.com hooks

09716fb

for more information, see https://pre-commit.ci

jnnr self-requested a review September 25, 2026 12:55

jnnr approved these changes Sep 25, 2026

View reviewed changes

jnnr merged commit f6e04be into main Sep 25, 2026
4 checks passed

jnnr deleted the fixed-integration-tests branch September 25, 2026 14:41

Conversation

ddahawkins-TUDelft commented Sep 7, 2026

Summary of changes in this pull request

Reviewer checklist

Uh oh!

irm-codebase left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ddahawkins-TUDelft commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

irm-codebase left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jnnr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jnnr Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

jnnr Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

jnnr Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

ddahawkins-TUDelft Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

jnnr Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

jnnr Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jnnr commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ddahawkins-TUDelft commented Sep 16, 2026 •

edited

Loading