Skip to content

Let found hyperparameters outlive the search - #25

Merged
DerOetzi merged 2 commits into
mainfrom
feat/hyperparameters-outlive-the-search
Aug 28, 2026
Merged

DerOetzi merged 2 commits into
mainfrom
feat/hyperparameters-outlive-the-search

Conversation

@DerOetzi

Copy link
Copy Markdown
Contributor

Problem

Forecaster.train() builds a fresh pipeline on every call. With tuning enabled it replaced that pipeline with a clone of the search's best estimator; with tuning disabled it kept the constructor defaults of HistGradientBoostingRegressor. best_params_ was logged and then dropped.

Consumers have moved to throttled retraining: solaredge2mqtt writes training data hourly, rebuilds the model at most daily, and wants the expensive HalvingGridSearchCV search on a slower cadence still. The only way to express that was to flip Forecaster.enable_hyperparameter_tuning from outside, which made the model alternate between tuned parameters and library defaults — the tuning held until exactly the next retraining. A consumer had to choose between fresh data and tuned parameters.

Change

  • train() takes hyperparametertuning: bool | None = None. None follows the configured default, True searches, False skips the search and fits with the parameters the last search found.
  • _hyperparametertuning() returns (Pipeline, best_params_). The parameters and the search timestamp are published at the same point as model_pipeline and metadata, so a failed run leaves the previous ones in place.
  • ModelMetadata carries hyperparameters and hyperparameters_tuned_at, both defaulted so an older sidecar still validates, and Forecaster.load() restores them. Neither takes part in raise_on_mismatch — ADR 0003 keeps model compatibility at the release version alone.
  • hyperparameters_tuned_at is the timestamp of the search, not of the training run, and untuned runs carry it forward. A consumer reads it to decide when a new search is due.
  • Applying stored parameters cannot fail a training run: set_params is wrapped and, on ValueError, the rejected keys are logged, the parameters are dropped, and the run continues with the defaults. A parameter grid that changes between releases degrades into an untuned run rather than into an exception in a retraining loop.
  • ADR 0004 records the decision, the rejected alternatives, and why PFISelector keeps its own untuned clone of the base estimator.

Verification

ruff check ., ruff format --check . and pyright are clean. The full suite is green at 249 passed.

tests/test_baseline_forecast.py and tests/test_extraction_regression.py are unmodified and still pass: with no stored parameters and no per-call override, the code path is identical to the previous one.

Compatibility

Nothing breaks at the call sites. The new train() argument is optional and both metadata fields have defaults. A model persisted before this change loads with no stored parameters and trains untuned with the defaults until the next search, which is what it was already doing. Suitable for a 0.5.0 minor release.

🤖 Generated with Claude Code

DerOetzi and others added 2 commits August 28, 2026 17:42
Training rebuilt the pipeline on every call and dropped the search result,
so an untuned retraining silently reverted the model to the library
defaults. Consumers retrain on their data cadence and want to search on a
much slower one, which the old API could only express by flipping
enable_hyperparameter_tuning from outside.

ADR 0004 records the decision, why the parameters are training state
rather than configuration, and why PFISelector keeps its own untuned
clone of the base estimator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Johannes Ott <mail@johannes-ott.net>
train() takes an optional hyperparametertuning argument that decides for
one run alone: None follows the configured default, True searches, False
skips the search and fits with the parameters the last search found.

_hyperparametertuning now returns best_params_ alongside the tuned
pipeline. They are published together with model_pipeline and metadata,
so a failed run keeps the previous ones, and they are persisted in the
sidecar so a restored model keeps them across a restart.
hyperparameters_tuned_at is the timestamp of the search and is carried
forward by untuned runs, so a consumer can decide when a new search is
due. Neither field takes part in raise_on_mismatch.

Applying stored parameters cannot fail a training run: on ValueError the
rejected keys are logged, the parameters are dropped, and the run
continues with the defaults, so a grid that changed between releases
degrades into an untuned run rather than an exception.

See ADR 0004.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Johannes Ott <mail@johannes-ott.net>
@DerOetzi
DerOetzi force-pushed the feat/hyperparameters-outlive-the-search branch from 1e754cc to 4fef7bb Compare August 28, 2026 15:42
@DerOetzi
DerOetzi merged commit 4be14d8 into main Aug 28, 2026
5 checks passed
@DerOetzi
DerOetzi deleted the feat/hyperparameters-outlive-the-search branch August 28, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant