Let found hyperparameters outlive the search - #25
Merged
Merged
Conversation
Training rebuilt the pipeline on every call and dropped the search result, so an untuned retraining silently reverted the model to the library defaults. Consumers retrain on their data cadence and want to search on a much slower one, which the old API could only express by flipping enable_hyperparameter_tuning from outside. ADR 0004 records the decision, why the parameters are training state rather than configuration, and why PFISelector keeps its own untuned clone of the base estimator. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Johannes Ott <mail@johannes-ott.net>
train() takes an optional hyperparametertuning argument that decides for one run alone: None follows the configured default, True searches, False skips the search and fits with the parameters the last search found. _hyperparametertuning now returns best_params_ alongside the tuned pipeline. They are published together with model_pipeline and metadata, so a failed run keeps the previous ones, and they are persisted in the sidecar so a restored model keeps them across a restart. hyperparameters_tuned_at is the timestamp of the search and is carried forward by untuned runs, so a consumer can decide when a new search is due. Neither field takes part in raise_on_mismatch. Applying stored parameters cannot fail a training run: on ValueError the rejected keys are logged, the parameters are dropped, and the run continues with the defaults, so a grid that changed between releases degrades into an untuned run rather than an exception. See ADR 0004. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Johannes Ott <mail@johannes-ott.net>
DerOetzi
force-pushed
the
feat/hyperparameters-outlive-the-search
branch
from
August 28, 2026 15:42
1e754cc to
4fef7bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Forecaster.train()builds a fresh pipeline on every call. With tuning enabled it replaced that pipeline with a clone of the search's best estimator; with tuning disabled it kept the constructor defaults ofHistGradientBoostingRegressor.best_params_was logged and then dropped.Consumers have moved to throttled retraining:
solaredge2mqttwrites training data hourly, rebuilds the model at most daily, and wants the expensiveHalvingGridSearchCVsearch on a slower cadence still. The only way to express that was to flipForecaster.enable_hyperparameter_tuningfrom outside, which made the model alternate between tuned parameters and library defaults — the tuning held until exactly the next retraining. A consumer had to choose between fresh data and tuned parameters.Change
train()takeshyperparametertuning: bool | None = None.Nonefollows the configured default,Truesearches,Falseskips the search and fits with the parameters the last search found._hyperparametertuning()returns(Pipeline, best_params_). The parameters and the search timestamp are published at the same point asmodel_pipelineandmetadata, so a failed run leaves the previous ones in place.ModelMetadatacarrieshyperparametersandhyperparameters_tuned_at, both defaulted so an older sidecar still validates, andForecaster.load()restores them. Neither takes part inraise_on_mismatch— ADR 0003 keeps model compatibility at the release version alone.hyperparameters_tuned_atis the timestamp of the search, not of the training run, and untuned runs carry it forward. A consumer reads it to decide when a new search is due.set_paramsis wrapped and, onValueError, the rejected keys are logged, the parameters are dropped, and the run continues with the defaults. A parameter grid that changes between releases degrades into an untuned run rather than into an exception in a retraining loop.PFISelectorkeeps its own untuned clone of the base estimator.Verification
ruff check .,ruff format --check .andpyrightare clean. The full suite is green at 249 passed.tests/test_baseline_forecast.pyandtests/test_extraction_regression.pyare unmodified and still pass: with no stored parameters and no per-call override, the code path is identical to the previous one.Compatibility
Nothing breaks at the call sites. The new
train()argument is optional and both metadata fields have defaults. A model persisted before this change loads with no stored parameters and trains untuned with the defaults until the next search, which is what it was already doing. Suitable for a 0.5.0 minor release.🤖 Generated with Claude Code