feat: add CodSpeed benchmarks for core pipeline and models - #23
Merged
Merged
Conversation
Introduce pytest-codspeed benchmarks and a dedicated CI workflow to track performance regressions in core utilities and a representative model subset. Co-authored-by: Cursor <cursoragent@cursor.com>
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
AzulGarza
added a commit
that referenced
this pull request
Sep 11, 2026
Document PR #23 after rebasing feat/model-weight-cache onto main. Co-authored-by: Cursor <cursoragent@cursor.com>
AzulGarza
added a commit
that referenced
this pull request
Sep 11, 2026
* feat: add process-wide model weight cache for all forecasters Reuse loaded checkpoint weights across repeated forecast() calls via a global LRU cache (default max_cached_models=1). All local weight-loading models accept reuse_loaded_model and expose clear_model_cache(). Co-authored-by: Cursor <cursoragent@cursor.com> * docs: add model weight cache guide Document reuse_loaded_model, set_max_cached_models(), clear_model_cache(), and FoundationForecast clean_cache for long-lived GPU workers. Co-authored-by: Cursor <cursoragent@cursor.com> * release: v0.1.5 Co-authored-by: Cursor <cursoragent@cursor.com> * chore: fix pre-commit formatting and docs link Co-authored-by: Cursor <cursoragent@cursor.com> * docs: fix model weight cache examples for mktestdocs Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address cache review feedback and refresh gift-eval lock Add thread-safe LRU eviction before load, bounded prefix cleanup, and zero-capacity release paths. Include fallback models in clean_cache, drop Toto/TimesFM v3 cache-key pitfalls, and refresh the editable gift-eval lock metadata so CI benchmarks run against current source. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: fix mypy and pre-commit formatting Co-authored-by: Cursor <cursoragent@cursor.com> * docs: document reuse_loaded_model on all model constructors Co-authored-by: Cursor <cursoragent@cursor.com> * fix: coordinate concurrent cache loads and singleton init Use per-key in-flight loading so concurrent misses share one loader, evict outside the lock before loading, and protect cache singleton creation with double-checked locking. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: add CodSpeed benchmarks to v0.1.5 changelog Document PR #23 after rebasing feat/model-weight-cache onto main. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: sort uv.lock with toml-sort Fix pre-commit toml-sort-fix failure after rebasing onto main. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pytest-codspeedand abenchmarkpytest marker, excluded from the default test runmaybe_infer_freq,TimeSeriesDataset,QuantileConverter,SeasonalNaiveModel) and model benchmarks (Chronos-Bolt, TimesFM 1.0, Toto, Moirai 1.1).github/workflows/codspeed.ymlworkflow with simulation (core) and walltime/macro runner (models) jobsTest plan
uv run pytest tests/benchmarks/test_core.py --codspeed -m benchmark -o addopts=uv run pytest tests/benchmarks/test_models.py --codspeed -m benchmark -o addopts=TimeCopilot/foundationforecastbefore expecting CI reportsbenchmarks-coreandbenchmarks-modelsjobs pass on this PRMade with Cursor