Skip to content

feat: local model-mix what-if repricing (v0.10.0 draft) - #54

Merged
kwisatzh merged 3 commits into
mainfrom
feat/model-mix-whatif
Oct 16, 2026
Merged

kwisatzh merged 3 commits into
mainfrom
feat/model-mix-whatif

Conversation

@kwisatzh

Copy link
Copy Markdown
Contributor

Summary

Proposed for v0.10.0, independently of the next v0.9.0 row-coverage release.
Draft for review; no release or site changes are included.

  • Add a local fleetdiff what-if command for dated model-mix re-pricing of covered LiteLLM records, with simple moves, scenarios, key/team scopes, workload mappings, price overrides, text and JSON.
  • Level 1 preserves output volume and observed cache shares; level 2 shows the measured workload output factors and their mean-interval endpoints. Unknown splits, scope eligibility and prices remain explicit gaps.
  • Validate collector summaries through the existing reader, reporting their missing model/component split with null costs rather than reconstructing it from rankings.
  • Publish 468 numeric fit observations, dataset/license pins, prices, method, reproduction code and the original failed prediction-gate summaries. No prompts, responses or account identifiers are included.
  • Add a checked synthetic guide, privacy and arithmetic regressions, factor reproduction in CI, and the what-if case in the network-disabled binary check.

Measurement Contract

The study's estimator is the geometric mean of paired token ratios, with approximate 95% log-mean intervals. The cross-provider input pool is about 0.669, not a claim that token counts differ by less than 5%. The report labels these as measured factors and assumptions, not validated per-request or billing prediction ranges. The original failed range result stays recorded.

Current summaries do not carry per-model input/output/cache splits, so the summary path reports that concrete gap. Valid recorded usage from failed requests stays included; unsupported records and missing identities needed for scoped moves are excluded and counted. Default existing command output and identity bytes stay unchanged.

Verification

  • Full Go race suite and vet, including the existing v0.8.0 output goldens and legacy identity hashes.
  • Independent adapter/provenance review and core/CLI review; missing-scope and price-tier findings fixed with regressions and re-reviewed.
  • 21 offline provenance tests and exact factor reproduction.
  • Pinned LiteLLM writer/Postgres suite: 39 tests, 37 passed and two optional v0.7.0 binary-compatibility checks skipped.
  • Nine Linux binary cases produce identical output with bridge networking and --network none.
  • Anonymous-clone, fresh-build-cache outsider check: own-data stand-in and sample match the checked output; mapped JSON passes. Dependencies came from the existing pinned offline cache.
  • Bounded repricing fuzz smoke, actionlint and diff checks.

No version bump, dependency change, paid call, merge or tag. Please review this separately after the v0.9.0 release review.

Signed-off-by: Vijay <evijay@gmail.com>
Signed-off-by: Vijay <evijay@gmail.com>
@kwisatzh
kwisatzh marked this pull request as ready for review October 16, 2026 03:44
@kwisatzh
kwisatzh merged commit 646b6b4 into main Oct 16, 2026
18 checks passed
@kwisatzh
kwisatzh deleted the feat/model-mix-whatif branch October 16, 2026 03:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant