Skip to content

feat(suite): expose provider sampling options - #23

Open
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:feat-17-sampling-options
Open

feat(suite): expose provider sampling options#23
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:feat-17-sampling-options

Conversation

@vsolano9

Copy link
Copy Markdown

Closes #17.

Suites and cases can now declare temperature and maxTokens. Case values override suite defaults; absent temperature deliberately resolves to 0 for reproducible regression runs, while an absent token cap remains omitted.

The loader rejects negative/non-finite temperatures and non-positive/non-integer token caps at both levels. README and both bundled suites demonstrate the public fields, including case overrides.

Validation:

  • RED: 7 behavioral failures plus expected public-type errors
  • 76/76 tests; 22 focused loader/runner tests repeated 20 times
  • lint, typecheck, and production build
  • both bundled suites passed unchanged: support 94.2%, summarizer 92.0%
  • full lint/typecheck/build/test/example gate on exact Node 20.20.2 and 22.23.2
  • clean installed-tarball public API/provider-request smoke
  • production dependency audit: 0 findings
  • git diff --check

All validation is offline and mock-first; no provider key or network model call is used.

Add validated suite- and case-level temperature and maxTokens fields, resolve them with case-over-suite precedence, and retain temperature zero as the deliberate deterministic default. Document the public format, update both examples, and cover parsing, validation, precedence, and provider request forwarding offline.

Closes AgentPostmortem#17
@vsolano9

Copy link
Copy Markdown
Author

Postflight clarification for the red quality-gate context:

  • the comparison itself completed successfully: base 94.2% -> head 94.2% (0.0pp)
  • it explicitly reported No regressions beyond tolerance.
  • the job then exited 2 only when the fork-scoped token received HTTP 403 creating the optional PR comment

The independent Node 20/22 CI run passed typecheck, lint, build, all 76 tests, and the example smoke on this exact head.

Runs: quality comparison · Node 20/22 CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

temperature and maxTokens are plumbed through the provider but cannot be set by a suite

1 participant