temperature and maxTokens are plumbed through the provider but cannot be set by a suite

[royalpinto007]

Problem

ProviderRequest in src/types.ts declares two sampling parameters:

/** Sampling temperature, if the provider supports it. */
temperature?: number;
/** Maximum tokens to generate. */
maxTokens?: number;

OpenAICompatibleProvider.complete faithfully forwards both to the API:

temperature: request.temperature,
max_tokens: request.maxTokens,

But nothing upstream can ever set them. toRequest in src/runner.ts is the only place a ProviderRequest is built for a run:

function toRequest(c: EvalCase, model: string): ProviderRequest {
  return {
    model,
    prompt: c.input.prompt,
    messages: c.input.messages,
    temperature: 0,
  };
}

temperature is hardcoded to 0 and maxTokens is never passed at all. EvalCase and EvalSuite have no fields for either. So the plumbing exists end to end, and both ends are wired to a wall.

Temperature 0 is a reasonable default for a regression suite and should stay the default. But a user who wants to evaluate a creative task at temperature 0.7, or who wants to cap output length to keep costs down, currently has no way to express that, and reading types.ts suggests they do.

Suggested approach

1. Add optional temperature and maxTokens to EvalSuite and EvalCase in src/types.ts, and to the suite loader and its validation in src/suite.ts.
2. Resolve them in toRequest with the same precedence the file already uses for provider and model: case, then suite, then default. Keep temperature defaulting to 0 and document why in a comment.
3. Update the example suites in examples/ to show the fields.
4. Document them in the README's suite reference.
5. Add tests: case level override wins, suite level applies when the case is silent, and the default is 0 when neither is set.

Done when

• A suite or case can set temperature and maxTokens.
• The default stays 0 and is documented as deliberate.
• Precedence matches the existing provider and model resolution.
• Covered by tests.

Good first issue: it is a small, well fenced change across four files, and it follows a resolution pattern already in the codebase. Read the providerName and model resolution in runCase first and copy its shape.

---

If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.

[vsolano9]

I would like to work on this issue. I will add validated suite/case sampling options with case-over-suite precedence, preserve deliberate temperature 0 defaults, document the public suite format, update examples, and prove the exact provider request through the offline mock-first test seam.

[github-actions]

It's yours, @vsolano9. Marked this as claimed.

Take your time, and shout in this thread if you get stuck. If there's no activity for 14 days the claim is released so someone else can pick it up.