Skip to content

fix(judge): omit seed for non-default OpenAI base URLs (Gemini compat) - #12

Open
Alozo wants to merge 1 commit into
jfrog:mainfrom
Alozo:fix/omit-seed-for-openai-compat-base-url
Open

Alozo wants to merge 1 commit into
jfrog:mainfrom
Alozo:fix/omit-seed-for-openai-compat-base-url

Conversation

@Alozo

@Alozo Alozo commented Jul 26, 2026

Copy link
Copy Markdown

Summary

OpenAIBackend.build_request unconditionally includes the OpenAI-proper seed field. OpenAI-compatible servers reject unknown fields rather than ignoring them — notably Gemini's /v1beta/openai endpoint:

HTTP 400 Invalid JSON payload received. Unknown name "seed": Cannot find field.

The agent run succeeds; only the LLM-judge scoring step fails (judge-infra error), which makes every --modes llm run against such a backend ungradeable.

Fix: send seed only when the resolved base URL is the canonical OpenAI host (_DEFAULT_BASE_URL); omit it for any custom base URL. Determinism for custom backends is still governed by temperature. This unblocks Gemini and any other OpenAI-compatible provider that doesn't implement seed, and doesn't change behavior against api.openai.com.

Repro

Point the OpenAI backend at Gemini's compat endpoint and run an llm-mode score:

export BELT_OPENAI_API_KEY="<gemini-key>"
export BELT_OPENAI_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
belt eval <scenarios> --modes rules,llm   # judge step 400s on `seed` before this fix

Test plan

  • test_build_request_default_base_includes_seed — seed still sent to api.openai.com.
  • test_build_request_custom_base_omits_seed — seed omitted for a Gemini-style custom base URL.
  • pytest tests/scorer/llm/test_backend.py -k OpenAIBackend passes.

Notes

Found while wiring Gemini (OpenAI-compat) as the judge for a docs-accessibility eval built on agent-belt. Happy to adjust the gating (e.g. an explicit opt-out env var instead of host-matching) if maintainers prefer a different shape.

`OpenAIBackend.build_request` always included the OpenAI-proper `seed`
field. OpenAI-compatible servers reject unknown fields rather than
ignoring them — notably Gemini's `/v1beta/openai` endpoint, which returns:

    HTTP 400 Invalid JSON payload received. Unknown name "seed": Cannot find field.

The agent run succeeds; only the LLM-judge scoring step fails with a
judge-infra error, so every `--modes llm` run against such a backend is
ungradeable.

Only send `seed` when talking to the canonical OpenAI host
(`_DEFAULT_BASE_URL`); omit it for any custom base URL. Determinism for
custom backends is still governed by `temperature`. This unblocks Gemini
and any other OpenAI-compatible provider that doesn't implement `seed`.

Adds tests asserting `seed` is present for the default base and absent
for a custom base URL.
@github-actions

Copy link
Copy Markdown


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

This branch had an error being deployed

1 failed deployment
frogbot — f11afaf4 Deployed Jul 26, 2026 by Alozo via scan-pull-request #30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant