Official Task Variations (1/2): Refactor Prompt Parsing into Reusable Components

[kargibora]

Problem

Prompt presets currently return a parser-mode string, while the parser code remains fixed inside evaluate.py. Adding a new judge output format therefore requires changes to both the prompt registry and the shared evaluation path. Custom prompt files also always use the score parser and cannot state which output format they produce.

The scorer receives only a preference series. This is enough for a basic win rate, but task scorers may also need fields such as category, model, baseline, or completion text.

The core idea behind this PR is to refactor parsing logic into much reusable component, so models only receive the resolved prompt and the parser required for it. This allow us to implement different strategies like

• criteria where model should output one score for each criteria and it should be parsed properly,
• a single token with also the log prob of it required by arena-hard
• defining the preference as not just score, but also A>>B, A>B,...

Before and after

Area	Before	After
Prompt parsing	evaluate.py owns PairScore and selects behavior through a parser-mode string	judgearena/prompts/parsing.py owns named parser implementations
Prompt presets	Prompt text and parser choice are handled separately	Each preset carries the parser that matches its output format
Custom prompts	System and user files are separate options with an implied score parser	judge.prompt groups both files and names the parser explicitly
Runner behavior	Pairwise, ELO, and MT-Bench know parser details	Runners receive the resolved parser and pass it to the shared judging code
Scoring input	Scorers receive only preferences	Scorers receive normalized battle rows containing the preference and related fields
Explanation prompt	Controlled by a separate boolean flag	Selected as the explicit default_with_explanation preset

New flow

Task YAML or run config
└── resolve prompt preset
    └── ResolvedJudgePrompt
        ├── system prompt
        ├── user prompt template
        └── JudgeParser
            └── normalized preference
                └── normalized battle rows
                    └── task scorer

Implementation

• Adds JudgeParser and the parser registry under judgearena/prompts/parsing.py. The existing PairScore calculation moves there without changing its formula or default temperature.
• Makes every prompt preset own its parser, so prompt text and output parsing are resolved together.
• Keeps evaluate.py focused on building judge inputs, running inference, and applying the parser supplied by the resolved prompt.
• Passes the resolved parser through pairwise, ELO, and MT-Bench without adding benchmark-specific parser branches to the shared judging path.
• Builds normalized pairwise battle rows before scoring. This keeps the current win-rate scorer simple while allowing task scorers to use other battle fields.
• Adds optional deterministic random swapping, category-specific prompt selection, and first-token logprob support for task definitions that request them.
• Moves packaged prompt files under judgearena/prompts/templates/ without changing their text.

Existing tasks keep the same default prompt, PairScore behavior, and fixed or both-order judging behavior. The new options are used only when selected by a task or run config.

Custom prompt example

judge:
  model: OpenRouter/example/judge
  prompt:
    system_file: prompts/system.txt
    user_file: prompts/user.txt
    parser: score

Tests

The PR adds focused checks for config defaults and overrides, custom prompt validation, prompt/parser resolution, and parser handoff in ELO and MT-Bench. Existing tests continue to cover the normal pairwise flow. Benchmark-specific verdict and logprob tests stay in the follow-up official-task-variants PR instead of being duplicated here.

uv run ruff check judgearena tests
uv run pytest -q tests

Result: 280 passed.

Notes

This PR deliberatly edits a lot of file (39) however they are all minimal

[kargibora]

This is a bit optional but I thought we can move them to a better place, as now judgearena/prompts includes code about the prompt and parsing, not just the template

[kargibora]

Note: This is required to allow different categories/languages to receive different prompt. Personally this will be useful if we want to, for example, use translated prompt for each different language. At the moment, it will be required by the arena-hard as creative_writing judge is different, as well as its user prompt.

[kargibora]

This is deterministic and hence, seed=0. We can make it non-deterministic later

[kargibora]

This feels very specific implementation to me but I dont know how to achieve this without changing ChatVLLM. we should either let it stay general or impelemnt a custom workflow but it can stay (we may want to output a single score and take a logprob of it to report uncertainity)

This will be used by arena-hard