(8/n) Add ELO/Battles datasets as Task - #92
Conversation
Define task-owned arena and scoring defaults while keeping the Bradley-Terry implementation behind a registered scorer.
Separate battle-backed datasets from instruction adapters and allow ELO tasks to load their pinned arena sources through the dataset registry.
Route ELO through the shared benchmark dispatcher using resolved task definitions, pinned arena sources, and task-selected scoring defaults.
judge_extra_kwargs already carries max_tokens from judge.model_kwargs(), so passing it explicitly made every calibrate_temperature run crash with 'got multiple values for keyword argument max_tokens'. Matches the main judge construction.
55cbf2b to
4ec6e12
Compare
# Conflicts: # judgearena/tasks/registry.py
| @@ -0,0 +1,28 @@ | |||
| """Runtime scoring adapters for ELO rating tasks. | |||
There was a problem hiding this comment.
Note: Although it seems like this only provides a single function, scoreres are generally needed for implementing specific behaviours defined in official protocls like length-controlled results.
|
|
||
| judge_chat_model_cal = make_model( | ||
| model=cfg.judge.model, | ||
| max_tokens=cfg.judge.max_out_tokens, |
There was a problem hiding this comment.
@geoalgo this was a bug introduced by some PR in main. I actually don't know when this is introduced (possibly alex's PR). Unfortunutely it was breaking the execution.
There was a problem hiding this comment.
Thanks for catching this could you add a test for this?
There was a problem hiding this comment.
Yes of course, I will add it
| """Run-level settings (seed, output, caching, logging).""" | ||
|
|
||
| @model_validator(mode="after") | ||
| def _validate(self) -> RunConfig: |
There was a problem hiding this comment.
This is getting more complicated, which basically validates the resolved config file we pass for succesful execution. Might have to find better solution in future (not related to this PR)
| logger.info("Step 1: Loading battles from %s", cfg.elo.arena) | ||
| df_arena_all = load_arena_dataframe(arena=cfg.elo.arena) | ||
| logger.info("Step 1: Loading battles from %s", arena) | ||
| df_arena_all = load_battles(task) |
There was a problem hiding this comment.
Not necessarily for now but dont we want to have each task define a load_data run_annotations post_process report which are called by a common main?
There was a problem hiding this comment.
This is a nice question, as PR's are already large (and we have enough tool to just build this) this can be doable. We can discuss this perhaps? (Currently the pipelines are almost same, just dispatched in their own runners with more task configuration. Next steps would be to decompose pipelines into more repeatable steps)
|
|
||
| judge_chat_model_cal = make_model( | ||
| model=cfg.judge.model, | ||
| max_tokens=cfg.judge.max_out_tokens, |
There was a problem hiding this comment.
Thanks for catching this could you add a test for this?
| task: elo-lmarena-100k | ||
| description: Arena-anchored ELO evaluation using LMSYS Chatbot Arena 100k battles. | ||
|
|
||
| dataset: |
There was a problem hiding this comment.
how do we support subsetting different languages here?
There was a problem hiding this comment.
Same as we did in others, we can put a resolver that takes a look at column of the dataset lang and filters the samples. Should be a few line change (I will add it)
There was a problem hiding this comment.
I have added the support. Now we can run xxx-en to select languages as well for each of the arena
| # Fluency tasks are not packaged yet and run through the legacy path. | ||
| if self.elo is not None: | ||
| raise ValueError("elo config is only valid for ELO tasks.") | ||
| if self.model.name is None: |
There was a problem hiding this comment.
cant we add a general validate_config in tasks which always called in a common main? (not for this PR but as potential improvement)
# Conflicts: # judgearena/cli.py # judgearena/tasks/registry.py
2 participants
Summary
This PR packages the ELO/arena benchmarks as declarative tasks, unifying them with the rest of the task system. ELO runs are now selected, validated, and configured the same way as every other benchmark instead of through special-cased code paths.
Previously, ELO was dispatched by matching a task-name prefix, arena datasets were downloaded through their own hardcoded path, and arena choices and dataset revisions lived in Python constants. Adding or changing an arena meant touching several unrelated files.
After this change, each arena is a task definition:
elo-lmarena,elo-lmarena-100k,elo-lmarena-140k, andelo-comparia. The task YAML declares the arena, its pinned data sources, and its scoring defaults; the runtime resolves the task once and routes it through the shared benchmark registry.What changed
eloprotocol (arena, judging, Bradley-Terry scoring defaults), validated like the pairwise and MT-Bench protocols.dataset_revisions.pynow live with the task that uses them.judgearena tasks list; ELO runtime settings (battle sampling, calibration, bootstraps) stay in the run config, while stable arena identity stays in the task YAML.