(3/n) Introduce declarative YAML task definitions by kargibora · Pull Request #81 · OpenEuroLLM/JudgeArena

kargibora · 2026-07-21T09:44:37Z

Summary

Task-specific configuration is currently distributed across Python constants, dataset loaders, baseline mappings, prompt
registries, and runner conditions. Adding or changing a task therefore requires modifying several unrelated modules.

This PR introduces declarative YAML task definitions, inspired by the task packaging approach used by lm-evaluation-
harness.

AlpacaEval is migrated as the first packaged task. Existing benchmarks continue using their current fallback behavior
until they are migrated in later PRs.

Task definitions

A task YAML describes the stable contract of a benchmark:

task: alpaca-eval
task_version: 1

dataset:
  adapter: judgearena_tables
  sources: ...
  fields: ...

protocol:
  runner: pairwise
  generation: ...
  baseline: ...
  judge: ...
  scoring: ...

The main sections are:

task: stable task identity, version, description, and tags.
dataset: source locations, pinned revisions, loader adapter, and canonical field mappings.
protocol: how the task is generated, judged, and scored.
metadata: links to the reference implementation or paper.

The protocol does not contain implementation code. It references registered components such as pairwise,
judgearena_tables, pairwise_preference, and pairwise_win_rate.

Main components

The new task package contains three main responsibilities:

judgearena/tasks/
├── definitions/   # YAML task definitions
├── schema.py      # Valid task structure
├── loader.py      # YAML loading and inheritance
├── registry.py    # Task discovery and component validation
└── cli.py         # list, show, and validate commands

The schema defines the supported dataset sources, baseline strategies, judge settings, and scoring configuration.
Unknown fields are rejected.
The loader reads YAML safely, resolves optional _base.yaml inheritance, and calculates hashes for the source and
resolved definitions.
The registry discovers packaged tasks by ID and verifies that every referenced runner, dataset adapter, prompt,
parser, and scorer exists.
The CLI allows users to inspect and validate task definitions before running them.

Runtime behavior

When a task is executed, JudgeArena resolves it through the registry:

Task ID
-> load and validate task YAML
-> resolve dataset source and field mappings
-> apply task protocol defaults
-> select the registered runner
-> generate, judge, and score

Task definitions provide benchmark defaults, while users can still configure experiment-specific values.

The effective precedence is:

CLI flags -> run config YAML -> task-defined defaults -> framework defaults

For example:

The task selects the runner and dataset adapter.
The task provides the default baseline, prompt, swap mode, and optional judge temperature.
--model.baseline may override the task baseline when the task permits it.
Explicit judge settings from the CLI or run config remain unchanged.
Stable task identity remains in the task YAML, while model and judge choices remain run-level configuration.

Additional changes

AlpacaEval dataset loading now uses the source revision and field mappings declared in its task YAML.
Task definitions are included in the installed Python package.
Run metadata records the task version and resolved definition hash without copying the complete YAML into every
result.
Legacy fallback behavior is preserved for tasks that have not yet been packaged.

geoalgo

LGTM overall, I have left some comments.

geoalgo · 2026-08-03T14:05:51Z

+    args = list(sys.argv[1:] if argv is None else argv)
+    if args[:1] == ["tasks"]:
+        from judgearena.tasks.cli import run_task_command
+
+        run_task_command(args[1:])
+        return
+
    try:
-        cfg = build_run_config(argv)
+        cfg = build_run_config(args)


Can you add a comment here on the rationale? Why doing run_task_command only when "tasks" is passed as CLI?
It is not obvious to understand what this code does.

Hi David,
This function basically changes the execution flow for the main entry point. If judgearena tasks if provided we can execute arbitrary command like --list to see all the available tasks (or validate any) - if not provided simply execute the workflow. I think we can discard this if you think we dont need such functionality but for completeness I did implement it anyways.

Ok I would suggest something like this to improve readability:

is_task_cli = args[:1] == ["tasks"] if is_task_cli: # handle the case where we run `judgearena tasks ARGS` from judgearena.tasks.cli import run_task_command run_task_command(args[1:]) else: # generic case for other commands cfg = build_run_config(args)

which is more readable (but still a bit hacky).

I agree, this is more readable

geoalgo · 2026-08-03T14:17:10Z

+    return merged
+
+
+class TaskLoader:


See my other comment, we just need one function to load all tasks which returns a dict all of those can be in the same file.

I thing merging these two into a single registry would make the registry larger, as function wise, registry.py handles resolving the tasks by calling the loader. However as registry is the only script that calls TaskLoader, it is also OK to put them all in.

However, compared to TaskRegistry which is basically a dict, I feel like TaskLoader class is more graceful. Instead of providing the functions like load,..., discover, it just provides a single object loader which just handles all the things we want to do.

Happy to convert it to functions or merge it with registry if you think that is more cleaner and readable approach

Why do we need both? Both seems to be utils to load tasks?
It seems that this class has complex logic to detect whether tasks are valid after a recursive walk.

I think we can make it much simpler, just browse for files with exact names "task_config.yaml" in subdirs.
We do not want to do any patching, anything that does not match convention should just be rejected with a warning possibly.

So the catch is future PR's introduces some smarter logic for loading and extending tasks. For example specifying xxx-en will be load the english dataset or some datasets like official protocol configs we are willing to add extendes the `base.yaml.

sorry cant parse your sentence.

Ah sorry. I got your question wrong. We can indeed just merge these two functionailties, because TaskLoader is never called in other places. There should be some complex logic for resolving the task, but it can be handled by the functions instead of some class object. I will simplify it

Define validated YAML task specifications with discovery commands and package AlpacaEval as the first task.

Use registered definitions for runner, baseline, prompt, and judge defaults while retaining fallbacks for unmigrated tasks.

Use task-declared source revisions and field mappings for instruction and pre-generated output tables.

Store compact task versions and resolved YAML hashes in run metadata without embedding the full definition.

- cli.py: comment the reason tasks subcommands are routed before build_run_config - paths.py: remove the dead duplicate download_hf (shadowed by utils/io.py) and its now-unused imports - utils/io.py: use if/else and comment the packaged-task vs legacy dataset branches

Replace the TaskRegistry class and its dict-facade API (get/find/list/ validate_all) with a cached load_tasks() -> dict[str, ResolvedTaskSpec] and dict access at call sites. Drop the ValidationReport, TaskSummary, and UnknownTaskError wrappers; the CLI now formats the unknown-task message and counts with len(). get_packaged_task stays as the thin load_tasks().get() helper used across the codebase. Addresses review comments #3 and #6.

Route the 'tasks validate' confirmation through logger.info (status, to stderr) and configure logging on the tasks CLI path, keeping print for list/show which emit results to stdout.

…' into refactor/task-yaml-core

geoalgo

approving to not block the other PRs, doing the else would be better, I could not parse one of your sentence.

geoalgo · 2026-08-06T13:49:49Z

+    return merged
+
+
+class TaskLoader:


sorry cant parse your sentence.

geoalgo · 2026-08-06T13:50:34Z

+
+        configure_logging()
+        run_task_command(args[1:])
+        return


we still have return, can you do else as described in the comment bellow?

Fixed it, as there was few blocks I did not put else: but it makes it more readable.

geoalgo · 2026-08-06T13:51:02Z

+    if is_task_cli:
+        from judgearena.tasks.cli import run_task_command
+
+        configure_logging()


why would we configure logging only in this case? dont we want to have it before the if?

So there was a bit confusion.. Because run_task_command basically executes task/cli.py, which has logging but in cli.py, we configure logging using verbosity and log_file for the main executions. However we actually do not need logging for the task commands so I think its safer to simply remove it

Fixed this, we dont need logging for task commands which just validates or prints a task. logging is useful if we want to have some organized hierarchy in logging as well as a way to save them for debugging & reporting.

kargibora closed this Jul 22, 2026

kargibora reopened this Jul 23, 2026

geoalgo reviewed Aug 3, 2026

View reviewed changes

kargibora marked this pull request as ready for review August 4, 2026 09:33

kargibora force-pushed the refactor/task-yaml-core branch from 0ca3154 to 14bb329 Compare August 4, 2026 11:17

kargibora added 7 commits August 5, 2026 15:32

Add declarative task registry

2155e29

Define validated YAML task specifications with discovery commands and package AlpacaEval as the first task.

Resolve task runtime policy from YAML

54627ab

Use registered definitions for runner, baseline, prompt, and judge defaults while retaining fallbacks for unmigrated tasks.

Load registered datasets from task YAML

782ec88

Use task-declared source revisions and field mappings for instruction and pre-generated output tables.

Record task definition provenance

196ed9a

Store compact task versions and resolved YAML hashes in run metadata without embedding the full definition.

refactor: log task validate status instead of printing it

5297408

Route the 'tasks validate' confirmation through logger.info (status, to stderr) and configure logging on the tasks CLI path, keeping print for list/show which emit results to stdout.

geoalgo force-pushed the refactor/task-yaml-core branch from 14bb329 to 5297408 Compare August 5, 2026 13:32

kargibora added 3 commits August 6, 2026 09:52

Merge remote-tracking branch 'origin/refactor/benchmark-packages-main…

4e62ab5

…' into refactor/task-yaml-core

refactor: name the task-CLI dispatch condition (review feedback)

b95a160

fix: ruff check

3dc21c0

geoalgo approved these changes Aug 6, 2026

View reviewed changes

kargibora added 2 commits August 6, 2026 16:28

refactor: configure default logging before CLI dispatch

b744f37

refactor: task subcommands print directly instead of logging

9dcc6eb

Uh oh!

Conversation

kargibora commented Jul 21, 2026

Summary

Task definitions

Main components

Runtime behavior

Additional changes

Uh oh!

geoalgo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geoalgo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

geoalgo left a comment •

edited

Loading