Add dataset status source of truth.

[armaanvgrewal]

Closes #152

Summary

Adds a single source of truth for dataset staged/coded/blocked readiness and surfaces it through the existing benchmaxxing datasets command.

Changes

• Added benchmaxxing/datasets/status.py with structured dataset readiness metadata.
• Included status for registered adapters and planned/not-yet-registered datasets such as medmcqa.
• Updated benchmaxxing datasets to print dataset status columns:
  • dataset
  • lane
  • data
  • adapter
  • experiments
  • blocker
• Added dataset status coverage tests so every registered adapter has a status entry.
• Added a README pointer to the dataset status source.

Design note

I kept dataset readiness status in a new benchmaxxing/datasets/status.py module instead of adding fields to DatasetSpec. This makes the status source testable and discoverable while avoiding changes to every existing adapter spec or the adapter API.

Validation

• ruff check .
• python -m pytest -q

Result:

• ruff: all checks passed
• pytest: 569 passed, 6 skipped

[Copilot]

The job was not started because recent GitHub Actions payments have failed or your spending limit needs to be increased.

[armaanvgrewal]

Updated the branch with latest origin/main and resolved the benchmaxxing/cli.py conflict by keeping the new datasets list/stats subcommand structure while making datasets list surface the readiness status table.

Also updated the medmcqa status after the concurrent origin/main change registered the MedMCQA adapter, so it now shows adapter=coded instead of adapter=not registered.

Local validation after merge:

• ruff check .
• python -m pytest -q

Result:

• ruff: all checks passed
• pytest: 645 passed, 6 skipped

[armaanvgrewal]

@sebasmos

Added you as a reviewer, but please feel free to change to someone else as you see fit. I also made a safer design choice to use a new benchmaxxing/datasets/status.py file for tracking readiness status (updated readme too), so I would like your review/feedback on it too.

[sebasmos]

Clean, one authoritative status table beats scattered notes and the CLI table formatting is tidy. One ask before merge: confirm the status entries match what the paper's per-dataset coverage grid claims, so the two do not drift. Then good to go. Note #220 and #251 also touch cli.py, so expect a small rebase depending on merge order.

[armaanvgrewal]

@sebasmos

Updated the dataset readiness status to mirror the paper's per-dataset coverage grid more directly.

Instead of a single experiments string, benchmaxxing/datasets/status.py now tracks the paper-grid columns separately:

• solo
• cascade
• plausibility
• referee

The CLI now surfaces those fields alongside lane, staged status, adapter status, and blocker:

Dataset	Lane	Staged	Adapter	Solo	Cascade	Plausibility	Referee	Blocker
chexpert	imaging	blocked	coded	pending	pending	not applicable	pending	requires signed CheXpert license and local data staging
ehr	tabular	not staged	coded loader	pending	pending	not applicable	pending	requires a derived MIMIC-IV resource CSV
medmcqa	text	not staged	coded	pending	pending	pending	pending	stage the official MedMCQA release
medqa	text	staged	coded	done	done	done	done
mimic_cxr	imaging+text	blocked	coded	pending	pending	pending	pending	requires credentialed PhysioNet MIMIC-CXR-JPG access
nih_cxr14	imaging	staged	coded	done	done	not applicable	done
pubmedqa	text	not staged	coded	pending	pending	pending	pending	stage the official PubMedQA ori_pqal.json release

Local validation:

• benchmaxxing datasets
• ruff check .
• python -m pytest -q

Result:

• ruff: all checks passed
• pytest: 646 passed, 6 skipped

[sebasmos]

This is exactly it: solo / cascade / plausibility / referee now tracked per dataset, with medqa and nih reading done and the rest pending under the right blockers, so it mirrors the paper grid rather than drifting from it. Approving to merge. Land it after #251 since both touch cli.py.

[Agastya191]

Good work, this is a genuinely useful source of truth: keeping readiness in status.py instead of bolting fields onto DatasetSpec keeps it testable without touching every adapter, the per-experiment columns (solo/cascade/plausibility/referee) mirror the paper grid, and the imaging-vs-text plausibility="not applicable" split is internally coherent. One thing on the guard though: test_dataset_status_covers_registered_adapters only asserts registered <= status_names, and the adapter field is free text (every entry reads "coded" except ehr's "coded loader"), so nothing ties a "coded" claim back to registry.names(). Since the module's own docstring says it will carry not-yet-registered datasets, the day someone adds a planned entry that says adapter="coded", datasets list will advertise it as ready while registry.get(name) raises KeyError on it: the exact drift this module exists to prevent, and it only surfaces when someone tries to run it. You have options here, but I'd cross-check it in the test, asserting that any entry claiming a coded adapter is in registry.names() and pinning adapter to a small vocabulary so "coded" vs "coded loader" can't slip through, so the readiness claim stays tied to what's actually importable.

[armaanvgrewal]

@Agastya191: Thanks for the review and the feedback.

@sebasmos: Addressed the adapter-status guard.

Changes:

• Added ADAPTER_STATUSES and CODED_ADAPTER_STATUSES in benchmaxxing/datasets/status.py.
• Added a test that pins adapter to the small vocabulary.
• Added a test guard that any status entry claiming a coded adapter must exist in registry.names().

Local validation:

• ruff check .
• python -m pytest -q

Result:

• ruff: all checks passed
• pytest: 647 passed, 6 skipped