Skip to content

Dissenter / unanimity break (MedMCQA) #277

Description

@duckyquang

Replicate the Dissenter / unanimity break (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.

What it measures

Add one planted CORRECT peer alongside the wrong seed(s) and measure the drop in adoption.

What to do

Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the medmcqa adapter). Write results under experiments/medmcqa/results/ in the same JSON/JSONL convention as experiments/medqa/results/ (not CSV, so the analysis tooling reads them).

Acceptance

  • Results committed under experiments/medmcqa/ in the MedQA template's format.
  • The same headline statistic the template reports, on MedMCQA.
  • Verified against a real API run (no synthetic data, no smoke-test-only claims).
  • PR opened that closes this issue.

Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions