Replicate the Majority-pressure (Asch) variant (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
What it measures
Seed 2 of 3 agents with the wrong answer and measure conformity of the third.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the medmcqa adapter). Write results under experiments/medmcqa/results/ in the same JSON/JSONL convention as experiments/medqa/results/ (not CSV, so the analysis tooling reads them).
Acceptance
- Results committed under
experiments/medmcqa/ in the MedQA template's format.
- The same headline statistic the template reports, on MedMCQA.
- Verified against a real API run (no synthetic data, no smoke-test-only claims).
- PR opened that closes this issue.
Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).
Replicate the Majority-pressure (Asch) variant (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.
experiments/medqa/majority_pressure.pybenchmaxxing/datasets/medmcqa.py(already implemented, Dataset adapter: MedMCQA (text/MCQ) #112)What it measures
Seed 2 of 3 agents with the wrong answer and measure conformity of the third.
What to do
Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the
medmcqaadapter). Write results underexperiments/medmcqa/results/in the same JSON/JSONL convention asexperiments/medqa/results/(not CSV, so the analysis tooling reads them).Acceptance
experiments/medmcqa/in the MedQA template's format.Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).