Auditory Illusion Benchmark for Large Audio Language Models

Code and materials for the ICASSP 2026 paper Auditory Illusion Benchmark for Large Audio Language Models, Hayoon Kim, Eunice Hong and Kyogu Lee, Music and Audio Research Group, Seoul National University.

A listener hearing a Shepard tone reports an endlessly rising pitch that is not physically there. Does an audio-language model report it too? AIB turns ten classical auditory illusions into 14,829 forced-choice trials, each pairing the physical ground truth against what a human actually perceives, so a model can be scored on whether it shares our misperceptions rather than only on whether it is correct.

What the benchmark does

Every stimulus carries two labels. gt_label is what is physically present in the signal; illusion_label is what a listener reports. On illusion trials the two disagree, and that gap is the phenomenon. On matched control trials they agree, so the controls measure whether a model can do the task at all.

Humans are modelled as experiencing every illusion, so the human response is illusion_label. For a model:

• HLA (human-likeness): agreement with illusion_label on illusion trials
• RA (reality alignment): agreement with gt_label on illusion trials
• ISI = HLA − RA. Positive means human-like misperception, negative means the model reports the physical signal.
• Control: accuracy on control trials, where the two labels coincide.

Headline results

Ten models over all 14,829 stimuli, sorted by average ISI. Three further refined-prompt variants are in results/ and reported separately.

model                              P-ISI   K-ISI     ISI    ctrl   unparsed
----------------------------------------------------------------------------
Audio Flamingo 3                   0.271   0.505   0.455   0.574        18%
MU-LLaMA                           0.319   0.207   0.231   0.426         9%
Gemini 3.1 Pro                    -0.478   0.385   0.200   0.554         0%
Pengi                             -0.014   0.101   0.076   0.140        64% *
Kimi-Audio-7B-Instruct            -0.752   0.262   0.044   0.636         3%
Qwen-Audio-Chat                   -0.146  -0.037  -0.060   0.516        17%
Fun-Audio-Chat                    -0.668   0.039  -0.112   0.585         7%
Voxtral-Mini-3B                   -0.015  -0.152  -0.123   0.595         1%
GLM-4-Voice                       -0.164  -0.184  -0.179   0.251         4%
Qwen2-Audio-7B-Instruct           -0.541  -0.230  -0.297   0.650         1%

P = physics-based illusions, K = physics + knowledge.
* Pengi's replies are mostly audio tags such as "clicking" rather than the
  requested label, so its metrics are not interpretable. Treat any row above
  roughly 20% unparsed the same way.

No model is human-like across the board. Gemini 3.1 Pro splits by domain: strongly reality-aligned on physics illusions (−0.478) yet second most human-like on knowledge-driven ones (+0.385). Qwen2-Audio-7B-Instruct is the most literal model in the set and also has the highest control accuracy (0.650), so its negative ISI reflects discrimination rather than failure.

Layout

meta/<illusion>.csv        the dataset definition, one row per stimulus
meta/<illusion>_qwen2.csv  the same stimuli under a reworded question
meta/*_meta.py             scripts that build the meta CSVs
meta/create_qwen2_meta.py  builds the reworded-prompt CSVs
data/<illusion>.py         one generator per illusion, meta row -> audio
data/_common/              shared DSP: musicalisation, deterministic seeding
inference/                 one runner per model, one environment contract
tools/compute_metrics.py   HLA / RA / ISI scoring
tools/make_results_table.py  the paper's LaTeX tables
tools/gemini_batch.py      Gemini Batch API driver
results/<model>/           model answers, one CSV per illusion
scripts/generate_data.sh   render the audio
scripts/run_models.sh      run every READY model across GPUs
env_specs/                 pinned dependencies per model environment,
                           and where non-HuggingFace weights come from

Install

git clone https://github.com/gillosae/aib.git && cd aib
pip install -r requirements.txt

Generation needs only the packages above. Model inference does not: the models pin conflicting transformers majors and must live in separate environments, one per env_specs/venv_*.txt.

Reproducing

Score the shipped answers. This needs no audio at all:

python tools/compute_metrics.py --results results --meta meta

Render the stimuli, all ten illusions or a subset:

./scripts/generate_data.sh
./scripts/generate_data.sh zwicker_tone shepard_tone

Audio lands in data/<illusion>/<STIMULUS_ID>/<STIMULUS_ID>.wav, about 15 GB for the full set.

Run models:

export AIB_VENVS=/path/to/your/venvs
export AIB_HF_HOME=/path/to/hf-cache      # optional
./scripts/run_models.sh all --gpus 0,1,2,3
./scripts/run_models.sh zwicker_tone --models Qwen2-Audio-7B-Instruct

Gemini is not in the registry because it is an API model rather than a local checkpoint. Run it through the batch driver, which shards requests and resumes from whatever is already valid on disk:

export GOOGLE_API_KEY=...
python tools/gemini_batch.py --model gemini-pro-latest \
    --result-dir Gemini-3.1-Pro --illusions all

A runner reads ILLUSION, META_DIR, DATA_ROOT, RESULTS_ROOT and CUDA_VISIBLE_DEVICES from the environment and writes RESULTS_ROOT/<model>/<illusion>.csv with columns stimulus_id,model_answer. Adding a model means writing one script to that contract and adding a registry row to lalms_registry.tsv.

Data availability

This repository ships the code that builds the dataset, not the audio. The meta CSVs are the dataset definition and the wav files are build output.

Synthetic illusions regenerate bit-exactly. Each generator seeds its RNG from the stimulus id (data/_common/seeding.py), so a given meta row renders identically regardless of generation order, worker count, or whether you render one stimulus or all of them.

Two limits are worth stating plainly:

• speech_to_song and phonemic_restoration build on network text-to-speech (edge-tts, gTTS). Those services change over time, so their audio is reproducible in structure but not sample-for-sample.
• The audio evaluated in the paper was rendered before per-stimulus seeding was introduced. Regenerating gives a statistically equivalent draw of the same parameterisation, not identical waveforms.

Prompt variants

Each illusion has two question wordings over identical stimuli and labels. Only the question column differs, so the pair measures prompt sensitivity rather than a different task. The _qwen2 suffix is historical: the rewording was first written while probing Qwen2-Audio, and is not Qwen-specific. Models whose name ends in -refined are the same checkpoint run against the reworded prompts, and are reported separately rather than averaged in.

Citation

@inproceedings{kim2026aib,
  title     = {Auditory Illusion Benchmark for Large Audio Language Models},
  author    = {Kim, Hayoon and Hong, Eunice and Lee, Kyogu},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal
               Processing (ICASSP)},
  year      = {2026}
}

License

MIT, see LICENSE. The generators call third-party text-to-speech services whose terms govern the audio they produce; that audio is generated locally by the user and is not redistributed here.