VLAN is a user-conditioned navigation benchmark and evaluation toolkit for multimodal models.
- [TBD], Release of a fine-tuned compact baseline model.
- [TBD], Training split and fine-tuning pipeline release.
- [06/2026], Evaluation toolkit and benchmark code released.
- [06/2026], Dataset released on Hugging Face.
| Five embodiments, one scene | Humanoid, quadruped robot, drone, BVI user, and wheelchair user are evaluated on identical geometry and obstacle layouts. |
| Accessibility-conditioned labels | Every trajectory carries a user-specific label from {Passable, Bypass, HighRisk, Blocked}. |
| Two task families | Low-level action prediction for embodied agents; high-level language guidance for human users. |
| Benchmark-aligned metrics | Goal-reaching, collision-aware, and accessibility-aligned scores, with a balanced cross-user aggregate (AccSR). |
| Real-to-interactive pipeline | Egocentric videos → Gaussian Splatting → Unity editing → trajectory generation → keyframe rendering. |
The codebase supports two task families:
Action Prediction Navigation
dronequadrupedRobot
Language-Guided Navigation
humanoidblindwheelchair
Notes on task naming:
humanoidis the public task name;pedestrianis retained as a legacy alias.blindis the public task name; olderbvifields are mapped internally.
The public release supports the reorganized dataset layout used in the benchmark release, while retaining compatibility with the older internal layout.
The codebase supports two task families:
- Action Prediction Navigation
dronequadrupedRobot
- Language-Guided Navigation
humanoidblindwheelchair
Python 3.10 or newer is required.
git clone https://github.com/InSAI-Lab/VLAN.git
cd VLAN
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtA CUDA-capable GPU is required to run inference with local checkpoints.
The dataset is hosted on Hugging Face
# Option 1: huggingface_hub CLI
pip install -U huggingface_hub
hf download InSAI-Lab/VLAN --repo-type dataset --local-dir ./data
# Option 2: git with LFS
git lfs install
git clone https://huggingface.co/datasets/InSAI-Lab/VLAN ./dataThe release provides the test split as ten compressed shards plus a small sample set:
data/
vlan/ # 10 shards, ~19 GB total
vlan_part01.tar.zst
...
vlan_part10.tar.zst
samples/
samples.tar.zst # small browsable subset
Extract the shards into a single directory:
cd data
mkdir -p test
for f in vlan/vlan_part*.tar.zst; do tar -I zstd -xf "$f" -C test; donepython tools/build_targets.py --data_root /path/to/test --artifacts_root artifacts
python tools/build_contexts.py --data_root /path/to/test --artifacts_root artifactspython tools/run_inference.py \
--config configs/runs/default.yaml \
--task humanoid \
--model-config configs/models/qwen2_5_vl_3b_instruct.yaml \
--data-root /path/to/test \
--artifacts-root artifactsThe legacy alias pedestrian is also accepted for --task.
python tools/evaluate.py \
--config configs/runs/default.yaml \
--task humanoid \
--model-config configs/models/qwen2_5_vl_3b_instruct.yaml \
--data-root /path/to/test \
--artifacts-root artifactsAction-prediction tasks use the same entry point with --task drone or --task quadrupedRobot.
Predicted action sequences take the form:
[{"dx": ..., "dz": ..., "yaw": ...}, ...]Model configuration files in configs/models/ assume checkpoints are available under a local
ckpts/ directory:
ckpts/Qwen2.5-VL-3B-Instruct
ckpts/Qwen2.5-VL-7B-Instruct
ckpts/Qwen3-VL-8B-Instruct
ckpts/InternVL3.5-8B
ckpts/Idefics3-8B-Llama3
ckpts/Molmo2-8B
ckpts/Llama-3.2-11B-Vision-Instruct
We would like to express our gratitude to the authors of the open-source projects on which this work is built. In particular, we thank the Vid2Sim team for releasing their code and models, which greatly facilitated our research.
This repository is under the MIT license.

