EgoArgus

Official camera-ready release draft for EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports.

📄Paper

EgoArgus evaluates two capabilities:

• Understanding: multiple-choice reasoning under multimodal, contradictory, off-topic, on-topic, and text-grounded conditions.
• Decision: whether an assistant should intervene, when it should intervene, and which assistance steps it should provide.

This repository contains the benchmark annotations and the code used for the paper's main evaluation, modality-separation analysis, attention reweighting, and NaPO-style preference alignment.

Release Contents

EgoArgus-camera-ready/
├── dataset/
│   ├── train.csv
│   ├── dev.csv
│   ├── test.csv
│   ├── paper_eval.csv
│   ├── manifest.json
│   └── videos/README.md
├── case_studies/
│   ├── README.md
│   ├── figures/
│   └── media/
├── code/
│   ├── evaluation/
│   │   ├── evaluate_vlm.py
│   │   ├── evaluate_decision.py
│   │   └── score_decision.py
│   ├── analysis/linear_probe.py
│   └── mitigation/
│       ├── attention_reweighting.py
│       ├── prepare_napo_alignment_data.py
│       ├── train_napo_alignment.py
│       └── evaluate_qwen_lora.py
├── scripts/
│   ├── build_release_dataset.py
│   ├── prepare_understanding_videos.py
│   └── validate_release.py
├── requirements.txt
└── requirements-alignment.txt

Dataset

The release contains 7,767 examples:

Part	Examples	Video status
Understanding	6,978	3,719 clips mapped to official source annotations
Decision	789	311 videos on Hugging Face

paper_eval.csv contains the complete evaluation scope used in the paper. train.csv, dev.csv, and test.csv are deterministic 80/10/10 partitions provided for future model development. Samples sharing an underlying source video are assigned to the same split. These new split labels do not imply that the paper trained on train.csv.

Fifteen reviewed understanding rows have an answer outside their four listed choices. They are explicitly marked label_status=out_of_option and retained only to preserve the evaluated paper scope; see the dataset documentation.

Each understanding row records the official source annotation, source video, and exact clip boundary used by the evaluator. Each decision example maps to a video in the Hugging Face dataset through video_id and video_filename, for example VISTA/<video_id>.mp4. See dataset/README.md for the schema and counts.

Representative Failure Modes

The decision benchmark tests more than whether a model chooses to intervene. A useful situational assistant must also identify the right issue, use temporal evidence, and intervene at a useful moment. The following frame-reviewed examples come from the released 789-row decision evaluation.

Visual evidence	Failure mode
	Cross-modal contradiction. The dialogue claims that the user's hands are dry, but the video shows wet hands approaching an electrical plug. Several VLMs recover individual visual facts yet fail to connect them to the impending electrical action.
	Intervention quality. Models differ in whether they recognize the metal-mug microwave hazard, identify the right issue, and intervene at the reviewed time. A nominal intervention can still be late or unhelpful.
	Resolved-risk false alarm. The user dries their hands before unplugging the fan, yet six model runs raise unnecessary warnings. These runs trigger on hazard-associated objects while ignoring that the risk was resolved.

Together, the examples expose missed cross-modal contradictions, incomplete issue grounding, delayed intervention, and false alarms after a risk has already been resolved. The full qualitative analysis adds a non-safety forgotten-item case, per-model outcomes, protocol caveats, and traceability to dataset/paper_eval.csv. It compares the seven paper models with an additional Gemini 3.1 Pro strong-model reference run.

Installation

Python 3.10 or later is recommended.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

NaPO alignment additionally requires:

pip install -r requirements-alignment.txt

Copy .env.example to .env and set credentials only when using an OpenAI-compatible hosted endpoint. Never commit .env.

Prepare Understanding Videos

Understanding videos are not redistributed. Obtain the official QaEgo4D/GroundVQA annotations, Ego4D full-scale videos, and EgoPlan-Bench2 annotations and source videos under their respective terms. First verify that the official annotations reproduce every released mapping:

python scripts/prepare_understanding_videos.py \
  --qaego4d-annotations /PATH/TO/annotations.QaEgo4D_all.json \
  --egoplan-annotations /PATH/TO/EgoPlan-Bench2.json \
  --validate-only

Then reconstruct all 3,719 evaluator clips:

python scripts/prepare_understanding_videos.py \
  --qaego4d-annotations /PATH/TO/annotations.QaEgo4D_all.json \
  --egoplan-annotations /PATH/TO/EgoPlan-Bench2.json \
  --ego4d-video-dir /PATH/TO/EGO4D/FULL_SCALE \
  --egoplan-video-dir /PATH/TO/EGOPLAN/SOURCE_VIDEOS \
  --output-dir /PATH/TO/EGOARGUS_UNDERSTANDING_VIDEOS

Both source roots may be flat or nested, but each original video must retain the filename <source_video_uid>.mp4. The script joins by the exact official sample_id, validates all released provenance fields, transcodes to H.264, and writes:

QaEgo4D/<sample_id>.mp4
EgoPlan-Bench2/<sample_id>.mp4

Use --source QaEgo4D or --source EgoPlan-Bench2 to prepare one source at a time. The output subdirectories can be passed directly to the local evaluation arguments below. Instead of the combined QaEgo4D JSON, the three official train, validation, and close-ended test JSON files may be supplied after the same --qaego4d-annotations argument.

Main Evaluation

The understanding evaluator accepts any OpenAI-compatible endpoint that supports video content. Use either a public video base URL or local video roots accessible to a local endpoint.

python code/evaluation/evaluate_vlm.py \
  --model MODEL_NAME \
  --input dataset/paper_eval.csv \
  --video-base-url https://HOST/PATH \
  --output results/understanding.json

Use --modalities dialogue, --modalities video, or --drop-distractor-modality for the modality ablations. When using local understanding videos, replace --video-base-url with --qaego4d-video-dir and --egoplan-video-dir.

Download the decision videos from Hugging Face, then run decision inference with the downloaded root containing VISTA/<video_id>.mp4:

hf download tommytyc/EgoArgus \
  --repo-type dataset \
  --include "VISTA/*.mp4" \
  --local-dir dataset/videos

python code/evaluation/evaluate_decision.py \
  --model MODEL_NAME \
  --decision-video-dir dataset/videos \
  --output results/decision_predictions.jsonl

python code/evaluation/score_decision.py \
  --predictions results/decision_predictions.jsonl \
  --output results/decision_scores.json

The endpoint and credential variables are documented in .env.example.

Modality Separation

linear_probe.py reproduces the Qwen3.5-2B hidden-state analysis. It samples up to 200 examples from each of the five understanding scenarios, extracts layer-wise representations, trains linear probes, and plots selected layers with truncated SVD.

python code/analysis/linear_probe.py extract \
  --qaego4d-video-dir /PATH/TO/QAEGO4D \
  --egoplan-video-dir /PATH/TO/EGOPLAN

python code/analysis/linear_probe.py probe
python code/analysis/linear_probe.py visualize

Bias Mitigation

Attention reweighting applies an epsilon bias to visual-token attention logits in Qwen3.5 full-attention layers:

python code/mitigation/attention_reweighting.py \
  --input dataset/paper_eval.csv \
  --qaego4d-video-dir /PATH/TO/QAEGO4D \
  --egoplan-video-dir /PATH/TO/EGOPLAN \
  --eps-list -10 -5 -2 0 2 5 10

Video failures are skipped by default so they cannot silently become a text-only modality condition. --allow-text-fallback is available only for explicit diagnostics.

The NaPO pipeline builds preference pairs from the external RLAIF-V-Bias dataset, trains a LoRA adapter with DPO or NaPO, and evaluates the adapter on the held-out EgoArgus understanding examples:

python code/mitigation/prepare_napo_alignment_data.py
python code/mitigation/train_napo_alignment.py --method napo
python code/mitigation/evaluate_qwen_lora.py \
  --adapter results/napo_alignment/qwen3_5_2b_lora/napo \
  --input dataset/paper_eval.csv \
  --qaego4d-video-dir /PATH/TO/QAEGO4D \
  --egoplan-video-dir /PATH/TO/EGOPLAN

Validation

Run the release checks before publishing:

python scripts/validate_release.py
python -m compileall code scripts

Data and License Notes

• Understanding videos are not included. Obtain their source datasets under the respective EgoPlan-Bench2 and QaEgo4D/Ego4D terms, then use the included preparation script. MIntRec text-grounded rows use a mapped QaEgo4D or EgoPlan-Bench2 distractor video rather than MIntRec video files.
• Decision videos are hosted in the EgoArgus Hugging Face dataset.
• RLAIF-V-Bias is an external alignment dataset and is not redistributed here.
• A code/data license and final proceedings citation must be added before the repository is made public. Until then, this draft does not grant reuse rights.

Citation

The camera-ready BibTeX entry will be added after the proceedings metadata is available.