A harness benchmark for agentic long-video generation. Can an AI agent read a brief and turn it into a complete video series? VAHBench gives agents five short-drama briefs and scores what they deliver.
Paper (PDF) · Baseline examples · Setup and usage · Citation
An agent receives a brief that fixes the story, cast, dialogue, number of episodes and length. It plans and produces the series on its own with shared generation tools. VAHBench then scores the saved videos and the saved execution trace:
| Score | What it measures | Evidence |
|---|---|---|
| C Completeness | Seconds of genuine footage against the requested runtime. Held, blank and repeated footage, broken splices and artificial slowdown earn no credit. | Videos |
| Q Quality | Specification, consistency and plausibility of the delivery. | Videos |
| P Process | Reliability of production tasks and honesty of the agent's claims. | Trace |
S = 100 · max(C, ε) · Q^(3/4) · max(P, ε)^(1/4), with ε = 0.05.
Completeness gates the total, so a long file of filler cannot score well. A
language model only helps to parse logs; every score comes from fixed rules.
The five briefs, B1 to B5, ask for 3 × 60 s, 3 × 120 s, 5 × 120 s, 5 × 240 s and 6 × 300 s. The paper reports 240 runs from 12 systems with four text models each.
The code follows the five stages of the evaluation pipeline in the paper:
| Stage | Directory | Role |
|---|---|---|
| 1 Brief & Trace | stage1_inputs/ |
Read the brief and the trace into a structured checklist |
| 2 Delivered Videos | stage2_delivery/ |
Locate the delivered episodes |
| 3 Evidence Extraction | stage3_evidence/ |
Collect frames, cuts, audio and subtitles |
| 4 Detectors & Cross-checks | stage4_checks/ |
Run detectors and check claims against files |
| 5 C / Q / P | stage5_scoring/ |
Compute C, Q, P and S |
Around them: run.py is the entry point, config/ holds the settings,
shared/ the helpers every stage uses, reference_detectors/ the bundled
detectors, examples/ three baseline runs and tests/ the test suite.
Three runs, one for each kind of delivery, are bundled in
examples/ with their execution trace and delivered episodes.
Scores are those reported in the paper for these runs. Each preview shows ten
seconds of episode 1 and links to the full video.
The third run has the highest quality and the lowest total: still images look clean, but they are not the footage the brief asked for.
Where to find each example, relative to examples/:
| Example | Run directory | Trace | Videos |
|---|---|---|---|
| Complete | opensquilla/deepseek-V4-flash/prompt_01_the_woman_behind_the_numbers/ |
.opensquilla/trajectory-exports/trajectory/events.jsonl |
out/ |
| Padded | hermes/deepseek-V4-flash/prompt_03_terms_of_the_borrowed_vow/ |
.hermes/trajectory-exports/trajectory/events.jsonl |
seapromise/final/ |
| Static | openclaw/deepseek-V4-flash/prompt_03_terms_of_the_borrowed_vow/ |
.openclaw/trajectory-exports/trajectory/events.jsonl |
video/ |
Requirements. Python 3.10 to 3.13, ffmpeg and ffprobe on PATH, and
an API key for the extraction model.
python -m pip install -e ".[reference]"Configuration. Settings are in config/default.yaml
and score thresholds in config/thresholds.yaml.
| Setting | Default | Meaning |
|---|---|---|
modules |
all six | Detector modules: spec, artifact, visual, physical, av, seam |
lm.provider, lm.model |
openrouter, z-ai/glm-5.3 |
Extraction model; deepseek and openai also work |
lm.api_key_file |
unset | Optional key file. Never put the key itself in the YAML |
asr_model |
base |
Whisper model for transcripts, used when openai-whisper is installed |
jobs |
1 |
Runs evaluated in parallel |
The key is read from OPENROUTER_API_KEY, DEEPSEEK_API_KEY or
OPENAI_API_KEY. The model receives trace text only, with known key formats
masked, and is called only for checklist fields the rules could not read.
Run. Locate the deliveries first; this needs no key and gives no scores.
python run.py --data examples --discover-onlyThen load the key and score the three examples.
read -s OPENROUTER_API_KEY && export OPENROUTER_API_KEYpython run.py --data examples --out-dir out/demo --allow-partialThe output directory gets summary.csv with S, C, Q and P for each run, and
cases/<run>.json with the per-episode ledger, measurements and evidence. On
the examples, completeness comes out as 0.99, 0.67 and 0.05, against 1.00,
0.66 and 0.05 in the paper.
To evaluate your own agent, save each run as
<harness>/<model>/prompt_NN_<name>/ with its trace and videos, and point
--data at the parent directory. --run selects runs by a part of their path.
Detectors. The detectors that produced the paper's measurements are not in
this repository. A model-free reimplementation in reference_detectors/
stands in for them, so every run is labelled partial, needs
--allow-partial, and does not reproduce the paper's numbers.
- Measured: held, black and flat frames, cuts and join artifacts, repeated footage with motion, slowdown against source clips, audio clipping, cut continuity.
- Not measured, so these abstain: face identity, physical contact, audio dynamic range.
- Known limit: a still image that is shown again is not counted as a repeat, so completeness is overstated for deliveries built from re-shown stills.
Original detectors placed under detectors/ are picked up automatically.
Tests. No key or model download is needed.
python -m unittest discover -s tests -t .@article{zhang2026vahbench,
title = {VAHBench: A Harness Benchmark for Agentic Long-Video Generation},
author = {Zhang, Hongbo and Song, Liuyang and Zhou, Hang and Liu, Xinchen and He, Wei and Han, Kai and Wang, Yunhe},
year = {2026},
journal = {arXiv preprint arXiv:2611.20535},
url = {https://arxiv.org/pdf/2611.20535}
}MIT for the code. FFmpeg and model weights keep their own terms.
The videos in examples/ were generated by the agents under test.



