diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 5c0a54d7..2901024d 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -34,7 +34,7 @@ } }, "onCreateCommand": "grep -qF scripts/shell_aliases.sh ~/.bashrc || echo \"source ${containerWorkspaceFolder}/scripts/shell_aliases.sh\" >> ~/.bashrc", - "postCreateCommand": "uv sync --inexact --all-extras && HADOLINT_DEST_DIR=/usr/local/bin bash scripts/install_hadolint.sh && uv run pre-commit install && (cd viewer && bun install)", + "postCreateCommand": "uv sync --inexact --all-extras && ln -sf ${containerWorkspaceFolder}/.venv/bin/videoannotator /usr/local/bin/videoannotator && ln -sf ${containerWorkspaceFolder}/.venv/bin/videoannotator /usr/local/bin/va && HADOLINT_DEST_DIR=/usr/local/bin bash scripts/install_hadolint.sh && uv run pre-commit install && (cd viewer && bun install)", "postStartCommand": "bash scripts/check_workspace_mount.sh", "mounts": [ "source=videoannotator-claude-home,target=/root/.claude,type=volume", diff --git a/.github/workflows/docs-site.yml b/.github/workflows/docs-site.yml index 0b1041e9..89eae685 100644 --- a/.github/workflows/docs-site.yml +++ b/.github/workflows/docs-site.yml @@ -5,7 +5,8 @@ name: Docs site on: push: branches: [master] - paths: [docs/**, mkdocs.yml, scripts/mkdocs_hooks.py] + # pyproject.toml: a version bump changes the version shown on every page. + paths: [docs/**, mkdocs.yml, scripts/mkdocs_hooks.py, pyproject.toml] workflow_dispatch: permissions: diff --git a/.gitignore b/.gitignore index d7cd036e..05b62702 100644 --- a/.gitignore +++ b/.gitignore @@ -234,6 +234,9 @@ htmlcov/ # Database files videoannotator.db +# The Hugging Face token saved next to the database (spec 031) +huggingface_token +huggingface_token.tmp test_*.db *.sqlite *.sqlite3 diff --git a/.specify/feature.json b/.specify/feature.json index a01f9d18..6f5b2e59 100644 --- a/.specify/feature.json +++ b/.specify/feature.json @@ -1,3 +1,3 @@ { - "feature_directory": "specs/024-container-feels-local" + "feature_directory": "specs/030-model-defaults" } diff --git a/CHANGELOG.md b/CHANGELOG.md index 8bb63d0f..f9b3d3a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,11 +7,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -Release candidate **1.6.0-rc1** (2026-10-14) is everything below, for testing the install a new -user follows from GitHub. +## [1.6.0] - 2026-10-18 + +The consolidation release: one repository, current dependencies, a clean first contact, results +out in formats researchers use (a tidy table, ELAN files, a methods paragraph), a guide for every +pipeline, and new default models. Released quietly: the viewer is being redesigned for v1.7.0, +the public release, so expect its pages to change. Results from v1.6.0 differ from v1.5's because +the default models changed (see Changed); v1.6.0-rc1 was the last version with openai-whisper. +v1.6.0-rc1 (2026-10-14) was a release candidate for testing the install from GitHub. ### Added +- **Add the Hugging Face token from the viewer, no restart** (spec 031). Speaker turns needs a + Hugging Face token to download its models. Its card now walks through it: make a Read token, + accept each model's licence (each shown as accepted or not), paste the token and press **Save + and check**. The server checks it with Hugging Face, names the account, saves it next to the + database (readable only by the server's user, so it survives container recreation), and uses + it from the next job, with no file to edit and no restart. Admin only; anyone else is told who to + ask. **Settings → Hugging Face token** shows, changes or removes it, and + `videoannotator settings huggingface-token [--status|--remove]` does the same from a terminal. + The token is never sent back to the viewer, logged or exported. A token in the server's + environment (`HUGGINGFACE_TOKEN`, `HF_AUTH_TOKEN`, `HF_TOKEN`) still takes priority. + +- **A guide for every pipeline** (spec 026, roadmap v1.6.0 Phase 3b). Each pipeline now says what + it finds in plain words ("People: where they are and how they move"), splits its settings into the few + about *what* is measured (shown first, with units) and the technical ones (collapsed, with how + many differ from the defaults), hides settings that don't apply (the OpenCV-only face + thresholds, burst offsets outside burst mode) unless they've been changed, and gives its + homepage, its published benchmarks quoted from their sources, and what to cite. The figures + say what they measure: none of the face figures is for infants, and Whisper's are for English + read aloud by adults. Written once in each pipeline's metadata, so the viewer (Pipelines page, + the job wizard, Copy BibTeX), the API (new fields on `/pipelines`, `/pipelines/{name}` and + `/schema`, nothing renamed) and the CLI agree; `docs/pipelines_spec.md` has a section per + pipeline. +- **`videoannotator pipelines describe [--json]`** prints a pipeline's guide, settings and + sources without a server. `pipelines list` shows each pipeline's title. + +- **Results out: one table, ELAN files and a methods paragraph** (spec 025, roadmap Phase 3). An + outside lab can now take results into R or Python without writing a parser per pipeline. + - **The results table**: every pipeline's results in one CSV (or Parquet), one row per event + and measure, the same 19 columns for every pipeline (`run`, `video`, `pipeline`, `event`, + `track`, `start`, `end`, `measure`, `label`, `value`, `confidence`, a box, `model`, + `videoannotator_version`, `file`, …). Times are seconds from the start of the video for + every pipeline. Keypoints, landmarks and embeddings stay in the native files. Format version + 1; later versions only add columns or measures. + - **ELAN files** (`.eaf`, EAF 3.0, validated against ELAN's schema): one per video, one tier per + pipeline track (scenes, speech, each speaker, each tracked person, faces, VLM labels), linked + to the video. Frame detections become "in view" spans; per-detection measures stay in the + table. The viewer still opens a file after a coder adds tiers in ELAN. + - **A methods paragraph** with BibTeX: each pipeline, its models with exact weight revisions, + settings that differ from the defaults, deterministic mode, VideoAnnotator's version, and + citations. A run's videos processed the same way are described once; differences are counted. + Anything not recorded says so. No video names, paths or ids. Pipeline metadata gains an + optional `citations` key (filled in for all eight shipped pipelines). + - **Everywhere**: the viewer's **Export** menu on job and run pages (and a table button on + saved datasets); `videoannotator export table|elan|methods `, which works on any + results folder with no server; and `GET /api/v1/{jobs,batches,datasets}/{id}/export/table`, + `.../export/elan`, `GET /api/v1/{jobs,batches}/{id}/methods`. All three read the same files + the same way, so they give the same result. Results from before provenance (v1.5.0) export + too, marked `not recorded`. + - **Parquet** is an optional extra, `pip install videoannotator[parquet]` (in `all`), so the core + install stays slim; an admin can install it from the viewer when it's first asked for. + - **Two worked notebooks** in `examples/notebooks/`, R Markdown and Jupyter, reading the demo + clip's table: speaking time per speaker, the transcript, who is in view when, facial + expression over time, going wide, and who is in view while someone speaks. The Python one + runs in CI; the R one before each release (`tests/manual/results_out_r_notebook.md`). + - Docs: [`docs/usage/exporting_results.md`](docs/usage/exporting_results.md). + - **Start VideoAnnotator: a container that feels local** (spec 024). For researchers on their own computer, with Docker Desktop or Podman Desktop and no Python. - **`videoannotator-start`** (sh for Linux and macOS, PowerShell for Windows, with a one-line @@ -192,6 +254,14 @@ user follows from GitHub. ### Removed +- **openai-whisper** (spec 030): speech recognition runs on faster-whisper. The `audio` extra no + longer builds a package from source at install. The unused Whisper embedding helpers and the + Hugging Face transformers branch of `WhisperBasePipeline` (its transcription was never + implemented) went with it. + +- **Two settings that did nothing** (spec 026): `scene_detection`'s `enable_clip_labels` and + `audio_processing`'s `enable_diarization` were never read by the code. A config that still sets + them runs as before; the validator says the field will be ignored. - Internal working documents moved out of the public docs into `docs/archive/`: the 2025 client/server team handoffs and QA reports from `docs/testing/`, `docs/Figure 1.docx`, dated fix notes, the coding-agent setup guide and the v1.3 scripts audit. @@ -213,6 +283,75 @@ user follows from GitHub. ### Changed +- **Security updates for known advisories in dependencies** (Trivy and `bun audit` before the + release). Server: FastAPI 0.146 with Starlette 1.9 (was 0.47), anyio 4.14, urllib3 2.8, aiohttp + 3.14, fsspec, mako, msgpack, soupsieve; lightning 2.6 for pyannote. Viewer: React Router's + router 1.23.4 (open-redirect fixes) in the shipped page; development tools (Lighthouse CI 0.15, + Playwright 1.64, and pinned fixes for tmp, basic-ftp and path-to-regexp). The demo-clip outputs + are identical before and after. Two high advisories have no fixed release yet, both in + development tools that never ship or run in CI: `braces` (Tailwind's build) and `extract-zip` + (Lighthouse's browser download). + +- **New default models** (spec 030, roadmap Phase 5c), so the public release's results come from + the defaults it keeps. Each changed on its upstream's published evaluation, after a run of the + demo clip compared with the same machine's previous run, one change at a time (library upgrades + first, with the old model). `videoannotator pipelines describe ` and the Pipelines page + give each default's reason and source ("Why this default"). Output formats are unchanged; their + content is not, so results from v1.5 and v1.6 differ. The comparisons are in + `specs/030-model-defaults/checks.md`. + - **Speech: Whisper `large-v3-turbo` on faster-whisper** (was `base` on openai-whisper). OpenAI's + card: large-v3 with its decoder cut from 32 layers to 4, "way faster, at the expense of a minor + quality degradation"; base is a tenth of its size. The new runtime alone gave the identical + transcript with `base`. With turbo, the demo clip keeps its 4 cues at the same moments and 3 are + reworded; it hears "Peek-a-boo!" in a peekaboo game where base heard "Take a move!". First use + downloads 1.6 GB, and it needs about 3 GB of GPU memory; on a CPU, `whisper_model: small` is + much faster. turbo can't translate, so `task: translate` with it now fails with a message + naming `large-v3` or `medium` (it used to return untranslated text). New setting + `compute_type`; the precision used is recorded in provenance and the methods paragraph ("at + float32"). To get the old model: `{"speech_recognition": {"whisper_model": "base"}}` (same + model on the new runtime; **v1.6.0rc1 is the last version with openai-whisper**, so v1.5 + transcripts can't be reproduced bit for bit). + - **Speaker turns: `pyannote/speaker-diarization-community-1`** (was `speaker-diarization-3.1`). + pyannote's benchmark gives it a lower diarization error rate on 10 of 12 sets (AMI headset + 18.8 → 17.0 %, DIHARD 3 21.4 → 20.2 %), equal on VoxConverse, worse on REPERE (broadcast TV); + it counts speakers better, and setup asks for one model licence instead of three. Demo clip: + 3 turns from one speaker where 3.1 gave 4 from two; 3.1 had split the mother's "Gotta smile, + huh?" and given its last half-second to a second speaker (checked by listening). To get the + old model: `{"speaker_diarization": {"model": "pyannote/speaker-diarization-3.1"}}`, with its + licences accepted. + - **People and pose: `yolo26n-pose`** (was `yolo11n-pose`), on ultralytics 8.4. COCO pose mAP + 50-95 57.2 vs 50.0 at the same size (Ultralytics). ultralytics 8.4 alone gave the same output. + Demo clip: the same people, tracks and labels; boxes overlap the old ones by a median IoU of + 0.92, keypoints moved a median 19 px. `iou_threshold` has no effect with YOLO26 (it has no + suppression step), and says so. To get the old model: + `{"person_tracking": {"model": "yolo11n-pose.pt"}}`. + - **Faces: age and gender are off by default**, and when off their models are no longer + downloaded or run (they used to run whenever any attribute was on, and were only left out of + the file). Trained on adult faces, they say nothing useful about infants. A first face run + downloads 125 MB instead of 1.2 GB. Demo clip: the same 19 faces in 11 frames with identical + boxes and expressions. Face analysis is now part of the demo-clip output check. To get them + back: `{"face_analysis": {"detect_age": true, "detect_gender": true}}`. + - The deprecated `audio_processing` pipeline follows the speech and speaker-turn defaults. + +- **The server's terminal says only what matters.** It used to print two lines for every request + (including every viewer asset and the health check every two minutes) and a page of startup + internals. Now it shows a ready line, each job started, finished (with how long it took and + where the results are) or failed (with the reason), pipeline installs, warnings and errors, as + `15:12:03 Finished clip.mp4 [1a2b3c4d] in 1m 12s; results in ~/VideoAnnotator/…`. Everything + still goes to the log files in `logs/`. If the viewer is calling the server without a working + API key, the terminal says so once (and how to connect it) instead of a stream of 401s. + `videoannotator server --verbose` brings back every line, including uvicorn's per-request log. + +- **The docs say which version they document.** Every page's header and browser tab read + "VideoAnnotator 1.6.0", and the footer says how to check yours. The version comes from + `pyproject.toml` when the site is built, so it can't drift, and a version bump now rebuilds the + site. `videoannotator --version` works too (before, only `videoannotator version` did). + +- **The viewer's docs links go to the documentation site.** The footer's "Docs" and Getting + started's "Documentation" pointed at a DeepWiki of the archived `video-annotation-viewer` repo; + they now open https://infantlab.github.io/VideoAnnotator/. Getting started no longer calls the + viewer "VAV". + - **A container never shows its own folders** (spec 024). In a container with no folder shared, My folders lists nothing (it used to list the container's home, `/root`) and says how to share a folder; upload still works. Plain installs keep the home-folder default. @@ -387,6 +526,26 @@ user follows from GitHub. ### Fixed +- **API keys were written into the log files** (security). Every request's headers, including + `Authorization: Bearer va_…`, went into `logs/api_server.log`, and rejected or expired keys were + logged with their first 10–16 characters. Logs now identify a key only by a short fingerprint + (`key:1a2b3c4d`) and never record headers. **If you ran an earlier version, delete the files in + `logs/` and make new API keys** (`videoannotator generate-token`); the old files may contain + your keys. + +- **The results player's timeline no longer misleads** (found 2026-10-16 in spec 023's + inventory). Every speaker had the same colour on the timeline and the overlay, because the + colour came from the ID's first letter and every diarization ID starts with "S"; VLM labels and + ELAN tiers could collide the same way. Each value now gets its own colour. The faces and + emotions rows had timeline switches with no lane behind them; they now say "overlay only". The + lane labelled as person tracking's timeline plots the largest person box's area, and now says + so ("size, not movement") instead of passing for motion. + +- **Person tracking's first run records its weights** (found 2026-10-16 with spec 025's methods + paragraph). The weights were hashed for provenance before ultralytics downloaded them, so a + first run recorded "weights file not found" and the methods said "revision not recorded". They + are now hashed after loading, from where ultralytics loaded them. + - **A new install couldn't install pipelines from the viewer** (found in the first-time-user run, 2026-10-05). Two faults, each enough on its own: - The API key the server makes on first start asked for admin rights, but they were never diff --git a/CITATION.cff b/CITATION.cff index cd7a58dc..7dd729bf 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -28,5 +28,6 @@ authors: orcid: "https://orcid.org/0000-0001-5846-3444" license: "MIT" repository-code: "https://github.com/InfantLab/VideoAnnotator" -version: "1.5.0" -date-released: "2026-07-19" +version: "1.6.0" +date-released: "2026-10-18" +doi: "10.5281/zenodo.16961751" diff --git a/CLAUDE.md b/CLAUDE.md index 6c92beb4..0690ae88 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,7 +25,7 @@ viewer/ # Video Annotation Viewer (React/Vite, Bun); built into src/ tests/ ├── unit/ integration/ pipelines/ api/ contract/ specs/-/ # spec-kit feature specs (spec.md, plan.md, tasks.md, ...) -docs/development/roadmap_v1.{5,6,7}.0.md, roadmap_v1.7_to_v2.0.md # release roadmap (v1.6.0 = public release) +docs/development/roadmap_v1.{5,6,7,8}.0.md, roadmap_v1.8_to_v2.0.md # release roadmap (v1.6.0 = quiet consolidation, v1.7.0 = public release with the new viewer, v1.8.0 = plugins) ``` ## Commands @@ -54,16 +54,13 @@ Backward Compatibility by Default). `/speckit-plan`'s Constitution Check gate ev against it. ## Current Plan - -`specs/024-container-feels-local/plan.md`: researchers start VideoAnnotator with `launcher/ -videoannotator-start` (sh, plus a PowerShell twin), which finds Docker or Podman, shares chosen -folders read-only at their real paths (remembered in a host `start.conf`), runs the slim GHCR -image on 127.0.0.1 and opens the viewer connected. App, installed pipelines (restored after -updates and restarts), model weights, activity and the researcher's files each persist on their -own. Inside a container the server never lists its own filesystem. After 024: -`specs/023-viewer-overhaul` (needs research first). - - +`specs/030-model-defaults/plan.md` (roadmap v1.6.0 Phase 5c), built 2026-10-18: faster-whisper +`large-v3-turbo` (openai-whisper removed; Principle V exception), pyannote +`speaker-diarization-community-1`, `yolo26n-pose` (ultralytics 8.4), face age/gender off and not +run. Open: the diarization demo-clip check and `.rttm` fixture need a Hugging Face token +(`docs/development/handover_v1.6.0_release.md`). Then the quiet v1.6.0 release; v1.7.0 is the +public one with the viewer overhaul (design `specs/023-viewer-overhaul`, built by `027-ask-ai`, +`028-video-page`, `029-videos-and-projects`). ## Recent Changes - 012-python-313-support: `requires-python` widens to `>=3.12,<3.14`; CI tests both; `.python-version` makes 3.13 the dev/Docker default; ruff/mypy targets stay at 3.12 (oldest diff --git a/README.md b/README.md index 68fb7f93..5dc5af78 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ You choose videos and pipelines in the browser; VideoAnnotator runs them and kee | Pipeline | What you get | File format | |---|---|---| | Person tracking | Each person's box and 17-point pose, with an ID that follows them | COCO | -| Face analysis | Faces with emotion, age and gender estimates | COCO | +| Face analysis | Faces with emotion estimates (age and gender optional) | COCO | | Face embeddings (OpenFace 3) | A 512-number description of each face, for telling people apart | JSON | | Scene detection | Where the scene cuts, and what kind of place each scene is | JSON | | Speech recognition | A transcript with word timings (Whisper) | WebVTT | @@ -37,10 +37,14 @@ The viewer then lets you: - Keep named lists of videos (datasets) to run studies on. - Write and test prompts for the VLM pipeline, and compare two models' labels against your own coding (ELAN `.eaf`). +- Get every result out as **one table** for R or Python (CSV or Parquet, the same columns for + every pipeline), as **ELAN files** to check alongside your own coding, or as a **methods + paragraph** with citations. Worked [R and Python notebooks](examples/notebooks/) start from the + table. Every output file records what made it: the pipeline and VideoAnnotator version, the model and its -exact weights, and the settings. That is what you need to report the method, and to get the same -results again later. +exact weights, and the settings. That is what you need to report the method (Export → Methods +paragraph writes it for you), and to get the same results again later. ## Get started @@ -66,7 +70,8 @@ You need [Docker Desktop](https://www.docker.com/products/docker-desktop/) or 3. **Run a job.** Choose **New job**, tick some videos and the pipelines to run. The first time you choose a pipeline, VideoAnnotator offers to install it. **Jobs** shows progress; open a finished job to see the results on the video. Results also go to a `VideoAnnotator` folder in - your home folder. + your home folder; **Export** on a run's page gives you all of them as one table for R or + Python. Allow about 1 GB of disk, plus up to 12 GB if you install every pipeline, and 16 GB of RAM. A GPU is optional. The VLM pipeline needs [Ollama](https://ollama.com) running on the same @@ -82,6 +87,8 @@ and installing from source in your own Python environment - **[Documentation](docs/README.md)**: install options, every pipeline, output formats, configuration and troubleshooting. - **[Getting started](docs/usage/GETTING_STARTED.md)**: a first job, step by step. +- **[Results out](docs/usage/exporting_results.md)**: the results table, ELAN files and the + methods paragraph, from the viewer, the command line (`videoannotator export`) or the API. - **[Command line and API](docs/usage/demo_commands.md)**: everything the viewer does can also be scripted, via `videoannotator` commands or the REST API (documented at http://127.0.0.1:18011/docs while the server runs). diff --git a/configs/README.md b/configs/README.md index 410f5237..92cf6b51 100644 --- a/configs/README.md +++ b/configs/README.md @@ -35,7 +35,7 @@ scene_detection: ```yaml person_tracking: # YOLO11 model settings - model_name: "yolo11n-pose.pt" # YOLO11 pose model + model_name: "yolo26n-pose.pt" # YOLO26 pose model confidence_threshold: 0.5 # Detection confidence threshold iou_threshold: 0.5 # IoU threshold for NMS diff --git a/configs/default.yaml b/configs/default.yaml index d76aadd2..e734fbe9 100644 --- a/configs/default.yaml +++ b/configs/default.yaml @@ -30,7 +30,7 @@ scene_detection: # Person Tracking Pipeline person_tracking: - model_name: "yolo11n-pose.pt" + model_name: "yolo26n-pose.pt" confidence_threshold: 0.5 iou_threshold: 0.5 track_mode: true diff --git a/configs/high_performance.yaml b/configs/high_performance.yaml index 2dd9ae13..4671a026 100644 --- a/configs/high_performance.yaml +++ b/configs/high_performance.yaml @@ -26,7 +26,7 @@ scene_detection: # Person Tracking Pipeline person_tracking: - model_name: "yolo11l-pose.pt" # Larger model for better accuracy + model_name: "yolo26l-pose.pt" # Larger model for better accuracy confidence_threshold: 0.4 # Lower threshold for more detections iou_threshold: 0.4 track_mode: true diff --git a/configs/lightweight.yaml b/configs/lightweight.yaml index 03b7e222..4d00a6ad 100644 --- a/configs/lightweight.yaml +++ b/configs/lightweight.yaml @@ -17,7 +17,7 @@ scene_detection: # Person Tracking Pipeline person_tracking: - model_name: "yolo11n-pose.pt" # Nano model for speed + model_name: "yolo26n-pose.pt" # Nano model for speed confidence_threshold: 0.6 # Higher threshold for precision iou_threshold: 0.6 track_mode: false # Disable tracking to save resources diff --git a/configs/openface3.yaml b/configs/openface3.yaml index fd8887f7..98373dbf 100644 --- a/configs/openface3.yaml +++ b/configs/openface3.yaml @@ -9,7 +9,7 @@ scene_detection: person_tracking: enabled: true - model: "yolo11n-pose.pt" + model: "yolo26n-pose.pt" conf_threshold: 0.4 iou_threshold: 0.7 track_mode: true diff --git a/configs/person_identity.yaml b/configs/person_identity.yaml index b78fcb79..c703ccd5 100644 --- a/configs/person_identity.yaml +++ b/configs/person_identity.yaml @@ -3,7 +3,7 @@ person_tracking: # Core tracking settings - model: "yolo11n-pose.pt" + model: "yolo26n-pose.pt" conf_threshold: 0.4 iou_threshold: 0.7 track_mode: true diff --git a/docs/README.md b/docs/README.md index f49073b3..bf52bb42 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,8 +40,8 @@ New here? Start with **[Getting started](usage/GETTING_STARTED.md)**. [pre-commit hooks](development/PRE_COMMIT_GUIDE.md). - [Testing overview](testing/testing_overview.md), [testing standards](testing/testing_standards.md) and [coverage](testing/coverage_report.md) -- [Roadmap for v1.6.0](development/roadmap_v1.6.0.md), then [v1.7.0](development/roadmap_v1.7.0.md) - and [v1.7 to v2.0](development/roadmap_v1.7_to_v2.0.md) +- [Roadmap for v1.6.0](development/roadmap_v1.6.0.md) (consolidation), then [v1.7.0](development/roadmap_v1.7.0.md) + (public release), [v1.8.0](development/roadmap_v1.8.0.md) (plugins) and [v1.8 to v2.0](development/roadmap_v1.8_to_v2.0.md) - [Changelog](../CHANGELOG.md) ## Reviewing the software diff --git a/docs/development/dependency_audit_v1.6.0.md b/docs/development/dependency_audit_v1.6.0.md index 5df1efdd..c599e2f5 100644 --- a/docs/development/dependency_audit_v1.6.0.md +++ b/docs/development/dependency_audit_v1.6.0.md @@ -236,3 +236,28 @@ In order, each one small enough to review. The pipeline review's recommendations opencv `<5` cap. This is the path to Python 3.14, so do it before rc1 if Phase 5's benchmark allows, and add the 3.14 CI job with it. 6. **Viewer dependencies** (separate audit). + +## 9. Model-default upgrades (spec 030, 2026-10-18) + +- **ultralytics 8.3.183 → 8.4.182** (`person`: `>=8.4,<9`, needed for YOLO26). Adds + `ultralytics-platform` (an SDK for Ultralytics' hosted platform, imported only when + `ultralytics.Platform` is used, which VideoAnnotator never does), `polars` and `nvidia-ml-py`; + drops `py-cpuinfo`. Analytics moved to `ultralytics.utils.events`; still switched off for the + process (`tests/unit/test_no_telemetry.py`). Same output on the demo clip with yolo11n-pose + (spec 030 checks.md, step 4a). Licence unchanged (AGPL-3.0). +- **openai-whisper → faster-whisper 1.2.1** (`audio`: `>=1.2.1,<2`). Brings CTranslate2 4.8.3 + (wheels for every supported platform and Python), PyAV, tokenizers and onnxruntime (used only by + its voice-activity filter, which we leave off); removes openai-whisper (the only package built + from source at install, and `[tool.uv.extra-build-dependencies]` with it) and tiktoken. No torch + dependency. On Linux, CTranslate2 found the CUDA 12 / cuDNN 9 libraries from torch's NVIDIA + wheels with no configuration. onnxruntime gets imported anyway (torchmetrics, via pyannote, + imports it when present), so its Windows telemetry is switched off when the audio pipelines + load. Same transcript on the demo clip with `base` (spec 030 checks.md, step 5a). All MIT. +- **Security updates before v1.6.0 (2026-10-18)**: Trivy reported 2 critical and 19 high advisories + in `uv.lock` and `viewer/bun.lock`, none new in v1.6.0. Upgraded within our constraints: + fastapi 0.142 → 0.146 (starlette 0.47 → 1.9), anyio, urllib3, aiohttp, fsspec, mako, msgpack, + soupsieve, lightning/pytorch-lightning 2.5 → 2.6. Viewer: `bun audit fix`, react-router-dom 6.30 + (@remix-run/router 1.23.4), @lhci/cli 0.15, @playwright/test 1.64, and `resolutions` for tmp, + basic-ftp and path-to-regexp (only Lighthouse CI uses them). Every demo-clip output identical + after. Left, with no fixed release published: `braces` 3.0.3 (Tailwind build) and + `extract-zip` 2.0.1 (Lighthouse/Puppeteer); both development-only. diff --git a/docs/development/handover_v1.6.0_release.md b/docs/development/handover_v1.6.0_release.md new file mode 100644 index 00000000..109742b5 --- /dev/null +++ b/docs/development/handover_v1.6.0_release.md @@ -0,0 +1,59 @@ +# Handover: finishing v1.6.0 (updated 2026-10-18) + +For the next session. Branch `1.6-dev`, all work committed locally, nothing pushed. v1.6.0 is the +**quiet consolidation release**; v1.7.0 is the public one +([`roadmap_v1.6.0.md`](roadmap_v1.6.0.md), [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)). + +## Done 2026-10-18 + +- The size-analysis test failure (a mocked YOLO with no `ckpt_path`), fixed. Roadmaps updated. +- **Spec 030 (model defaults, Phase 5c)**: planned, tasked and built, one change per commit, each + checked on the demo clip against the previous step on the same machine (a GTX 1060 6GB in this + container). Record: [`specs/030-model-defaults/checks.md`](../../specs/030-model-defaults/checks.md). + Faces: age/gender off and not run (the `_deepface_actions()` bug); speaker turns: community-1; + people: yolo26n-pose on ultralytics 8.4; speech: large-v3-turbo on faster-whisper (openai-whisper + removed). Settings can carry `reason`/`reason_source` ("Why this default"). Fixtures + re-baselined; CHANGELOG written. + +## Next, in order + +1. ~~A run with a Hugging Face token~~: done 2026-10-18. Community-1 gives one speaker where 3.1 + gave two; listening showed 3.1 had split the mother's sentence. Fixtures complete; the + baseline passes 8 of 8. **Revoke the token pasted into the chat on 2026-10-18** if it isn't + already (Hugging Face rejected it by then; the saved one is a different token). +2. **Re-run the R notebook** (`tests/manual/results_out_r_notebook.md`): its table was + regenerated from the new fixtures. +3. **Release** (ask before each outward step): optional `v1.6.0rc2`; PR `1.6-dev` → `master` + (spec 024's CI end-to-end runs start there; CI's install matrix confirms the `audio` extra on + macOS/Windows); tag `v1.6.0`; GitHub release whose notes say the viewer is being redesigned for + v1.7.0 and that v1.6.0rc1 was the last with openai-whisper; Zenodo DOI; `CITATION.cff`. + +## Waiting on the maintainer + +- Look in a browser at: the pipeline guide (Pipelines page; New job's settings step) and the + Hugging Face token flow with a real token (Speaker turns card; Settings → API Configuration). +- Delete the old `logs/` files and make a new API key: before `c2edfbd` every request's + `Authorization` header (full keys) went into `logs/api_server.log`. +- Decide: should a server with authentication off (`scripts/dev.sh`, `--dev`) treat callers as + admin? Today admin endpoints refuse every keyless request, so admin actions fail under `dev.sh`. +- API responsiveness during a long, heavy job (roadmap Phase 2): check during spec 030's GPU runs. + +## Done 2026-10-16/17 (for the roadmap and release notes) + +- Spec 023 (viewer overhaul) designed: research, decisions 1–8, journeys, split into 026–029. + 027–029 are v1.7.0. +- Spec 026, pipeline guide: built (`c018523`, `27510c4`, `2da16c6`). +- Spec 031, Hugging Face token from the viewer/Settings/CLI, no restart (`0ffd059`, `ab400a7`, + `fdc5346`, `9e8b530`). +- Docs: version on every page; `videoannotator --version` (`79582bd`). +- Viewer: timeline colours and labels that don't mislead (`41a38d2`); DeepWiki links gone + (`4842843`). +- Server: terminal shows only what matters; API keys no longer logged (`c2edfbd`). + +## Environment notes + +- The container has no git identity: commit with + `git -c "user.name=Caspar Addyman" -c user.email=infantologist@gmail.com commit …`. +- Test state 2026-10-18: full suite green; the real-model baseline passes 8 of 8 on the GTX 1060. `bats` isn't installed here, so `pre-commit run --all-files` reports the + launcher hook failing; it also reformats about 20 old Markdown files (pre-existing drift). +- `mkdocs` needs `uv sync --inexact --group docs`; build with `uv run --no-sync mkdocs build --strict`. diff --git a/docs/development/roadmap_v1.5.0.md b/docs/development/roadmap_v1.5.0.md index 0bbb828e..851e2600 100644 --- a/docs/development/roadmap_v1.5.0.md +++ b/docs/development/roadmap_v1.5.0.md @@ -13,7 +13,7 @@ This release scopes down the full modular-pipeline-architecture design in to the subset that matters for the resubmission: a slim, extras-based install, and a runtime integration between VideoAnnotator and Video Annotation Viewer. The plugin-discovery and remote-dispatch seams from that spec are real and worth building, but they are not blocking a -resubmission — they move to [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md) (planned as v1.6.0 until +resubmission — they move to [`roadmap_v1.8.0.md`](roadmap_v1.8.0.md) (now v1.8.0; planned as v1.6.0 until 2026-09-24, when v1.6.0 became the public release: [`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)). **Target Release**: Alongside/ahead of the JOSS resubmission (Phase 2 — /viewer integration — @@ -42,7 +42,7 @@ source files, CUDA-only torch pin, stale `openai-whisper` pin) — see commits ` - ✅ **LAION is opt-in, not default** — LAION Empathic-Insight (face and voice emotion) is an actively-changing external project with a narrow user base; it should not tax every install. -**Out of scope for v1.5.0** (moved to v1.7.0 — see [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)): +**Out of scope for v1.5.0** (moved to v1.8.0 — see [`roadmap_v1.8.0.md`](roadmap_v1.8.0.md)): - ❌ Third-party plugin discovery via `importlib.metadata.entry_points` - ❌ Dispatcher ABC / HTTPDispatcher / SlurmDispatcher seam - ❌ Cross-cutting utilities split into a sibling `videoannotator-utils` package diff --git a/docs/development/roadmap_v1.6.0.md b/docs/development/roadmap_v1.6.0.md index 8b571798..fdad9564 100644 --- a/docs/development/roadmap_v1.6.0.md +++ b/docs/development/roadmap_v1.6.0.md @@ -1,9 +1,21 @@ # 🚀 VideoAnnotator v1.6.0 Development Roadmap +> **Rescoped 2026-10-17 (Caspar).** v1.6.0 was to be the public release. Announcing it and piloting +> it just before the viewer is redesigned would leave the public version in an awkward state, and +> pilot labs would report on a viewer about to change. So v1.6.0 is now a **quiet consolidation +> release**: tagged, citable and installable, not announced. The public release, with the +> redesigned viewer, the tutorial, the pilot and the announcements, is v1.7.0 +> ([`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)); the plugin ecosystem moved to v1.8.0. Phases 4–7 +> below were moved there, except the model-default refresh (Phase 5c), which stays so the public +> version's outputs are stable. New here: Phase 3b, the pipeline guide (spec 026). References to +> Phases 4, 5, 6 and 7 in earlier entries mean the plan before this date; in v1.7.0 they are Phase 5 +> (Agents), Phase 2 (Connectors, old 5a), spec 029 (Corpus overview) and Phase 3 (Docs). + ## Release Overview -VideoAnnotator v1.6.0 is the **Public Release**: the first version we announce to the developmental -research community, with a tutorial, mailing-list posts and a webinar. +VideoAnnotator v1.6.0 is the **Consolidation Release**: one repository, current dependencies, a +clean first contact, and results out in formats researchers use. It is released quietly; the +public release is v1.7.0. v1.5.0 made the tool usable by us without a terminal: in-app pipeline installs, batches, folder ingest, the VLM pipeline, and the viewer bundled at `/viewer` (see @@ -11,7 +23,7 @@ ingest, the VLM pipeline, and the viewer bundled at `/viewer` (see to install it, run it on their own videos, and get the results into their own analysis without asking us. -**Why this shape**: on 2026-09-24 the JOSS submission +**Why this shape** (written for the public release, which v1.7.0 now is): on 2026-09-24 the JOSS submission ([#10182](https://github.com/openjournals/joss-reviews/issues/10182)) was sent for an editorial scope check because no one outside the author team has used or contributed to it yet. JOSS accepts publications, outside use, integration with other research tools, or benchmarks with reproducible @@ -20,14 +32,15 @@ for its own reason, outside use. The pilot follows the release, and a benchmark (see Phase 5b). > Until 2026-09-24 this file held the plugin-ecosystem plan. That plan is now -> [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md), with its git history. - -**Target Release**: as complete as possible, as soon as possible. No conference deadline (BCCCD -dropped 2026-10-14). The pilot (and spec 024's T083) runs after the release. -**Current Status**: Phases 0–1 done bar non-blocking items; Phase 2 closing with spec 024's -walkthroughs (2026-10-14) -**Main Goal**: an outside lab can go from install to results in R or Python, on its own videos, -without help +> [`roadmap_v1.8.0.md`](roadmap_v1.8.0.md) (v1.7.0 until 2026-10-17), with its git history. + +**Target Release**: as soon as Phases 3b and 5c are done. No deadline (BCCCD dropped 2026-10-14). The +pilot (and spec 024's T083) moved to v1.7.0's release candidates. +**Current Status**: Phases 0–1 done bar non-blocking items; Phase 2 done bar the long-video checks +(spec 024 walkthroughs done 2026-10-17); Phase 3 (Results Out, spec 025) built 2026-10-16; Phase 3b +(pipeline guide, spec 026) built 2026-10-17; Phase 5c (spec 030) done 2026-10-18 +**Main Goal**: everything an outside lab will need under v1.7.0's new viewer, done and stable: +install, pipelines, results out **Prerequisites**: v1.5.0 tagged, after spec 011's SC-001 manual run (`tests/manual/pipeline_readiness_e2e.md`) @@ -46,14 +59,14 @@ without help - ✅ **No model change without a reason on record.** A pipeline default changes on the upstream model's published evaluation, after its output on the demo clip has been checked against the baseline (`tests/integration/test_output_baseline.py`). The CHANGELOG and model card say why. -- ✅ **Pilot after the release.** Outside testing is what the release is for. +- ✅ **Pilot on what will be announced.** Outside testing runs on v1.7.0's release candidates, not + on a viewer about to be replaced (2026-10-17). --- ## 📋 v1.6.0 Deliverables -Phases are in order. Phases 0–3 make release candidate 1; the pilot runs from there to release and -reorders Phases 4–7. +Phases 0–3 made the release candidates; Phases 3b and 5c finish the release. ### Phase 0: One Repository @@ -573,9 +586,10 @@ job in the wizard from memory, although every job already stores its `selected_p folder the videos are in, shares it read-only at its real path, remembers it, and never shows the container's own folders. Spec: `specs/024-container-feels-local/`. Built 2026-10-06 (`videoannotator-start` for Linux/macOS and Windows, the slim image on GHCR, - installed pipelines restored after recreation, Settings' Stop sharing); open: the CI - end-to-end runs with Docker and Podman (first run on a push to `master`), the quickstart - walkthroughs on Linux and Windows, and a pilot researcher on macOS (T082-T084). + installed pipelines restored after recreation, Settings' Stop sharing). Quickstart + walkthroughs done 2026-10-17: Windows by hand (T084), Linux reliable in daily use (T082). + Open: the CI end-to-end runs with Docker and Podman (first run on a push to `master`), and a + pilot researcher on macOS (T083, with v1.7.0's pilot). - [ ] **The API and CLI from a launcher install** (found 2026-10-16): a researcher started with `videoannotator-start` has no `videoannotator` command on the host, and the API key it made sits in `start.conf`, which no command prints. Phase 3's R and Python notebooks and @@ -593,8 +607,27 @@ job in the wizard from memory, although every job already stores its `selected_p Compose to its own page, troubleshooting into `troubleshooting.md`, and drop the manual OpenFace build. -**Not in this phase**: a visual redesign. Decide on one after the pilot, from what outside users -say. +- [x] **Fixes from 2026-10-16/17**, found while designing spec 023 and using the server daily: + - **API keys were written to the log** (security): request logging put every request's + headers, `Authorization: Bearer va_…` included, into `logs/api_server.log`, and auth + failures logged 10–16 key characters. Headers are no longer logged; a key appears only as a + sha256 fingerprint (`c2edfbd`). Anyone with logs from before this should delete them and + make a new key. + - **A terminal that says what matters** (`c2edfbd`): the server prints ready, each job + started/finished/failed (with duration, results folder or reason), pipeline installs and + stopping; per-request lines and library warnings go to `logs/` only. A viewer calling + without a working key gets one plain hint instead of a stream of 401s. + - **Hugging Face token from the viewer, Settings or the CLI, no restart** (spec 031: + `0ffd059`, `ab400a7`, `fdc5346`, `9e8b530`): an admin saves it to a 0600 file beside the + database, checked with Hugging Face first and never returned; + `videoannotator settings huggingface-token` does the same from a terminal. + - **Timeline colours that don't mislead** (`41a38d2`): every speaker had the same colour + (the hue came from the ID's first letter, always "S"); hues now come from each value's + rank. Timeline switches with no lane behind them are gone. + - **The version on every docs page**, and `videoannotator --version` (`79582bd`). + +**Not in this phase**: a visual redesign. It is v1.7.0's first phase (design in +[`specs/023-viewer-overhaul`](../../specs/023-viewer-overhaul/spec.md), built by specs 026–029). --- @@ -605,80 +638,95 @@ right for provenance and wrong for analysis: to get one table in R, a researcher parser per pipeline. **Solution**: -- [ ] **Tidy export**: one row per event (video, pipeline, track or person, start, end, label, +- [x] **Tidy export**: one row per event (video, pipeline, track or person, start, end, label, value, confidence, model), as CSV and Parquet, per job and per batch or dataset, from the API, the CLI and a viewer button. This table also feeds the corpus view (Phase 6) and agents (Phase 4). -- [ ] **ELAN export**: `.eaf` with one tier per pipeline track. The viewer already parses `.eaf` + Done 2026-10-16 (spec 025, `specs/025-results-out/`): one long table, a row per event and + measure, 19 fixed columns (`contracts/table.md`, format version 1). Built from the result + files and their provenance by `src/videoannotator/exporters/`, so + `videoannotator export table ` works on any results folder with no server, and the + API and the viewer's Export menu give the same rows. Parquet is the optional `parquet` + extra. +- [x] **ELAN export**: `.eaf` with one tier per pipeline track. The viewer already parses `.eaf` (`src/lib/parsers/elan.ts`), so this closes the loop with the tool many labs already code in. -- [ ] **Methods paragraph**: `GET /api/v1/jobs/{id}/methods`, a CLI command and a viewer button. + Done 2026-10-16: EAF 3.0, validated against ELAN's schema in tests; built from the table's + rows; frame detections become "in view" spans. The viewer contract test opens the export + and a copy edited in ELAN. Not yet opened in ELAN itself by a person (spec 025 quickstart). +- [x] **Methods paragraph**: `GET /api/v1/jobs/{id}/methods`, a CLI command and a viewer button. Returns the models, versions, weight revisions, parameters and citations as prose plus BibTeX, ready for a methods section. Needs the provenance fields from the reproducibility - item in [`roadmap_v1.7_to_v2.0.md`](roadmap_v1.7_to_v2.0.md), pulled forward: model revision + item in [`roadmap_v1.8_to_v2.0.md`](roadmap_v1.8_to_v2.0.md), pulled forward: model revision SHA, prompt SHA-256, quantisation. -- [ ] Two worked analysis notebooks, one R and one Python, reading the tidy export of the demo - video. - -**→ Release candidate 1 (`v1.6.0rc1`)**: Phases 0–3 done. - ---- - -### Pilot (after the release) - -Outside testing is what the release is for (Caspar, 2026-10-14), so it follows the tag. Spec 024's -T083 runs here too. + Done 2026-10-16, for jobs and runs (`/batches/{id}/methods` too). Citations live in each + pipeline's metadata (`citations:`). Found on the way, not fixed: some metadata defaults + differ from the pipelines' code defaults (scene `min_scene_length` 1.0 vs 2.0, Whisper's + punctuation strings), so the paragraph lists those settings as changed. +- [x] Two worked analysis notebooks, one R and one Python, reading the tidy export of the demo + video. Done 2026-10-16: `examples/notebooks/`; the Python one runs in CI, the R one was run + on R 4.3 and is checked before each release (`tests/manual/results_out_r_notebook.md`). -- [ ] 3–5 labs outside the team (conference contacts first) run it on about ten of their own - videos, with a concrete ask: tell us where it broke. -- [ ] Their reports go in the public issue tracker. Label starter tasks `good first issue`. -- [ ] Pilot feedback sets the order of v1.6.x and v1.7 work. -- [ ] With permission, record who used it and for what. +**→ Release candidates**: rc1 was cut on 2026-10-14 to test the install before Phase 3 landed; +rc2 carries Phase 3. --- -### Phase 4: Agents +### Phase 3b: Pipeline Guide -**Problem**: researchers increasingly work alongside an agent of their own (Claude Code, Codex, -Gemini CLI and others). The REST API is complete but an agent has to discover it from scratch, and -parts of the CLI still prompt interactively. - -**Solution**: -- [ ] **MCP server**: `videoannotator mcp` (stdio), and the same tools over HTTP on a running server. - Tools: pipeline readiness, submit a folder or batch, job and batch status, annotations for a - time window (tidy rows), the frame at time *t* as an image, comparison with an `.eaf` file, - and the methods paragraph. With frames plus annotations an agent can audit ("look at the ten - moments where the speaker label and the face on screen disagree"). -- [ ] **Agent Skill** shipped in the repo (`SKILL.md`): installing, choosing pipelines for a - research question, reading outputs, auditing disagreements. -- [ ] **A CLI that agents can drive**: `--json` on every command, no prompts when the flags are given - (`generate-token` still prompts), documented exit codes. -- [ ] `llms.txt` and a "Using VideoAnnotator with your agent" docs page. It says plainly that frames - sent to a cloud-hosted agent leave the machine. +- [x] [`specs/026-pipeline-guide`](../../specs/026-pipeline-guide/spec.md): each pipeline's metadata + says what it does in behavioural terms, splits its settings into behavioural (shown) and + technical (collapsed), and gives its homepage, published benchmarks and citation; the + viewer, the API and `videoannotator pipelines describe` show the same guide. Groundwork for + v1.7.0's run setup and model cards; useful on today's Pipelines page. + Done 2026-10-17: all 8 pipelines have a guide with sourced figures (spec 026 + research.md, "Sourced figures"); two settings the code never read are removed. +- [x] Remove the DeepWiki links (they point at the archived viewer repo) and "VAV" from Getting + started. Done 2026-10-17: the footer's and Getting started's docs links go to the docs site. --- -### Phase 5: Models - -**Candidates**: [`pipeline_landscape_v1.6.0.md`](pipeline_landscape_v1.6.0.md) (2026-10-01) lists, -per kind of processing, what we have, what's obsolete, the current best tools with licences, and -what developmental research uses that we don't do at all. Top of its list: voice type -classification (VTC 2) with adult word counts (ALICE) and conversational turns; infant looking -(iCatcher+); motion energy and dyadic synchrony; adult/child role per person; de-identified -export; caregiver prosody. - -#### 5a. Connectors - -- [ ] **OpenAI-compatible connector** alongside the Ollama one (`backends: [ollama, - openai_compatible]`), moved here from v1.7.0. One client covers Ollama, llama.cpp, vLLM, - LM Studio, SGLang and hosted APIs. -- [ ] **Structured outputs**: the prompt carries a JSON schema, so VLM labels come back parseable - rather than as free text. -- [ ] **Video input** through vLLM (Qwen-VL models) as a third sampling mode beside single frame and - burst. Ollama's Qwen3-VL has no video input as of September 2026. -- [ ] Hosted endpoints are off by default. Enabling one shows a plain warning that frames leave the - machine; for infant video that is an ethics-approval question. - -#### 5b. Benchmark: not in v1.6.0 +### Phase 5c: Refresh Model Defaults + +Each on the upstream evaluation plus a demo-clip check (principle: no model change without a reason +on record). Done before the public release so its outputs are stable. + +Spec: [`specs/030-model-defaults`](../../specs/030-model-defaults/spec.md) (specified and clarified +2026-10-17; next: plan, tasks). Library upgrades are checked on the demo clip on their own, before +the model changes (FR-002). Each switch needs a real-model run on a GPU and a deliberate +re-baseline; spec 026's benchmark entries change with each default. Backward compatibility is +waived for this change (Principle V exception, recorded in the spec): old diarization and pose +models stay reachable as a setting, the old speech runtime goes. + +- [x] **Speech**: `openai-whisper` `base` → faster-whisper with `large-v3-turbo`, in two steps + (runtime with `base`, then the model). openai-whisper is removed (Q3), so v1.6.0rc1 is the + last version with it. + Done 2026-10-18: the runtime alone gave the identical transcript; turbo rewords 3 of the demo + clip's 4 cues (hears "Peek-a-boo!"). A translate request with turbo now fails clearly. +- [x] **Diarization**: pyannote `speaker-diarization-3.1` → `speaker-diarization-community-1` + (Q2): lower DER on 10 of 12 published sets, and one model licence to accept instead of three. + Switched 2026-10-18. Demo clip: one speaker where 3.1 found two; listening showed 3.1 had + split the mother's sentence, so community-1 stays. +- [x] **Person**: `yolo11n-pose` → `yolo26n-pose` (COCO-pose mAP50-95 50.0 → 57.2). Ultralytics + stays for v1.6.0. Done 2026-10-18 (ultralytics 8.4 first: same output; it had moved the + analytics switch-off, now followed). +- [x] **Face**: age and gender off by default, and their models neither downloaded nor run when off. + Today `_deepface_actions()` (`pipelines/face_analysis/face_pipeline.py`) runs all three + models whenever any is on: a bug, fixed here. DeepFace's own expression model stays (Q1), + described in the guide as trained on adult faces; replacing `face_analysis` is later work. + The detector is already RetinaFace (replaced the Haar cascade earlier in v1.6.0); its reason + goes into the metadata and guide. Face analysis joins the demo-clip baseline if RetinaFace + finds faces in it. + Done 2026-10-18: 125 MB first download instead of 1.2 GB; RetinaFace finds 19 faces in the + demo clip, so face analysis joined the baseline. +- [x] **OpenFace 3**: `openface-test` confirmed as the distribution CMU names (dormant since + 2025-06, non-commercial licence); the `==0.1.13` pin stays. Spec 030 writes this into its + metadata (FR-026). +- [ ] Caspar: ask LAAC-LSCP about VTC 2's licence (the repository has none). Open; record the + date asked. VTC 2 is a v1.7.0+ candidate whatever the answer. +- Moved to v1.7.0 (2026-10-17, spec 030 Q2): the half-day Sortformer spike. It informs lifting the + torch cap later, not this release's default. Audit §8. + +**Benchmark: not in v1.6.0 or v1.7.0** (was Phase 5b). Dropped 2026-10-14 (Caspar). The pipelines wrap off-the-shelf models that come with their own published evaluations, and a proper benchmark is a project in itself: no human-coded infant video @@ -687,115 +735,50 @@ CC BY-NC-SA 3.0) or can't be redistributed (BabyView, on Databrary). Model cards upstream evaluations and say plainly that few were trained or tested on infants. Labs can check a pipeline against their own ELAN codes in the viewer's Compare page. Revisit after the pilot. -#### 5c. Refresh defaults (each on the upstream evaluation plus a demo-clip check) - -- [ ] **Face** (if the Phase 1 review keeps a deepface-based pipeline): replace the default OpenCV - Haar-cascade detector (`face_pipeline.py`, `detector_backend: opencv`) with YuNet or - RetinaFace. Turn age and gender off by default: the models were trained on adults and say - nothing useful about infants. Choose the emotion model using Uwerikowe et al.'s comparison of - facial-emotion models on caregiver–child video. -- [ ] **Speech**: `openai-whisper` (built from source at install) → faster-whisper with - large-v3-turbo. -- [ ] **Diarization**: pyannote `speaker-diarization-3.1` → `speaker-diarization-community-1`, - which mainly improves speaker counting and keeps speaker identity consistent across a - recording. The pyannote.audio 4 library upgrade itself happens in Phase 1. Weigh NVIDIA - Streaming Sortformer alongside, with dependency health as a criterion: pyannote is what holds - torch at 2.11 (via the discontinued torchaudio) and upstream shows no plan to drop it. - Background and decision: `dependency_audit_v1.6.0.md` §8. -- [ ] Before choosing: a half-day Sortformer spike (installs on torch 2.14 without - torchaudio? NeMo's telemetry? install size? demo-clip output). Audit §8. -- [ ] Caspar: ask LAAC-LSCP about VTC 2's licence (the repository has none). VTC 2 also depends on - pyannote, so adopting it keeps the torch cap. Audit §8. -- [ ] **Person**: `yolo11n-pose` → `yolo26n-pose`. -- [ ] **OpenFace 3**: confirm `openface-test` is the maintained distribution (the Phase 1 - audit keeps it at `==0.1.13`). - -#### 5d. New pipelines: whichever the pilot labs ask for - -Candidates, none committed (the Phase 1 pipeline review may promote some before the pilot): -- **Voice Type Classifier**: key child, other child, female adult, male adult. Already standard in - child-language research. VTC 2 depends on pyannote.audio and has no stated licence (audit §8). -- **Gaze target** (Gaze-LLE), for joint attention. -- **Text-prompted segmentation and tracking** (SAM 3), e.g. "infant", "adult". Licence to check. - --- -### Phase 6: Corpus Overview - -**Problem**: the viewer shows one video at a time. Researchers think in corpora: which videos -failed, where the pipelines disagree, what the whole dataset looks like. - -**Solution**: -- [ ] Implement [`specs/010-corpus-analysis-foundations`](../../specs/010-corpus-analysis-foundations/spec.md) - (dataset summary endpoint) on top of the tidy export. -- [ ] A corpus page in the viewer: one row per video, one small track per pipeline, sortable (by - speaker/face disagreement, coverage, failures). A click opens the existing single-video - timeline. This is the paper's cross-modal audit, at corpus scale. -- [x] Previous/next between a batch's videos in the results viewer (item 4 of spec 008's viewer - handoff). - Done in Phase 0 (see "Results view isn't batch-aware"). - ---- +### Release (quiet) -### Phase 7: Docs and Tutorials - -- [ ] **Tutorial**: the Peekaboo demo video from start to finish (install, run, review, export, - analyse in R and Python). -- [ ] **Which pipeline for which question**, in the field's terms: caregiver speech, infant - vocalisations, movement, faces, joint attention, touch. -- [ ] **A model card per pipeline**: the model, what it was trained on (few include infants), known - failure modes, licence (Ultralytics is AGPL-3.0; pyannote models are gated), what to cite. -- [ ] **Data protection**: what runs where, what never leaves the machine, what changes with a - hosted backend or a cloud agent. -- [ ] **Contributing a pipeline**: a guide and template, with `vlm_annotation` as the worked - example. Contributions go into core by pull request; entry-point plugins wait for v1.7.0. -- [ ] **A 10-minute screencast**, reused for the webinar and conference tutorial. +- [ ] `v1.6.0rc2` with Phase 3; then Phases 3b and 5c. +- [ ] Tag `v1.6.0`, GitHub release, Zenodo DOI, `CITATION.cff` updated. No announcements. +- [ ] Release notes say plainly that the viewer is being redesigned for v1.7.0, the public + release. --- -### Launch +## ➡️ Moved to v1.7.0 (2026-10-17) -- [ ] Tag `v1.6.0`, GitHub release, Zenodo DOI, `CITATION.cff` updated. -- [ ] Update `paper/paper.md` and its research impact statement with the pilot's use, and tell the - editors on #10182. -- [ ] Announce: mailing lists, webinar, conference tutorial. - ---- +The pilot, Phase 4 (Agents), Phase 5a (Connectors) and 5d (New pipelines), Phase 6 (Corpus +Overview, now spec 029's Videos page), Phase 7 (Docs and Tutorials) and the Launch. See +[`roadmap_v1.7.0.md`](roadmap_v1.7.0.md). ## ❌ Out of Scope for v1.6.0 +- The viewer overhaul (v1.7.0). +- Publicity, the pilot, the tutorial and screencast (v1.7.0, written against the new viewer). - Plugin ecosystem: entry-point discovery, `Dispatcher` ABC, `videoannotator-utils` - ([`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)). -- The viewer overhaul, placeholder [`specs/023-viewer-overhaul`](../../specs/023-viewer-overhaul/spec.md) - (timeline, movement metric, reaching the data, multiple videos, researcher notes). After the - release, informed by the pilot (Caspar, 2026-10-14). -- Remote and HPC dispatch ([`roadmap_v1.7_to_v2.0.md`](roadmap_v1.7_to_v2.0.md)). -- Pose, hand and motion specialist pipelines, unless pilot labs ask. + ([`roadmap_v1.8.0.md`](roadmap_v1.8.0.md)). +- Remote and HPC dispatch ([`roadmap_v1.8_to_v2.0.md`](roadmap_v1.8_to_v2.0.md)). --- ## ✅ Success Criteria -- [ ] Changing viewer code cannot ship a stale bundle: CI fails on a mismatch. -- [ ] Every version constraint in `pyproject.toml` and the viewer's `package.json` has a recorded +- [x] Changing viewer code cannot ship a stale bundle: CI fails on a mismatch (Phase 0, + `scripts/build_viewer.sh --check`). +- [x] Every version constraint in `pyproject.toml` and the viewer's `package.json` has a recorded reason in the dependency audit, and the release supports Python 3.13 (3.14 as soon as TensorFlow does). -- [ ] Every default model has a model card citing its upstream evaluation, and every change of - default has its reason in the CHANGELOG. -- [ ] Everything the viewer does is available from the CLI with `--json` and through MCP. -- [ ] The viewer has zero TypeScript errors and no placeholder pages. +- [x] Every change of default model has its reason in the CHANGELOG (Phase 5c). +- [x] The viewer has zero TypeScript errors (checked 2026-10-16). +- [ ] The viewer has no placeholder pages. - [ ] Nothing leaves the user's machine unless they switched on a hosted backend. -**After the release** (the pilot's measures, not release gates): - -- [ ] A researcher outside the team installs it, works through the tutorial on their own machine, - and gets the tidy export into R or Python without help. -- [ ] At least three labs outside the team have run it on their own videos (documented, with - permission). -- [ ] At least one issue or pull request from outside the team has been resolved or merged. +Moved to v1.7.0: model cards for every default model; everything the viewer does available from +the CLI with `--json` and through MCP; the pilot's measures. --- -**Last Updated**: 2026-09-30 -**Target Release**: as soon as complete (see the top of this file) -**Status**: Planning Phase — Public Release +**Last Updated**: 2026-10-18 (Phase 5c done) +**Target Release**: as soon as Phases 3b and 5c are done +**Status**: Release candidates — Consolidation Release diff --git a/docs/development/roadmap_v1.7.0.md b/docs/development/roadmap_v1.7.0.md index 091498a7..0b5329c2 100644 --- a/docs/development/roadmap_v1.7.0.md +++ b/docs/development/roadmap_v1.7.0.md @@ -1,180 +1,213 @@ # 🚀 VideoAnnotator v1.7.0 Development Roadmap -> **Renumbered 2026-09-24.** This plan was written as v1.6.0 (last updated 2026-07-08) and moved -> here unchanged except for status marks. v1.6.0 is now the public release (see -> [`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)): two parts of this plan moved into it, the -> OpenAI-compatible connector and the job-execution consolidation (done in spec 006). The plugin -> ecosystem below comes after the public release, shaped by what outside labs ask for. +> **New 2026-10-17.** The public release moved here from v1.6.0 (Caspar): the viewer is redesigned +> first, so what is announced and piloted is what stays. v1.6.0 is now a quiet consolidation +> release ([`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)). The plugin plan that was v1.7.0 is now +> [`roadmap_v1.8.0.md`](roadmap_v1.8.0.md). Phases moved from v1.6.0 keep their text and history. ## Release Overview -VideoAnnotator v1.7.0 (planned as v1.6.0) is the **Plugin Ecosystem & Local LLM Backends Release**. It picks up the parts -of [`specs/003-modular-pipeline-architecture/spec.md`](../../specs/003-modular-pipeline-architecture/spec.md) -that [`roadmap_v1.5.0.md`](roadmap_v1.5.0.md) deliberately deferred to keep the JOSS resubmission -scope small, and adds a new capability: local LLM/VLM pipelines running against a user's own -`ollama serve` or `llama.cpp` server instead of an in-process, bundled model. +VideoAnnotator v1.7.0 is the **Public Release**: the first version we announce to the developmental +research community, with a tutorial, mailing-list posts and a webinar, and the first with the +redesigned viewer. -**Target Release**: After the v1.6.0 public release -**Current Status**: Planning Phase -**Main Goal**: Complete the plugin architecture seam; make current-generation local models usable -without adding them as bundled, versioned dependencies +**Main Goal**: an outside lab can go from install to results in R or Python, on its own videos, +without help, and a researcher new to AI video analysis can ask questions of a video and get +systematic answers without help. +**Target Release**: when the pilot's fixes are in. No deadline. +**Current Status**: Planning. Viewer design done (spec 023); specs 026–029 specified 2026-10-16. +**Prerequisites**: v1.6.0 tagged. -**Prerequisites**: v1.5.0's extras-based install and metadata-driven registry loading (this release -builds directly on both — a plugin is just an externally-installed extras group, and a local-LLM -pipeline is just another `backends` entry in pipeline metadata). +--- + +## 🎯 Core Principles + +v1.6.0's principles, plus: + +- ✅ **Call things what they are** (spec 023 Decision 1): a video is a video, a folder a folder; the + viewer doesn't show the API's words. +- ✅ **Nothing to set up before the first result.** Projects and workflows are offered after the + fact, never required (spec 023 Decisions 2 and 5). +- ✅ **Pilot on what will be announced.** --- ## 📋 v1.7.0 Deliverables -### Phase 1: Plugin Ecosystem (User Story 3 of spec 003) +### Phase 1: Viewer Overhaul -**Problem**: v1.5.0 makes the *bundled* pipelines optional and slim, but a third party still cannot -ship a new pipeline without modifying VideoAnnotator's source. +Design: [`specs/023-viewer-overhaul`](../../specs/023-viewer-overhaul/spec.md) (inventory, comparable +tools, decisions, journeys). Built by, in order: -**Solution**: -- [ ] `Dispatcher` ABC with `LocalThreadDispatcher` preserving current behaviour — the seam for a - future `HTTPDispatcher`/`SlurmDispatcher`, not their implementation. -- [x] Consolidate the duplicate job-execution paths (`api/job_processor.py` and - `batch/batch_orchestrator._process_single_job`) into one function that both the CLI and API - call, per FR-007. Done in v1.5.0 as spec 006 (`54d2ca1`). -- [ ] Pipeline discovery via `importlib.metadata.entry_points`, unioned with the bundled YAML - metadata from v1.5.0. Reserved extras-group names enforced at the registry level; a - third-party plugin declaring a colliding name is a deterministic error (not silent override). -- [ ] Migrate cross-cutting utilities (`person_identity`, `automatic_labeling`, `model_loader`, - `size_based_person_analysis`) to a sibling `videoannotator-utils` package in the monorepo, so - plugin authors have a stable, independently-versioned surface to depend on. -- [ ] Reference plugin: a ≤ 50-line third-party pipeline package, installed alongside core, appearing - in `GET /api/v1/pipelines` and runnable through the standard CLI/API with zero core changes. - -**Reference**: spec 003 FR-006, FR-007, FR-008, FR-010; SC-004. - -**Still deferred to v2.0** (per spec 003 assumptions, unchanged): isolating Ultralytics (AGPL-3.0) -person-tracking into its own plugin package. - ---- - -### Phase 2: Local LLM/VLM Backend (Ollama / llama.cpp) - -**Status**: mostly done — the `vlm_annotation` pipeline landed on `v1.5.0` ahead of the rest of this -release, driven by a real user (a PhD researcher's touch-detection study). See -[`vlm_annotation_pipeline.md`](vlm_annotation_pipeline.md) for the full writeup, config reference, and -a step-by-step testing guide. Checklist below updated to reflect what actually shipped vs. what's -still open. - -**Problem**: VideoAnnotator's current model-loading pattern is in-process — pipelines call -`transformers.AutoModel.from_pretrained(...)` directly (as the LAION face pipeline did until its -removal in v1.6.0), -which means every capability upgrade means a new bundled, versioned model dependency. Meanwhile -capable open local models (e.g. the Qwen3 family) are now practically runnable on a researcher's own -machine via `ollama serve` or a `llama.cpp` server, without VideoAnnotator vendoring any weights. +- [ ] [`specs/027-ask-ai`](../../specs/027-ask-ai/spec.md): the VLM window. First answer in under a + minute, prompt history never lost, sampling shown on the timeline, a live local/remote + badge; the pipeline gains frame spacing, ranges and answers as they come. +- [ ] [`specs/028-video-page`](../../specs/028-video-page/spec.md): lanes as controls, one row per + person or speaker, zoom, the data table beside the video, the run per lane, notes; + streamed, with no folder question. +- [ ] [`specs/029-videos-and-projects`](../../specs/029-videos-and-projects/spec.md): one Videos page + with folders and projects (from saved datasets), settings projects remember, workflows + (from presets), the new nav, help in context. Takes over v1.6.0's Phase 6 (corpus + overview); [`specs/010-corpus-analysis-foundations`](../../specs/010-corpus-analysis-foundations/spec.md) + to be reconciled with it. +- [ ] Try the sketches on one or two researchers before 027's plan (spec 023 Process step 5). -**Solution**: -- [x] New pipeline family — landed as `vlm_annotation` (generic per-frame VLM classification/ - captioning driven by a user-supplied prompt, not scoped to a single task), not - `scene_description` as originally sketched here. -- [ ] One connector, not two: **deviation, not done as originally scoped.** `vlm_annotation` calls the - `ollama` Python package's `Client.chat()` directly rather than a hand-rolled OpenAI-compatible - `/v1/chat/completions` client — reusing the already-proven client de-risked landing a working - pipeline quickly. `backends: [ollama]` in the metadata leaves room for an `openai_compatible` - backend (covering both Ollama's and `llama.cpp`'s compatible endpoint) alongside it later - without a pipeline redesign, but that second connector is still unbuilt. **Moved to v1.6.0** - (Phase 4 of [`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)). -- [x] Gated behind a new `llm` extras group (`ollama` client + `cv2`; `httpx` was already core, so - needed no new declaration). -- [x] Uses `backends: list[str]` (`backends: [ollama]`) in the pipeline's metadata YAML. -- [x] Configurable base URL and model name (defaults `http://127.0.0.1:11434` / `qwen3.5:9b`); no - model weights are bundled or auto-pulled — `model` is just a default config value the user can - override, same as any other field. -- [x] `videoannotator diagnose` detects a reachable Ollama server. Done in v1.5.0 as spec 009 - (`diagnostics/ollama.py`). -- [x] Provenance: every annotation record carries `model`, `backend`, `base_url`, and `prompt` - directly (per-record, not just job-level metadata) — see Principle III in the constitution. - -**Also fixed along the way (not originally scoped here)**: `api/job_processor.py` was silently -dropping per-pipeline job config (`pipeline_class()` with no config, vs. -`batch/batch_orchestrator.py`'s correct `pipeline_class(pipeline_config)`) — jobs submitted via the -API would have ignored `vlm_annotation`'s prompt/model/sampling-mode config entirely. Fixed as part of -landing this pipeline; see `vlm_annotation_pipeline.md`'s "Known gaps" section for the broader -dual-job-execution-engine issue this points at, which Phase 1's job-execution-path consolidation -(above) should resolve properly. - -**Still not built**: the job-creation UX and review UI in `video-annotation-viewer` (prompt textarea, -sampling-mode picker, ELAN ground-truth comparison, cross-prompt disagreement view) — tracked as -Phases 3–4 in `vlm_annotation_pipeline.md`, not in this repo. - -**Why this matters for JOSS**: it's a genuine, defensible "why does this exist now" story — this -capability wasn't practical for most researchers' hardware a year ago — and it demonstrates the -plugin/backend seam from Phase 1 is real, not aspirational. - ---- - -### Phase 3: Server-Side Timeseries Post-Processing (pandas) - -**Problem**: Per-frame annotations (person bounding boxes/keypoints) ship as static per-frame COCO -records; anything temporal — e.g. how far a tracked person moved between frames — is currently left -to Video Annotation Viewer to compute client-side, where compute is more constrained than the -VideoAnnotator server. +--- -**Solution**: -- [ ] A `person_tracking` post-processing step that computes frame-to-frame movement deltas - (bbox-center or keypoint-centroid displacement) per `person_id`/`track_id`, using `pandas` — - already a core, no-extras dependency (confirmed unused in any active pipeline code as of - v1.5.0, so this adds zero new dependency weight, just a new use of an existing one). -- [ ] Additive-only export fields (e.g. `movement_delta_x`, `movement_delta_y`, `movement_speed`) - alongside the existing COCO annotation, not replacing anything — v1.5.x consumers unaware of - the new fields are unaffected (Principle V, Backward Compatibility by Default). -- [ ] Scope to person-tracking first (clearest "movement" semantics); revisit whether - face/gaze/other keypoint-bearing pipelines want the same treatment once the pattern is - proven. +### Phase 2: Connectors (was v1.6.0 Phase 5a) + +- [ ] **OpenAI-compatible connector** alongside the Ollama one (`backends: [ollama, + openai_compatible]`), moved here from the plugin plan (now v1.8.0). One client covers Ollama, llama.cpp, vLLM, + LM Studio, SGLang and hosted APIs. +- [ ] **Structured outputs**: the prompt carries a JSON schema, so VLM labels come back parseable + rather than as free text. +- [ ] **Video input** through vLLM (Qwen-VL models) as a third sampling mode beside single frame and + burst. Ollama's Qwen3-VL has no video input as of September 2026. +- [ ] Hosted endpoints are off by default. Enabling one shows a plain warning that frames leave the + machine; for infant video that is an ethics-approval question. +- [ ] The connector ships together with spec 027's local/remote badge and its confirmation before + a remote send, never before them. +- [ ] Structured outputs fit Ask AI's lesson that the question defines the answer (journey 1): the + window helps build the schema. + +--- + +### Phase 3: Docs and Tutorials, Written Against the New Viewer -**Reference**: raised in v1.5.0's merge follow-up discussion, not part of spec 003/004 — no -existing FR/SC coverage. Write a proper spec before implementing if this grows beyond a small, -additive post-processing step. +- [ ] **Tutorial**: the Peekaboo demo video from start to finish (install, run, review, export, + analyse in R and Python). +- [ ] **Which pipeline for which question**, in the field's terms: caregiver speech, infant + vocalisations, movement, faces, joint attention, touch. +- [ ] **A model card per pipeline**: the model, what it was trained on (few include infants), known + failure modes, licence (Ultralytics is AGPL-3.0; pyannote models are gated), what to cite. +- [ ] **Data protection**: what runs where, what never leaves the machine, what changes with a + hosted backend or a cloud agent. +- [ ] **Contributing a pipeline**: a guide and template, with `vlm_annotation` as the worked + example. Contributions go into core by pull request; entry-point plugins wait for v1.7.0. +- [ ] **A 10-minute screencast**, reused for the webinar and conference tutorial. --- +- [ ] Model cards and "which pipeline for which question" start from spec 026's pipeline guide + (v1.6.0 Phase 3b), so the docs and the app say the same. -### Phase 4: Model Management +--- + +### Phase 4: Pilot (on the release candidates) + +Outside testing runs on what will be announced (Caspar, 2026-10-17; before that it followed the +v1.6.0 tag, 2026-10-14). Spec 024's T083 (a pilot researcher on macOS) runs here too. -**Problem**: model weights are the largest thing VideoAnnotator puts on a user's disk (a full -install downloads several GB), and the only way to see or remove them is a file manager. Nothing -shows which model and which version a pipeline uses, how big it is, whether it's downloaded, or -which other pipelines share it. Builds on v1.6.0's single models directory -(`VIDEOANNOTATOR_MODELS_DIR`): one place to look makes one place to manage. +- [ ] 3–5 labs outside the team (conference contacts first) run it on about ten of their own + videos, with a concrete ask: tell us where it broke. +- [ ] Their reports go in the public issue tracker. Label starter tasks `good first issue`. +- [ ] Pilot feedback sets the order of v1.6.x and v1.7 work. +- [ ] With permission, record who used it and for what. + +--- + +--- + +### Phase 5: Agents (not a release gate) + +**Problem**: researchers increasingly work alongside an agent of their own (Claude Code, Codex, +Gemini CLI and others). The REST API is complete but an agent has to discover it from scratch, and +parts of the CLI still prompt interactively. **Solution**: -- [ ] A models inventory: every model the installed pipelines use, with its source (Hugging Face - repo, Whisper size, DeepFace file, YOLO checkpoint), version or revision, size on disk (or - download size if not downloaded), the pipelines that use it, and when it was last used. - `GET /api/v1/models` and `videoannotator models list`. -- [ ] Download ahead of time (`videoannotator models pull `, and a button), so the - first job doesn't sit at "Preparing" for minutes. -- [ ] Delete weights, with a warning naming the pipelines that will download them again. Admin-only - over the API, like extras install (005). -- [ ] A Models page in the viewer: the inventory, total disk use, pull and delete. -- [ ] Provenance: the model revision recorded with each job's results (Principle III) matches what - the inventory shows. +- [ ] **MCP server**: `videoannotator mcp` (stdio), and the same tools over HTTP on a running server. + Tools: pipeline readiness, submit a folder or batch, job and batch status, annotations for a + time window (tidy rows), the frame at time *t* as an image, comparison with an `.eaf` file, + and the methods paragraph. With frames plus annotations an agent can audit ("look at the ten + moments where the speaker label and the face on screen disagree"). +- [ ] **Agent Skill** shipped in the repo (`SKILL.md`): installing, choosing pipelines for a + research question, reading outputs, auditing disagreements. +- [ ] **A CLI that agents can drive**: `--json` on every command, no prompts when the flags are given + (`generate-token` still prompts), documented exit codes. +- [ ] `llms.txt` and a "Using VideoAnnotator with your agent" docs page. It says plainly that frames + sent to a cloud-hosted agent leave the machine. + +--- -**Reference**: raised in the v1.5.0 end-to-end run (2026-09-28), after a container rebuild silently -deleted every cached model. Write a spec before implementing. +--- + +### Phase 6: New Pipelines, Whichever the Pilot Labs Ask For + +Candidates, none committed (the Phase 1 pipeline review may promote some before the pilot): +- **Voice Type Classifier**: key child, other child, female adult, male adult. Already standard in + child-language research. VTC 2 depends on pyannote.audio and has no stated licence (audit §8). +- **Diarization on Sortformer** (moved from v1.6.0 Phase 5c, 2026-10-17, spec 030 Q2): first a + half-day spike. Does NVIDIA Streaming Sortformer install and run on a torch newer than 2.11 + without torchaudio? What does NeMo send over the network by default, and can it be switched off? + Install size? Output on the demo clip next to pyannote community-1's? Findings and the decision + go into the dependency audit §8 (spec 030 FR-028). pyannote is what holds torch at 2.11. +- **Gaze target** (Gaze-LLE), for joint attention. +- **Text-prompted segmentation and tracking** (SAM 3), e.g. "infant", "adult". Licence to check. + +--- + +--- + +### Phase 7: Test Audit (before the release candidates) + +The suite has grown to 1,934 tests in 159 files (2026-10-18), mostly by accretion. Spec 030 found +waste: `tests/pipelines/test_audio_speech_pipeline.py` and `test_audio_individual_components.py` +are near-copies; some tests can't fail (`test_whisper_base_pipeline_stage1.py` checks numpy +imports, `test_embedding_shape_validation` asserts only if a method that doesn't exist does, the +"integration" Whisper tests accept failure as success); about 290 lines mocked openai-whisper's +own API and had to be deleted, not updated, when the runtime changed. + +- [ ] Inventory: per file, what behaviour it protects, run time, and whether it can fail. +- [ ] Remove duplicates and tests that can't fail; merge the two audio files. +- [ ] Tests mock at our boundary (what a pipeline sends a model, what it does with the answer), + not a library's internals, so a library swap changes the mock, not the test's meaning. +- [ ] Slow or real-model tests carry their markers and stay out of the default run; record the + default run's time before and after. +- [ ] Coverage stays at or above the constitution's 80%, measured on what's left, with the gaps + that matter (provenance, exporters, the job path) named. + +--- + +### Launch + +- [ ] Tag `v1.7.0`, GitHub release, Zenodo DOI, `CITATION.cff` updated. +- [ ] Update `paper/paper.md` and its research impact statement with the pilot's use, and tell the + editors on #10182. +- [ ] Announce: mailing lists, webinar, conference tutorial. + +--- + +--- + +## ❌ Out of Scope for v1.7.0 + +- Plugin ecosystem ([`roadmap_v1.8.0.md`](roadmap_v1.8.0.md)). +- Remote and HPC dispatch ([`roadmap_v1.8_to_v2.0.md`](roadmap_v1.8_to_v2.0.md)). +- A benchmark of our own (see v1.6.0, Phase 5c). +- Human review or correction of pipeline output (shelved, spec 023). +- A movement measure per person (v1.8.0 Phase 3, server-side post-processing). --- ## ✅ Success Criteria -- [ ] A third-party plugin package (≤ 50 lines) is installable and runnable through the standard - CLI/API with zero VideoAnnotator core changes. -- [x] The CLI batch path and the API job-submission path share one job-execution function (spec 006). -- [x] A local-LLM pipeline runs end-to-end against a user-provided `ollama serve` instance with no - bundled model weights and no torch/transformers pulled in by the `llm` extra. -- [x] Provenance metadata for local-LLM pipeline runs records backend, base URL, and model identity. -- [ ] `person_tracking` output includes frame-to-frame movement-delta fields via a pandas-based - post-processing step, additive-only (no v1.5.x output field ever removed or renamed). -- [ ] A user can see every downloaded model's size, version and users, and pull or delete it, from - the CLI and the viewer without touching the filesystem. +- [ ] Every default model has a model card citing its upstream evaluation. +- [ ] The viewer shows none of the words job, batch, dataset or preset (spec 029 SC-002). +- [ ] A first-time user gets a first Ask AI answer on their own video in under a minute (spec 027 + SC-001). +- [ ] Nothing leaves the user's machine unless they switched on a hosted backend, and the viewer + says where the model runs. +- [ ] Everything the viewer does is available from the CLI with `--json` (MCP if Phase 5 lands). + +**The pilot's measures** (before launch): + +- [ ] A researcher outside the team installs it, works through the tutorial on their own machine, + and gets the tidy export into R or Python without help. +- [ ] At least three labs outside the team have run it on their own videos (documented, with + permission). +- [ ] At least one issue or pull request from outside the team has been resolved or merged. --- -**Last Updated**: 2026-09-24 (renumbered from v1.6.0; content last revised 2026-07-08) -**Target Release**: After the v1.6.0 public release -**Status**: Planning Phase — Plugin Ecosystem Focus +**Last Updated**: 2026-10-17 +**Target Release**: when the pilot's fixes are in +**Status**: Planning Phase — Public Release diff --git a/docs/development/roadmap_v1.8.0.md b/docs/development/roadmap_v1.8.0.md new file mode 100644 index 00000000..d2027ebd --- /dev/null +++ b/docs/development/roadmap_v1.8.0.md @@ -0,0 +1,185 @@ +# 🚀 VideoAnnotator v1.8.0 Development Roadmap + +> **Renumbered again 2026-10-17.** The public release moved from v1.6.0 to v1.7.0, so the viewer +> overhaul ships before any publicity ([`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)); v1.6.0 is a quiet +> consolidation release ([`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)). This plan moved from v1.7.0 +> to v1.8.0 unchanged except for version numbers. + +> **Renumbered 2026-09-24.** This plan was written as v1.6.0 (last updated 2026-07-08) and moved +> here unchanged except for status marks. v1.6.0 is now the public release (see +> [`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)): two parts of this plan moved into it, the +> OpenAI-compatible connector and the job-execution consolidation (done in spec 006). The plugin +> ecosystem below comes after the public release, shaped by what outside labs ask for. + +## Release Overview + +VideoAnnotator v1.8.0 (planned as v1.6.0, then v1.7.0) is the **Plugin Ecosystem & Local LLM Backends Release**. It picks up the parts +of [`specs/003-modular-pipeline-architecture/spec.md`](../../specs/003-modular-pipeline-architecture/spec.md) +that [`roadmap_v1.5.0.md`](roadmap_v1.5.0.md) deliberately deferred to keep the JOSS resubmission +scope small, and adds a new capability: local LLM/VLM pipelines running against a user's own +`ollama serve` or `llama.cpp` server instead of an in-process, bundled model. + +**Target Release**: After the v1.7.0 public release +**Current Status**: Planning Phase +**Main Goal**: Complete the plugin architecture seam; make current-generation local models usable +without adding them as bundled, versioned dependencies + +**Prerequisites**: v1.5.0's extras-based install and metadata-driven registry loading (this release +builds directly on both — a plugin is just an externally-installed extras group, and a local-LLM +pipeline is just another `backends` entry in pipeline metadata). + +--- + +## 📋 v1.8.0 Deliverables + +### Phase 1: Plugin Ecosystem (User Story 3 of spec 003) + +**Problem**: v1.5.0 makes the *bundled* pipelines optional and slim, but a third party still cannot +ship a new pipeline without modifying VideoAnnotator's source. + +**Solution**: +- [ ] `Dispatcher` ABC with `LocalThreadDispatcher` preserving current behaviour — the seam for a + future `HTTPDispatcher`/`SlurmDispatcher`, not their implementation. +- [x] Consolidate the duplicate job-execution paths (`api/job_processor.py` and + `batch/batch_orchestrator._process_single_job`) into one function that both the CLI and API + call, per FR-007. Done in v1.5.0 as spec 006 (`54d2ca1`). +- [ ] Pipeline discovery via `importlib.metadata.entry_points`, unioned with the bundled YAML + metadata from v1.5.0. Reserved extras-group names enforced at the registry level; a + third-party plugin declaring a colliding name is a deterministic error (not silent override). +- [ ] Migrate cross-cutting utilities (`person_identity`, `automatic_labeling`, `model_loader`, + `size_based_person_analysis`) to a sibling `videoannotator-utils` package in the monorepo, so + plugin authors have a stable, independently-versioned surface to depend on. +- [ ] Reference plugin: a ≤ 50-line third-party pipeline package, installed alongside core, appearing + in `GET /api/v1/pipelines` and runnable through the standard CLI/API with zero core changes. + +**Reference**: spec 003 FR-006, FR-007, FR-008, FR-010; SC-004. + +**Still deferred to v2.0** (per spec 003 assumptions, unchanged): isolating Ultralytics (AGPL-3.0) +person-tracking into its own plugin package. + +--- + +### Phase 2: Local LLM/VLM Backend (Ollama / llama.cpp) + +**Status**: mostly done — the `vlm_annotation` pipeline landed on `v1.5.0` ahead of the rest of this +release, driven by a real user (a PhD researcher's touch-detection study). See +[`vlm_annotation_pipeline.md`](vlm_annotation_pipeline.md) for the full writeup, config reference, and +a step-by-step testing guide. Checklist below updated to reflect what actually shipped vs. what's +still open. + +**Problem**: VideoAnnotator's current model-loading pattern is in-process — pipelines call +`transformers.AutoModel.from_pretrained(...)` directly (as the LAION face pipeline did until its +removal in v1.6.0), +which means every capability upgrade means a new bundled, versioned model dependency. Meanwhile +capable open local models (e.g. the Qwen3 family) are now practically runnable on a researcher's own +machine via `ollama serve` or a `llama.cpp` server, without VideoAnnotator vendoring any weights. + +**Solution**: +- [x] New pipeline family — landed as `vlm_annotation` (generic per-frame VLM classification/ + captioning driven by a user-supplied prompt, not scoped to a single task), not + `scene_description` as originally sketched here. +- [ ] One connector, not two: **deviation, not done as originally scoped.** `vlm_annotation` calls the + `ollama` Python package's `Client.chat()` directly rather than a hand-rolled OpenAI-compatible + `/v1/chat/completions` client — reusing the already-proven client de-risked landing a working + pipeline quickly. `backends: [ollama]` in the metadata leaves room for an `openai_compatible` + backend (covering both Ollama's and `llama.cpp`'s compatible endpoint) alongside it later + without a pipeline redesign, but that second connector is still unbuilt. **Moved to v1.7.0** + (Phase 2 of [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md); planned for v1.6.0 until 2026-10-17). +- [x] Gated behind a new `llm` extras group (`ollama` client + `cv2`; `httpx` was already core, so + needed no new declaration). +- [x] Uses `backends: list[str]` (`backends: [ollama]`) in the pipeline's metadata YAML. +- [x] Configurable base URL and model name (defaults `http://127.0.0.1:11434` / `qwen3.5:9b`); no + model weights are bundled or auto-pulled — `model` is just a default config value the user can + override, same as any other field. +- [x] `videoannotator diagnose` detects a reachable Ollama server. Done in v1.5.0 as spec 009 + (`diagnostics/ollama.py`). +- [x] Provenance: every annotation record carries `model`, `backend`, `base_url`, and `prompt` + directly (per-record, not just job-level metadata) — see Principle III in the constitution. + +**Also fixed along the way (not originally scoped here)**: `api/job_processor.py` was silently +dropping per-pipeline job config (`pipeline_class()` with no config, vs. +`batch/batch_orchestrator.py`'s correct `pipeline_class(pipeline_config)`) — jobs submitted via the +API would have ignored `vlm_annotation`'s prompt/model/sampling-mode config entirely. Fixed as part of +landing this pipeline; see `vlm_annotation_pipeline.md`'s "Known gaps" section for the broader +dual-job-execution-engine issue this points at, which Phase 1's job-execution-path consolidation +(above) should resolve properly. + +**Still not built**: the job-creation UX and review UI in `video-annotation-viewer` (prompt textarea, +sampling-mode picker, ELAN ground-truth comparison, cross-prompt disagreement view) — tracked as +Phases 3–4 in `vlm_annotation_pipeline.md`, not in this repo. + +**Why this matters for JOSS**: it's a genuine, defensible "why does this exist now" story — this +capability wasn't practical for most researchers' hardware a year ago — and it demonstrates the +plugin/backend seam from Phase 1 is real, not aspirational. + +--- + +### Phase 3: Server-Side Timeseries Post-Processing (pandas) + +**Problem**: Per-frame annotations (person bounding boxes/keypoints) ship as static per-frame COCO +records; anything temporal — e.g. how far a tracked person moved between frames — is currently left +to Video Annotation Viewer to compute client-side, where compute is more constrained than the +VideoAnnotator server. + +**Solution**: +- [ ] A `person_tracking` post-processing step that computes frame-to-frame movement deltas + (bbox-center or keypoint-centroid displacement) per `person_id`/`track_id`, using `pandas` — + already a core, no-extras dependency (confirmed unused in any active pipeline code as of + v1.5.0, so this adds zero new dependency weight, just a new use of an existing one). +- [ ] Additive-only export fields (e.g. `movement_delta_x`, `movement_delta_y`, `movement_speed`) + alongside the existing COCO annotation, not replacing anything — v1.5.x consumers unaware of + the new fields are unaffected (Principle V, Backward Compatibility by Default). +- [ ] Scope to person-tracking first (clearest "movement" semantics); revisit whether + face/gaze/other keypoint-bearing pipelines want the same treatment once the pattern is + proven. + +**Reference**: raised in v1.5.0's merge follow-up discussion, not part of spec 003/004 — no +existing FR/SC coverage. Write a proper spec before implementing if this grows beyond a small, +additive post-processing step. + +--- + +### Phase 4: Model Management + +**Problem**: model weights are the largest thing VideoAnnotator puts on a user's disk (a full +install downloads several GB), and the only way to see or remove them is a file manager. Nothing +shows which model and which version a pipeline uses, how big it is, whether it's downloaded, or +which other pipelines share it. Builds on v1.6.0's single models directory +(`VIDEOANNOTATOR_MODELS_DIR`): one place to look makes one place to manage. + +**Solution**: +- [ ] A models inventory: every model the installed pipelines use, with its source (Hugging Face + repo, Whisper size, DeepFace file, YOLO checkpoint), version or revision, size on disk (or + download size if not downloaded), the pipelines that use it, and when it was last used. + `GET /api/v1/models` and `videoannotator models list`. +- [ ] Download ahead of time (`videoannotator models pull `, and a button), so the + first job doesn't sit at "Preparing" for minutes. +- [ ] Delete weights, with a warning naming the pipelines that will download them again. Admin-only + over the API, like extras install (005). +- [ ] A Models page in the viewer: the inventory, total disk use, pull and delete. +- [ ] Provenance: the model revision recorded with each job's results (Principle III) matches what + the inventory shows. + +**Reference**: raised in the v1.5.0 end-to-end run (2026-09-28), after a container rebuild silently +deleted every cached model. Write a spec before implementing. + +--- + +## ✅ Success Criteria + +- [ ] A third-party plugin package (≤ 50 lines) is installable and runnable through the standard + CLI/API with zero VideoAnnotator core changes. +- [x] The CLI batch path and the API job-submission path share one job-execution function (spec 006). +- [x] A local-LLM pipeline runs end-to-end against a user-provided `ollama serve` instance with no + bundled model weights and no torch/transformers pulled in by the `llm` extra. +- [x] Provenance metadata for local-LLM pipeline runs records backend, base URL, and model identity. +- [ ] `person_tracking` output includes frame-to-frame movement-delta fields via a pandas-based + post-processing step, additive-only (no v1.5.x output field ever removed or renamed). +- [ ] A user can see every downloaded model's size, version and users, and pull or delete it, from + the CLI and the viewer without touching the filesystem. + +--- + +**Last Updated**: 2026-10-17 (renumbered from v1.7.0; before that from v1.6.0; content last revised 2026-07-08) +**Target Release**: After the v1.7.0 public release +**Status**: Planning Phase — Plugin Ecosystem Focus diff --git a/docs/development/roadmap_v1.7_to_v2.0.md b/docs/development/roadmap_v1.8_to_v2.0.md similarity index 89% rename from docs/development/roadmap_v1.7_to_v2.0.md rename to docs/development/roadmap_v1.8_to_v2.0.md index 8ad1920b..4879226e 100644 --- a/docs/development/roadmap_v1.7_to_v2.0.md +++ b/docs/development/roadmap_v1.8_to_v2.0.md @@ -1,27 +1,32 @@ -# VideoAnnotator v1.7 — v2.0 Roadmap +# VideoAnnotator v1.8 — v2.0 Roadmap ## Theme: Remote Pipelines, HPC Dispatch, Multimodal VLM, Slim Core > **Renumbered 2026-09-24.** v1.6.0 became the public release ([`roadmap_v1.6.0.md`](roadmap_v1.6.0.md)), -> so the plugin-ecosystem plan that was v1.6.0 is now v1.7.0 ([`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)), +> so the plugin-ecosystem plan that was v1.6.0 is now v1.7.0 ([`roadmap_v1.8.0.md`](roadmap_v1.8.0.md), since renumbered), > and everything below moved one version later: the old v1.7 is now v1.8, the old v1.8 is now v1.9. > Target dates predate the renumbering. Items pulled forward into v1.6.0 are marked. +> +> **Renumbered again 2026-10-17.** The public release moved to v1.7.0 (the viewer overhaul first; +> [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md)), so the plugin ecosystem is now v1.8.0 +> ([`roadmap_v1.8.0.md`](roadmap_v1.8.0.md)) and everything below moved one more version: remote +> pipelines v1.9, HPC dispatch v1.10. Mentions of v1.6.0 as done are history and stay. -This document covers the arc from v1.7 through v2.0 — the period when VideoAnnotator stops being a single-machine monolith and becomes a toolkit that can dispatch work to GPU servers, HPC clusters, and remote multimodal models. +This document covers the arc from v1.8 through v2.0 — the period when VideoAnnotator stops being a single-machine monolith and becomes a toolkit that can dispatch work to GPU servers, HPC clusters, and remote multimodal models. -The detail here is intentionally lighter than [v1.5](roadmap_v1.5.0.md), [v1.6](roadmap_v1.6.0.md) and [v1.7](roadmap_v1.7.0.md). These versions are 6-12+ months out; specifics will firm up as those land and field experience informs the design. +The detail here is intentionally lighter than [v1.5](roadmap_v1.5.0.md), [v1.6](roadmap_v1.6.0.md), [v1.7](roadmap_v1.7.0.md) and [v1.8](roadmap_v1.8.0.md). These versions are 6-12+ months out; specifics will firm up as those land and field experience informs the design. -**Prerequisite:** v1.5.0's extras-based install and metadata-driven registry loading (see [`specs/004-extras-based-install/spec.md`](../../specs/004-extras-based-install/spec.md)), v1.6.0's OpenAI-compatible connector, plus v1.7.0's entry-point plugin discovery and `Dispatcher` ABC seam (the local-LLM backend already establishes the `backends` metadata field's non-local-execution pattern this document extends to remote/HPC targets). +**Prerequisite:** v1.5.0's extras-based install and metadata-driven registry loading (see [`specs/004-extras-based-install/spec.md`](../../specs/004-extras-based-install/spec.md)), v1.7.0's OpenAI-compatible connector, plus v1.8.0's entry-point plugin discovery and `Dispatcher` ABC seam (the local-LLM backend already establishes the `backends` metadata field's non-local-execution pattern this document extends to remote/HPC targets). --- -## v1.7 — Plugin ecosystem +## v1.8 — Plugin ecosystem -Planned in full in [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md): entry-point pipeline discovery, the `Dispatcher` ABC with `LocalThreadDispatcher`, the `videoannotator-utils` package, a reference third-party plugin, and pandas-based movement deltas for `person_tracking`. +Planned in full in [`roadmap_v1.8.0.md`](roadmap_v1.8.0.md): entry-point pipeline discovery, the `Dispatcher` ABC with `LocalThreadDispatcher`, the `videoannotator-utils` package, a reference third-party plugin, and pandas-based movement deltas for `person_tracking`. --- -## v1.8 — Remote pipelines and the first VLM plugin +## v1.9 — Remote pipelines and the first VLM plugin **Theme:** Pipelines no longer have to run in-process. Adds the proxy layer that v2.0 needs and ships the first plugin that uses it. @@ -32,7 +37,7 @@ Planned in full in [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md): entry-point pipelin - [ ] **`RemotePipelineProxy(BasePipeline)`** in core. Reads `endpoint`, `model`, `auth`, sampling params from config. In `process()`, serialises args as JSON, POSTs to the endpoint, parses the response back to `list[dict]`. The pipeline class is locally installable and locally instantiable; only the heavy compute lives elsewhere. - [ ] **`HTTPDispatcher`** as a third `Dispatcher` implementation alongside `LocalThreadDispatcher`. Submits jobs to a remote VideoAnnotator instance over the existing FastAPI v1. Useful for "I have a GPU server in the next room and a laptop in front of me" workflows. -- [ ] **Transport security and credential handling for `HTTPDispatcher`/`RemotePipelineProxy`.** TLS-only endpoints by default (reject plaintext `http://` unless explicitly opted into for local-network use), and a defined credential-storage model for the `auth` config field (e.g. environment variable or OS keychain reference, never a literal secret in a checked-in config file). This was flagged as an open gap in earlier drafts of this roadmap and must be a first-class requirement of the v1.8.0 spec, not an afterthought — see [`specs/004-extras-based-install/spec.md`](../../specs/004-extras-based-install/spec.md) Assumptions section. +- [ ] **Transport security and credential handling for `HTTPDispatcher`/`RemotePipelineProxy`.** TLS-only endpoints by default (reject plaintext `http://` unless explicitly opted into for local-network use), and a defined credential-storage model for the `auth` config field (e.g. environment variable or OS keychain reference, never a literal secret in a checked-in config file). This was flagged as an open gap in earlier drafts of this roadmap and must be a first-class requirement of the v1.9.0 spec, not an afterthought — see [`specs/004-extras-based-install/spec.md`](../../specs/004-extras-based-install/spec.md) Assumptions section. - [ ] **Reference pipeline-runner microserver.** A 30-50 line FastAPI app at `packages/videoannotator-pipeline-runner/` that exposes a single `BasePipeline` instance over HTTP using the same JSON contract as `RemotePipelineProxy`. Production-deployable as a Docker image. - [ ] **`videoannotator-vlm` plugin (first VLM pipeline).** New sibling package. Default backend: Qwen2.5-VL-7B via Ollama (`qwen2.5vl:7b`). Optional backend: vLLM for batch throughput. Pipeline config: ```yaml @@ -58,10 +63,10 @@ Planned in full in [`roadmap_v1.7.0.md`](roadmap_v1.7.0.md): entry-point pipelin Note (2026-09-24): the VLM pipeline itself already exists in core as `vlm_annotation` (v1.5.0). What remains here is packaging it as a plugin and the remote proxy. -### Out of scope for v1.8 +### Out of scope for v1.9 -- HPC dispatcher (deferred to v1.9 unless someone needs it sooner). -- New specialist pipelines (pose, hand, motion) — deferred to v1.9. (v1.6.0 may add the voice type classifier, gaze target or SAM 3 tracking if pilot labs ask.) +- HPC dispatcher (deferred to v1.10 unless someone needs it sooner). +- New specialist pipelines (pose, hand, motion) — deferred to v1.10. (v1.7.0 may add the voice type classifier, gaze target or SAM 3 tracking if pilot labs ask.) - Splitting face/scene/person plugins — that's v2.0. - ~~**Auto-restart after in-app extras install.**~~ **Done in v1.5.0** (spec 011: installs activate without a restart when they can, and the server restarts from the viewer when they can't). Original note: [`specs/005-pipeline-extras-install/spec.md`](../../specs/005-pipeline-extras-install/spec.md) (self-service `pip install videoannotator[]` triggered from the API/viewer) ships with a @@ -72,7 +77,7 @@ Note (2026-09-24): the VLM pipeline itself already exists in core as `vlm_annota --- -## v1.9 — HPC dispatch and new pipeline categories +## v1.10 — HPC dispatch and new pipeline categories **Theme:** Lab-scale to cluster-scale. Adds Slurm and the pipelines researchers have been asking for that aren't VLMs. @@ -87,7 +92,7 @@ Note (2026-09-24): the VLM pipeline itself already exists in core as `vlm_annota - [ ] **`videoannotator-hand` plugin.** MediaPipe Hand Landmarker. 21 keypoints/hand on CPU, trivial CC-licensed dependency. Useful for caregiver-infant gesture coding. Source: [github.com/google-ai-edge/mediapipe](https://github.com/google-ai-edge/mediapipe). - [ ] **`videoannotator-motion` plugin (optional).** SEA-RAFT optical flow as an opt-in for "how much movement happened in this clip" features. BSD-3. Source: [github.com/princeton-vl/SEA-RAFT](https://github.com/princeton-vl/SEA-RAFT). -### Out of scope for v1.9 +### Out of scope for v1.10 - Distributed multi-node within a single job (still single-machine per pipeline call). - Audio events (BEATs) — defer unless requested. @@ -122,7 +127,7 @@ Note (2026-09-24): the VLM pipeline itself already exists in core as `vlm_annota --- -## Cross-cutting concerns for v1.7-v2.0 +## Cross-cutting concerns for v1.8-v2.0 ### Reproducibility @@ -158,7 +163,7 @@ The Ultralytics AGPL-3.0 issue is the only real licensing wrinkle. Ship `videoan ### VLM survey summary -[Full survey on file with maintainer.] Headline recommendation for v1.7 (now v1.8): **Qwen2.5-VL-7B-Instruct via Ollama as default, vLLM as power-user backend**. Apache-2.0, native video, 32K context, fits 24 GB VRAM, weak refusal behaviour around children. Runner-up: **InternVL3.5-8B via lmdeploy** (different vision encoder, useful as a second rater for reproducibility cross-checks). Future upgrade: Qwen3-VL-8B (256K native context, October 2025 weights) once Ollama publishes a first-party tag. +[Full survey on file with maintainer.] Headline recommendation for v1.7 (now v1.9): **Qwen2.5-VL-7B-Instruct via Ollama as default, vLLM as power-user backend**. Apache-2.0, native video, 32K context, fits 24 GB VRAM, weak refusal behaviour around children. Runner-up: **InternVL3.5-8B via lmdeploy** (different vision encoder, useful as a second rater for reproducibility cross-checks). Future upgrade: Qwen3-VL-8B (256K native context, October 2025 weights) once Ollama publishes a first-party tag. Pipelines a VLM can **replace**: scene labelling, coarse action description, video-segment summaries, person/object enumeration, simple emotion description. Pipelines a VLM **augments but doesn't replace**: face emotion (use as independent rater), behaviour description on top of person tracks. Pipelines a VLM **never replaces**: pose keypoints, AUs, diarisation timestamps, frame-accurate scene cuts. @@ -166,7 +171,7 @@ Pipelines a VLM can **replace**: scene labelling, coarse action description, vid ## Risks and open questions -- **Ollama video support.** Ollama currently has no video parameter on `/api/generate`; video is sent as multi-image at extracted frames. If Ollama adds native video before v1.8 lands, the plugin can use it; otherwise the multi-image pattern is fine. (Still no video input for Qwen3-VL in Ollama as of 2026-09; v1.6.0 gets real video input through vLLM instead.) +- **Ollama video support.** Ollama currently has no video parameter on `/api/generate`; video is sent as multi-image at extracted frames. If Ollama adds native video before v1.9 lands, the plugin can use it; otherwise the multi-image pattern is fine. (Still no video input for Qwen3-VL in Ollama as of 2026-09; v1.7.0 plans real video input through vLLM instead.) - **Cluster heterogeneity.** Slurm clusters vary wildly (partitions, GRES syntax, container runtimes). The `SlurmDispatcher` ships a sensible default but expect lab-specific config files in practice. - **Plugin release coupling.** Once we have 7+ sibling packages, version-resolution between core and plugins becomes a real maintenance task. Mitigation: lock-step semver-major releases of core + all official plugins until v3.0 if needed. - **Long-tail of specialist models.** Field will keep moving; v2.0+ should include a clear cadence ("review specialist defaults each minor release") rather than letting them drift. diff --git a/docs/installation/ENVIRONMENT_SETUP.md b/docs/installation/ENVIRONMENT_SETUP.md index f137a942..7b260f9e 100644 --- a/docs/installation/ENVIRONMENT_SETUP.md +++ b/docs/installation/ENVIRONMENT_SETUP.md @@ -8,11 +8,39 @@ The VideoAnnotator project uses PyAnnote for speaker diarization, which requires 1. Go to [HuggingFace Settings](https://huggingface.co/settings/tokens) 2. Create a new token with "Read" permissions -3. Accept the terms for [pyannote/speaker-diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) +3. Signed in as the same account, accept the licence on each model page the viewer lists on the + Speaker turns card (since v1.6.0 just one, `pyannote/speaker-diarization-community-1`) -### 2. Configure Your Environment +### 2. Give the token to VideoAnnotator -#### Option A: Using .env file (Recommended) +No file to edit and no restart, whichever way you choose. The token stays on the server's machine. + +#### Option A: In the viewer (recommended) + +On the **Pipelines** page (or when choosing pipelines for a run), the Speaker turns card shows the +three steps above. Paste the token into step 3 and press **Save and check**: the card names the +token's account and shows each licence as accepted or not. You need an admin key (the first key a +new install creates is one). To see, change or remove the token later: **Settings → API +Configuration → Hugging Face token**. + +#### Option B: From a terminal + +```bash +uv run videoannotator settings huggingface-token # asks for it, without showing it +uv run videoannotator settings huggingface-token --status # set? which account? +uv run videoannotator settings huggingface-token --remove +``` + +With no terminal (a script or an agent), it reads the token from standard input: +`printf '%s\n' "$TOKEN" | uv run videoannotator settings huggingface-token`. + +Options A and B save the token in a file next to the database (`huggingface_token`, readable only +by you), so in the container it survives being recreated. + +#### Option C: An environment variable + +A token in the server's environment takes priority over a saved one, and changing it needs a +restart. Use this for setups that already manage their environment. 1. Copy the example environment file: @@ -26,9 +54,7 @@ The VideoAnnotator project uses PyAnnote for speaker diarization, which requires HUGGINGFACE_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` -#### Option B: Using System Environment Variables - -Set the environment variable in your shell: +Or set it in your shell: **Windows (PowerShell):** @@ -85,11 +111,11 @@ If configured correctly, you should see: **"Authentication failed"** - Verify your token is valid at [HuggingFace Settings](https://huggingface.co/settings/tokens) -- Make sure you've accepted the terms for pyannote/speaker-diarization-3.1 +- Make sure you've accepted the terms for pyannote/speaker-diarization-community-1 - Try regenerating your token if it's old **"Model access denied"** -- Go to [pyannote/speaker-diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) +- Go to [pyannote/speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) - Click "Agree and access repository" - Wait a few minutes for permissions to propagate diff --git a/docs/installation/INSTALLATION.md b/docs/installation/INSTALLATION.md index d011d8ad..823e3681 100644 --- a/docs/installation/INSTALLATION.md +++ b/docs/installation/INSTALLATION.md @@ -178,7 +178,7 @@ These tips address the most common macOS installation issues. chmod u+w ~/.zshrc # If .zshrc is missing, write PATH to ~/.zprofile instead for login shells ``` -- Hugging Face token (for speaker diarization): set `HUGGINGFACE_TOKEN` in a `.env` file or your shell. See Environment Setup guide. +- Hugging Face token (for speaker diarization): paste it on the Speaker turns card in the viewer, or run `uv run videoannotator settings huggingface-token`; no restart needed. Environment variables still work. See the Environment Setup guide. - Viewer: available at `/viewer` on the running VideoAnnotator server, no separate setup. For standalone development of the viewer repo itself, use `npm run dev` (not `npm start`). @@ -261,9 +261,9 @@ install that matches what you're doing: | I want to... | Install command | What you get | | -------------------------------------- | ----------------------------------------------- | ------------ | | Run only scene labelling | `uv sync --extra scene` | torch, open-clip-torch, PySceneDetect — no face/audio/person deps | -| Run only person tracking | `uv sync --extra person` | torch, ultralytics (YOLO11), supervision | +| Run only person tracking | `uv sync --extra person` | torch, ultralytics (YOLO26) | | Run only face analysis (DeepFace) | `uv sync --extra face` | deepface, imutils — **no torch** (the one torch-free pipeline family) | -| Run only speech/diarization | `uv sync --extra audio` | torch, torchaudio, librosa, openai-whisper, pyannote.audio | +| Run only speech/diarization | `uv sync --extra audio` | torch, torchaudio, librosa, faster-whisper, pyannote.audio | | Mix a few families | `uv sync --extra scene --extra person` | union of the groups listed | | Run a slim API server (no local pipelines) | `uv sync` | core only — useful if pipelines run on separate workers/nodes | | Reproduce the old "everything installed" behaviour | `uv sync --all-extras` (or `uv sync --extra all`) | every pipeline family | diff --git a/docs/pipelines_spec.md b/docs/pipelines_spec.md index 6544373c..b58630aa 100644 --- a/docs/pipelines_spec.md +++ b/docs/pipelines_spec.md @@ -1,17 +1,130 @@ # Pipeline Specifications -Generated: 2026-10-01T13:31:13.669333Z +Generated: 2026-10-18T13:55:20.787330Z This file is auto-generated. Do not edit by hand. | Name | Display Name | Family | Variant | Tasks | Modalities | Capabilities | Backends | Stability | Outputs | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | face_analysis | Face Analysis (DeepFace) | face | deepface | face-detection,emotion-recognition,age-estimation,gender-prediction | video | person-linking,frame-level-analysis | tensorflow,opencv | stable | COCO:face_detections/emotions/demographics | -| face_openface3_embedding | OpenFace3 Face Embedding | face | openface3-embedding | face-embedding | image,video | embedding | onnx,pytorch | experimental | JSON:embeddings | -| person_tracking | Person Tracking & Pose | person | yolov11n-pose-bytetrack | object-tracking,pose-estimation | video | real-time,identity-persistence | pytorch | beta | COCO:person_detection/keypoints/tracking | +| face_openface3_embedding | OpenFace3 Face Embedding | face | openface3-embedding | face-embedding | image,video | embedding | onnx,pytorch | experimental | JSON:embeddings;JSON:face_features | +| person_tracking | Person Tracking & Pose | person | yolo26n-pose-bytetrack | object-tracking,pose-estimation | video | real-time,identity-persistence | pytorch | beta | COCO:person_detection/keypoints/tracking;JSON:person_tracks | | scene_detection | Scene Detection | scene | pyscenedetect-clip | scene-detection,scene-segmentation | video | batch,embedding | pytorch | beta | JSON:scene_boundary/scene_category | | speaker_diarization | Speaker Diarization | audio | pyannote | speaker-diarization,speaker-segmentation | audio | timeline,speaker-turns | pytorch | stable | RTTM:speaker_turns | -| speech_recognition | Speech Recognition | audio | whisper | speech-transcription,automatic-speech-recognition | audio | streaming,word-timestamps,multilingual | pytorch | stable | WebVTT:transcript | +| speech_recognition | Speech Recognition | audio | faster-whisper | speech-transcription,automatic-speech-recognition | audio | streaming,word-timestamps,multilingual | ctranslate2 | stable | WebVTT:transcript | | vlm_annotation | VLM Frame Annotation | vlm | ollama | frame-classification,vlm-captioning | video | batch | ollama | experimental | JSON:vlm_label/vlm_reasoning | Total pipelines: 7 + +## Pipeline guides + +### Faces: where they are and their expression (`face_analysis`) + +Finds faces in each sampled frame and estimates each face's expression. Can also estimate age and gender, which are off by default. + +What to measure: Smallest face (`min_face_size`, default `30` px), Most faces per frame (`max_faces`, default `10` faces), Expression (`detect_emotions`, default `True`), Age (`detect_age`, default `False`), Gender (`detect_gender`, default `False`). + +Homepage: + +- WIDER FACE (hard): AP 91.4% (RetinaFace, the face detector used here). [Source](https://arxiv.org/abs/1905.00641) +- DeepFace age model: MAE ± 4.65 (test data not named in the README). [Source](https://github.com/serengil/deepface) +- DeepFace gender model: accuracy 97.44% (test data not named in the README). [Source](https://github.com/serengil/deepface) + +None of these figures is for infants or young children. DeepFace's expression model was trained on adult faces, and DeepFace publishes no accuracy figure for it. + +Cite: `serengil2026deepface`, `deng2020retinaface`. + +### Faces: action units, gaze, head pose (`face_openface3_embedding`) + +Finds faces in each sampled frame and measures facial action units, gaze direction, head pose and expression. Also describes each face in a way that tells people apart. + +What to measure: Smallest face (`min_face_size`, default `32` px). + +Homepage: + +- DISFA: action unit F1 60 (OpenFace 3.0 (MTL) row, Table III). [Source](https://arxiv.org/abs/2506.02891) +- MPII: gaze, difference in angle 4.25 (OpenFace 3.0 (MTL) row, Table III). [Source](https://arxiv.org/abs/2506.02891) +- AffectNet: 8-emotion accuracy 0.56 (OpenFace 3.0 (MTL) row, Table III). [Source](https://arxiv.org/abs/2506.02891) +- 300W: landmark NME (inter-ocular) 2.87 (OpenFace 3.0 (MTL) row, Table III). [Source](https://arxiv.org/abs/2506.02891) + +The paper reports a multitask model (these figures) and single-task ones; it does not say which the released package uses. Installed from `openface-test`, the package CMU's repository names; neither has changed since June 2025. Its licence allows non-commercial research use only. + +Cite: `hu2025openface3`. + +### People: where they are and how they move (`person_tracking`) + +Finds each person in each sampled frame, follows them through the video, and marks 17 body points (nose, shoulders, wrists, knees and so on). + +What to measure: Follow each person through the video (`track_mode`, default `True`). + +Homepage: + +- COCO Keypoints val2017: mAP pose 50-95 (e2e) 57.2 (yolo26n-pose at 640 px, end-to-end (no NMS); yolo11n-pose, the previous default, 50.0). [Source](https://docs.ultralytics.com/models/yolo26/) + +Cite: `jocher2026yolo26`, `zhang2022bytetrack`. + +### Scenes: cuts and settings (`scene_detection`) + +Finds where the video cuts from one shot to another and labels the kind of setting each scene shows. + +What to measure: Shortest scene (`min_scene_length`, default `1.0` s). + +Homepage: + +No published benchmark. + +Neither PySceneDetect nor CLIP publishes an accuracy figure for finding cuts or labelling scene settings. + +Cite: `castellano_pyscenedetect`, `radford2021clip`, `ilharco2021openclip`, `schuhmann2022laion`. + +### Speaker turns: who spoke when (`speaker_diarization`) + +Splits the audio into turns and says which speaker each turn belongs to, without naming them. + +What to measure: Fewest speakers (`min_speakers`, default `1` speakers), Most speakers (`max_speakers`, default `10` speakers). + +Homepage: + +- AMI (IHM): DER % 17.0 (community-1; benchmark last updated 2025-09). [Source](https://github.com/pyannote/pyannote-audio#benchmark) +- DIHARD 3 (full): DER % 20.2 (community-1; benchmark last updated 2025-09). [Source](https://github.com/pyannote/pyannote-audio#benchmark) +- VoxConverse (v0.3): DER % 11.2 (community-1; benchmark last updated 2025-09). [Source](https://github.com/pyannote/pyannote-audio#benchmark) + +These sets are meetings, broadcasts and phone calls between adults; none measures parent–infant interaction or infant vocalisations. + +Cite: `plaquet2023powerset`, `pyannote2025community1`, `bredin2023pyannote`. + +### Speech: what was said (`speech_recognition`) + +Transcribes speech into timed text, with the time of each word. + +What to measure: Language (`language`, default `None`), Transcribe or translate (`task`, default `transcribe`), Time of each word (`word_timestamps`, default `True`), Shortest segment (`min_segment_duration`, default `1.0` s). + +Homepage: + +- Transcribing 13 minutes of audio (GPU, large-v2, beam 5): time 1m03s (openai-whisper 2m23s) (faster-whisper fp16 against openai-whisper fp16, on an RTX 3070 Ti). [Source](https://github.com/SYSTRAN/faster-whisper#benchmark) + +OpenAI publishes large-v3-turbo's accuracy only as charts: "Across languages, the turbo model performs similarly to large-v2", with larger drops for some languages such as Thai and Cantonese (github.com/openai/whisper/discussions/2363). No figure measures child-directed speech, infant vocalisations or overlapping talk. On a CPU, a smaller model (whisper_model small) is much faster. + +Cite: `radford2023whisper`. + +### Ask AI: your question about each moment (`vlm_annotation`) + +Asks a vision-language model your question about frames sampled every few seconds, and records its answer and reasoning for each one. + +What to measure: Question (`prompt`, default `Classify as TOUCH only if: +The mother's hand or arm is actually in physical contact with the infant's body. + +Classify as NO_TOUCH if: +The mother's hand or arm visually overlaps the infant in the image, but one is clearly in front of or behind the other (the apparent contact is camera perspective, not real contact). +The mother's hand or arm is near but not actually touching the infant. + +Return ONLY: TOUCH or NO_TOUCH. +`), Frames per sample (`sampling_mode`, default `single_frame`), Sample every (`frame_interval_sec`, default `5.0` s). + +Homepage: + +No published benchmark. + +Depends on the model you choose and on your question, so no figure applies to this pipeline as such. See your model's page, for example https://ollama.com/library/qwen3.5. + +Cite: `ollama`. diff --git a/docs/usage/GETTING_STARTED.md b/docs/usage/GETTING_STARTED.md index a6cdd85e..a9c00011 100644 --- a/docs/usage/GETTING_STARTED.md +++ b/docs/usage/GETTING_STARTED.md @@ -132,7 +132,8 @@ server, and prints the viewer login link once the server is up. It is safe to re options (port, extras to install, non-interactive use). To shorten the commands, add `source /path/to/VideoAnnotator/scripts/shell_aliases.sh` to your `~/.bashrc` or `~/.zshrc`: then `va-start` runs the script and `va` runs `uv run videoannotator`, from any folder. The dev -container sets these up for you. +container sets these up for you, and also puts `va` and `videoannotator` on the PATH as commands, +so they work in scripts and non-interactive shells too. **Viewer**: `/viewer` serves a bundled build of Video Annotation Viewer, pre-configured to talk to this server (same-origin, no setup needed). It also runs on its own — see @@ -257,7 +258,7 @@ scene_detection: enabled: true person_tracking: - model: "yolo11n-pose.pt" + model: "yolo26n-pose.pt" conf_threshold: 0.4 iou_threshold: 0.7 track_mode: true diff --git a/docs/usage/README.md b/docs/usage/README.md index 96586cff..6c81bdcd 100644 --- a/docs/usage/README.md +++ b/docs/usage/README.md @@ -8,6 +8,7 @@ Welcome to the VideoAnnotator usage documentation. - **[Configuration](configuration.md)**: Detailed configuration guide (YAML & Environment Variables). - **[Environment Variables](environment_variables.md)**: Reference for all environment variables. - **[Accessing Results](accessing_results.md)**: How to download and view job results. +- **[Results Out](exporting_results.md)**: One table of every pipeline's results for R or Python, ELAN files, and a methods paragraph. - **[Demo Commands](demo_commands.md)**: Copy-pasteable commands for common tasks. - **[Troubleshooting](troubleshooting.md)**: Common issues and solutions. diff --git a/docs/usage/accessing_results.md b/docs/usage/accessing_results.md index ce4bb7d1..7473a6e7 100644 --- a/docs/usage/accessing_results.md +++ b/docs/usage/accessing_results.md @@ -2,6 +2,9 @@ This page describes where VideoAnnotator stores outputs and how to retrieve them using the CLI or HTTP API. +> For analysis, export everything as one table (CSV or Parquet), as ELAN files, or as a methods +> paragraph: see [Results out](exporting_results.md). + ## Where results are stored ### Docker Compose (recommended) diff --git a/docs/usage/exporting_results.md b/docs/usage/exporting_results.md new file mode 100644 index 00000000..32843707 --- /dev/null +++ b/docs/usage/exporting_results.md @@ -0,0 +1,196 @@ +# Getting results out: one table, ELAN, a methods paragraph + +Each pipeline writes its own standard format (COCO JSON, WebVTT, RTTM). Those files stay as they +are. For analysis, VideoAnnotator also turns them into: + +- **One table** of every pipeline's results, the same columns for every pipeline, as CSV (or + Parquet). Read it with one `read.csv` in R or `pandas.read_csv` in Python. +- **ELAN files** (`.eaf`), one per video, with one tier per pipeline track, to check or correct the + results next to your own coding. +- **A methods paragraph** with BibTeX, naming every model and its exact weights, the settings that + differ from the defaults, and what to cite. + +All three are built from the result files and the provenance recorded in them, so they work on any +results folder: one on your computer, one a colleague sent you, or one from an older version +(where something wasn't recorded, they say `not recorded`; they never guess). + +## In the viewer + +On a finished job's page or a run's page, click **Export**: + +- **Table (CSV)** / **Table (Parquet)**: downloads the table. +- **ELAN files (.eaf)**: one file for a job; a zip with one file per video for a run. +- **Methods paragraph**: shows the paragraph and BibTeX, each with a Copy button, and a Markdown + download. + +A saved dataset's card has a table button too: every run made from that dataset in one table, with +a `run` column telling them apart. + +If anything was left out (a file VideoAnnotator can't read, a video whose job failed), the viewer +lists it under the buttons. + +## From a terminal + +These need no server and no network: point them at any results folder (one video's folder, a run's +folder, a folder of runs, or an extracted results download). + +```bash +videoannotator export table "~/VideoAnnotator/BabyJokes wave 2 (2026-10-06)" +videoannotator export elan "~/VideoAnnotator/BabyJokes wave 2 (2026-10-06)" +videoannotator export methods "~/VideoAnnotator/BabyJokes wave 2 (2026-10-06)" -o methods.md +``` + +- `export table` writes `/.csv` unless you give `-o PATH` (`-o -` prints it). + `--format parquet` writes Parquet. +- `export elan` writes `