diff --git a/.claude/commands/run-generator.md b/.claude/commands/run-generator.md index c96b9972..138f92ff 100644 --- a/.claude/commands/run-generator.md +++ b/.claude/commands/run-generator.md @@ -1,4 +1,4 @@ -Help the user run the conversation generator (generate.py): +Help the user run the conversation generator (`uv run python vera.py generate`; flags in docs/cli.md): 1. Ask the user for parameters: - Model to use (default: claude-3-7-sonnet) diff --git a/.claude/commands/run-judge.md b/.claude/commands/run-judge.md index 95ba68b2..450c1b38 100644 --- a/.claude/commands/run-judge.md +++ b/.claude/commands/run-judge.md @@ -1,4 +1,4 @@ -Help the user run the conversation evaluator (judge.py): +Help the user run the conversation evaluator (`uv run python vera.py judge`; flags in docs/cli.md): 1. Ask the user for parameters: - Input directory (default: output/conversations) diff --git a/.claude/commands/verify.md b/.claude/commands/verify.md index 998863bc..f29c8cca 100644 --- a/.claude/commands/verify.md +++ b/.claude/commands/verify.md @@ -1,6 +1,6 @@ Verify that a code change works correctly without hitting live APIs. -IMPORTANT: Never run `generate.py`, `judge.py`, or `run_pipeline.py` directly — these make live LLM API calls. Verification must use the test suite only. +IMPORTANT: Never run the `vera` commands or the scripts in `legacy/` directly — these make live LLM API calls. Verification must use the test suite only. Steps: 1. Identify what changed (git diff --stat) to understand the scope diff --git a/.claude/settings.json b/.claude/settings.json index 45c78a2f..9e09a161 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -10,8 +10,8 @@ "Bash(.venv/bin/ruff check --fix .)", "Bash(.venv/bin/pyright:*)", "Bash(.venv/bin/pytest:*)", - "Bash(python3 generate.py:*)", - "Bash(python3 judge.py:*)", + "Bash(python3 -m legacy.generate:*)", + "Bash(python3 -m legacy.judge:*)", "Bash(python3 -m pytest:*)", "Bash(pytest:*)", "Bash(git status)", diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8815575b..accdb1fe 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -4,7 +4,7 @@ /llm_clients/ @SpringCare/vera-mh-maintainers /judge/ @SpringCare/vera-mh-maintainers /generate/ @SpringCare/vera-mh-maintainers -/run_pipeline.py @SpringCare/vera-mh-maintainers +/legacy/ @SpringCare/vera-mh-maintainers /openspec/ @SpringCare/vera-mh-maintainers /pyproject.toml @SpringCare/vera-mh-maintainers diff --git a/AGENTS.md b/AGENTS.md index b3579db4..1e937adf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -134,7 +134,7 @@ uv run pyright pre-commit run --all-files ``` -`generate.py`, `judge.py`, `run_pipeline.py`, `judge/score.py`, and `scripts/run_recommended_vera_pipeline.sh` are deprecated and will be removed; don't build on them. Their flags and `vera` equivalents are in [docs/legacy-scripts.md](docs/legacy-scripts.md). +The scripts in [`legacy/`](legacy/) (`generate.py`, `judge.py`, `score.py`, `run_pipeline.py`, `run_recommended_vera_pipeline.sh`) are deprecated and may be removed at any time; don't build on them, and never import them from `vera_cli/` or the domain packages. Their flags and `vera` equivalents are in [docs/legacy-scripts.md](docs/legacy-scripts.md). Use dated model IDs (e.g. `claude-sonnet-4-5-20250929`) as in README; shorthand aliases may not resolve. diff --git a/CHANGELOG.md b/CHANGELOG.md index 41dec0a4..c6beae17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,9 +13,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - **Faster tests**—Tests no longer wait out real retry backoff, so the suite drops from ~95s to ~10s. Each test has a 30s timeout. - **CI split**—PRs run unit tests; merges to `main` also run integration tests, including a mocked end-to-end `vera pipeline` run. The Docker build runs only when its inputs change. +## [v2.0.2] \- Unreleased + +### Breaking / migration + +- **Legacy scripts moved to [`legacy/`](legacy/)**—Run them from the repository root as modules: `uv run python -m legacy.generate`, `-m legacy.judge`, `-m legacy.score` (was `python -m judge.score`), and `-m legacy.run_pipeline`. They may be removed at any time; [`docs/legacy-scripts.md`](docs/legacy-scripts.md) maps their flags to `vera`. + ## [v2.0.0](https://github.com/SpringCare/VERA-MH/releases/tag/v2.0.0) \- 2026-09-25 -VERA-MH 2.0 is released. It breaks the legacy scripts: they still run in 2.0.0, but 2.0.1 moves them to `legacy/`, which changes how they are invoked. +VERA-MH 2.0 is released. It breaks the legacy scripts: they still run in 2.0.0, but 2.0.2 moves them to `legacy/`, which changes how they are invoked. ### Breaking / migration diff --git a/Dockerfile b/Dockerfile index fd4c91ab..264b4c35 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,4 +11,4 @@ RUN uv sync --frozen COPY . . -CMD ["python", "generate.py"] \ No newline at end of file +CMD ["python", "vera.py", "--help"] \ No newline at end of file diff --git a/README.md b/README.md index ca0cce54..9290f51a 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ pre-commit install # optional: format and lint on commit ## Legacy scripts -The pre-2.0 entry points—`generate.py`, `judge.py`, `run_pipeline.py`, `judge/score.py`, and `scripts/run_recommended_vera_pipeline.sh`—still work, but they are **deprecated and will be removed soon**. Their flags differ from the `vera` CLI (for example `-c` means `--max-concurrent` in `generate.py`). Use `vera` for new work; if you still depend on a script, see [docs/legacy-scripts.md](docs/legacy-scripts.md) for its options and the `vera` equivalent. +The pre-2.0 entry points live in [`legacy/`](legacy/) and run as modules, for example `uv run python -m legacy.generate`. They still work, but they are **deprecated and may be removed at any time**. Their flags differ from the `vera` CLI (for example `-c` means `--max-concurrent` in `legacy/generate.py`). Use `vera` for new work; if you still depend on a script, see [docs/legacy-scripts.md](docs/legacy-scripts.md) for its options and the `vera` equivalent. ## License diff --git a/docs/architecture.md b/docs/architecture.md index f97bb14e..79ed2969 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -123,14 +123,12 @@ namespaces, input config files, or target manifests. They define no CLI behavior defaults. Pooling likewise delegates to the function owned by the scoring domain rather than to a script entry point. -Legacy root scripts may remain temporarily while their replacement feature is -migrated. During that transition, `vera_cli` may import the reusable function -from a root script, but never its argument parser. For generation, the temporary -flow is `vera_cli.generate` → `generate.run_for_user_models` → `generate.main`. -Removing `generate.py` and moving those functions plus the existing -`generate_conversations/` code into the permanent `generate/` package is one -atomic later change, so a root `generate.py` module and a top-level `generate/` -package never coexist. +Legacy scripts live in `legacy/` until they are removed, and nothing outside +`legacy/` and its tests imports them. For generation, the flow is +`vera_cli.generate` → `generate_conversations.run_for_user_models` → +`generate_conversations.run_generation`. With `generate.py` out of the root, +renaming `generate_conversations/` to the permanent `generate/` package no +longer collides with a root `generate` module. `run_for_user_models` and its `_legacy_model_config` helper are explicit stopgaps. They put the expansion of a run's user models, and the flattening of @@ -141,7 +139,7 @@ models identically. | Subcommand | Delegates to | Purpose | |------------|--------------|---------| -| `vera generate` | generation application function (temporarily `generate.run_for_user_models`) | Simulate conversations → `/c_/u_*/conversations/` | +| `vera generate` | generation application function (temporarily `generate_conversations.run_for_user_models`) | Simulate conversations → `/c_/u_*/conversations/` | | `vera judge` | `judge.runner` | Evaluate transcripts → `/c_/u_*/evaluations/j_*` | | `vera score` | `score.score` | Aggregate `results.csv` → scores and visualizations | | `vera pool` | `score.pool` | Concatenate multiple evaluation folders into one pooled result → `/c_/pooled/u_+_*` | diff --git a/docs/evaluating.md b/docs/evaluating.md index 4ae2346d..3efbe068 100644 --- a/docs/evaluating.md +++ b/docs/evaluating.md @@ -242,8 +242,8 @@ Add configuration to [config.py](../llm_clients/config.py) if your LLM requires ### 5. Use the new LLM in your simulations ```bash -python3 generate.py -u your-model-name -p your-model-name -t 5 -r 1 -python3 judge.py -f conversations/{YOUR_FOLDER} -j your-model-name +uv run python vera.py generate -c your-model-name -u your-model-name -t 5 --target SI +uv run python vera.py judge --conversations output/{YOUR_RUN_FOLDER} -j your-model-name --target SI ``` ## Prompt caching (by provider) diff --git a/docs/legacy-scripts.md b/docs/legacy-scripts.md index 63de80b7..74f6c668 100644 --- a/docs/legacy-scripts.md +++ b/docs/legacy-scripts.md @@ -1,23 +1,25 @@ # Legacy scripts (deprecated) -> **These scripts will be removed soon.** Use the [`vera` CLI](cli.md) for new work. This page exists so workflows that still depend on a script keep running while they migrate. +> **These scripts may be removed at any time.** Use the [`vera` CLI](cli.md) for new work. This page exists so workflows that still depend on a script keep running while they migrate. + +The scripts live in [`legacy/`](../legacy/). Run the Python ones from the repository root as modules (`uv run python -m legacy.