Skip to content

feat: add vera pool, moving pooling into score/pool.py - #255

Merged
luca-belli merged 3 commits into
mainfrom
feat/vera-pool
Oct 13, 2026
Merged

luca-belli merged 3 commits into
mainfrom
feat/vera-pool

Conversation

@luca-belli

Copy link
Copy Markdown
Collaborator

Summary

Adds vera pool, the first step of the roadmap's Scoring split phase. vera judge --help, a JudgingConfig error, the docs, and vera pipeline's output already pointed users to this command.

Stacked on #251. Both PRs touch legacy/run_recommended_vera_pipeline.sh. GitHub retargets this PR to main once #251 merges.

  • vera pool (vera_cli/pool.py): --evaluations <a> <b> ... (2 or more j_* folders or their results.csv), -o (default output/), --personas, --skip-risk-analysis, --config, --print. The config section is pooling (new PoolingConfig in utils/config_schema.py). Built like vera score.
  • Logic moved from scripts/pool_vera_scores.py to score/pool.py (from score import run_pooling). The only intended behavior change is that no personas file skips the risk breakdown, as in vera score, instead of defaulting to data/SI/personas.tsv. Output layout and folder naming are unchanged until the Traceability phase.
  • Legacy: the script is now legacy/pool_vera_scores.py (python -m legacy.pool_vera_scores). Its flags, SI default, and --extract-from-log are unchanged. The recommended shell pipeline calls it as a module.
  • Docs: README, docs/cli.md (new vera pool section, removed from "Not built yet"), docs/scoring.md, docs/legacy-scripts.md, vera_cli/README.md, AGENTS.md, docs/roadmap.md, docs/ARCHITECTURE-SPINE.md (open question resolved), CHANGELOG.

Not in this PR: making vera pipeline end with the pooled score. That reverses a rule on main (test_pipeline_runs_three_stages_and_never_pools) and gets its own PR.

Behavior unchanged

I compared every moved function's parsed code with the original script. All helpers, naming, merging, and scoring code are identical. The only differences are the rename (pool_evaluation_directories → keyword-only run_pooling), imports moved to module level, and the personas default described above.

Test plan

  • uv run pytest -m "not live" (1211 passed, 30 new)
    • tests/unit/test_vera_pool.py: CLI and config handling, --print round trip, outputs, personas and skip-risk behavior
    • tests/unit/score/test_pool.py: naming and input helpers (previously untested)
    • tests/integration/test_pool_parity.py: legacy script vs vera pool give the same results.csv, scores.json, scores_by_risk.json; --extract-from-log still works
  • ruff check, ruff format --check, pyright generate score utils vera_cli vera.py (0 errors)
  • bash -n legacy/run_recommended_vera_pipeline.sh
  • Full recommended shell pipeline not run (calls live models)

🤖 Generated with Claude Code

@luca-belli
luca-belli requested a review from a team as a code owner October 9, 2026 19:10
@luca-belli
luca-belli marked this pull request as draft October 9, 2026 19:11
Comment thread docs/cli.md
Comment thread legacy/pool_vera_scores.py
Comment thread score/pool.py
Comment thread tests/unit/score/test_pool.py Outdated
Comment thread vera_cli/pool.py
@luca-belli
luca-belli added this pull request to stack #259 October 13, 2026 17:36
@luca-belli
luca-belli marked this pull request as ready for review October 13, 2026 17:40
Base automatically changed from fix/recommended-pipeline-pool-path to main October 13, 2026 20:10
luca-belli and others added 2 commits October 13, 2026 13:10
Pure move, so history follows the file. The recommended shell pipeline
now runs it as a module, like legacy.run_pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
score.run_pooling holds the logic from the pool script, unchanged except
that no personas file skips the risk breakdown, as in vera score. The
legacy script becomes a thin wrapper over it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jgieringer jgieringer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@luca-belli
luca-belli merged commit 3edce0d into main Oct 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants