[luca-belli]

Summary

With more than one user model, vera pipeline now finishes with the pooled headline score instead of printing the folders and leaving pooling to the user. This is the remaining "pipeline ends with the final score" item in the roadmap's Scoring split.

Stacked on #255 (vera pool). GitHub retargets this PR when #255 merges.

• vera_cli/pipeline.py: after scoring, if a run produced 2+ evaluations, it prints them, calls score.run_pooling, and prints Headline (pooled) score: <folder>.
  • Settings: the scoring section's personas/skip_risk_analysis; output goes under generation.output (where vera pool writes) until Traceability adds <target>/c_<chatbot>/pooled/.
  • Pools per PipelineRun, so never across targets. Errors propagate like any stage. One user model is unchanged.
• Rubric guard (score/pool.py): pooling refuses sources whose dimension columns differ (read from the data, not the SI rubric), a stand-in for the planned rubric fingerprint. vera pool reports it as a CLI error.
• Rule wording: "judge never auto-scores" now explicitly means vera judge; pipeline runs every stage. Updated in docs/architecture.md, docs/ARCHITECTURE-SPINE.md, docs/pipeline.md, and the module docstring.
• Docs: docs/cli.md (pipeline section; vera pool is now "used behind the scenes by vera pipeline", as requested in feat: add vera pool, moving pooling into score/pool.py #255), README recommended profile, docs/scoring.md, roadmap (moved to Completed), CHANGELOG.
• Roadmap: records four design issues this work surfaced:
  1. pipeline imports other commands' private functions
  2. score/ imports judge/
  3. pooling parses metadata from folder names
  4. rubric identity is only checked by dimension names

Test plan

• uv run pytest -m "not live" (1215 passed)
  • Unit: one user model doesn't pool; two pool with the expected arguments; config personas pass through; the rubric guard rejects mismatched dimensions
  • Integration (mocked LLMs): two user models end to end produce a pooled folder with results.csv, pool_metadata.json, scores/scores.json
• ruff check, ruff format --check, pyright generate score utils vera_cli vera.py
• Live pipeline e2e test (updated to the new output) not run; needs API keys

🤖 Generated with Claude Code

[luca-belli]

should we read this from the rubric?

[luca-belli]

Ideally yes, but there's no rubric to read yet. An evaluation folder doesn't record which rubric produced it: judging writes the per-conversation TSVs, answers.tsv and results.csv, and nothing names the rubric file. vera pool only receives evaluation folders, so the data is the only source today. Falling back to the SI rubric (what judge.score_utils.DIMENSIONS does) would be wrong for HFO.

Inside the pipeline we do know the rubric, but a pipeline run has exactly one, so the check can't fail there. It matters for a manual vera pool across runs.

The real fix is already on the roadmap: once a run records its config.json (Traceability) and evaluations persist a rubric fingerprint (Rubric-agnostic scoring), pooling should compare fingerprints instead of dimension names. The docstring and roadmap say this is a stand-in until then. Happy to make the note louder if you want.

[jgieringer]

thanks for putting in different PR 😄 LGTM