ci: run unit tests on PRs and integration tests on merge to main

[luca-belli]

Stacked on #234. This PR's base is that branch, so the diff shows only these changes.

Summary

• CI split: the PR test job runs -m "not integration and not live". A new integration job runs -m "integration and not live" only on push to main and on manual workflow_dispatch, so it can still be run on a branch before merging a risky change.

• Markers by directory: a pytest_collection_modifyitems hook in tests/conftest.py adds unit or integration based on the test's directory. Where a test lives now decides which job runs it. test_score_parity.py had no marker and would otherwise have run on PRs.

  Group	Tests	Time
  Unit (PRs)	980	7.1s
  Integration (main merge)	193	5.9s
  Live	12	not run

• Mocked end-to-end vera pipeline test: 1 persona, 4 turns. Every LLM is replaced with MockLLM through a single patch on LLMFactory.create_llm, and the run goes through generate, judge, and score in 0.2s. Until now, no test ran the three real stages together without live providers. The test checks:

  • the transcript has 2 persona and 2 chatbot messages;
  • results.csv came from the mock judge;
  • scores.json has a vera_score.

  It shares its config builder with the live test, so test_vera_pipeline_live.py becomes test_vera_pipeline_e2e.py.

• Coverage floor: 30% → 68%, just under the unit layer's 70%. The integration job passes --cov-fail-under=0.

• Docs: AGENTS.md, CLAUDE.md, and /test describe what runs where.

The trade-off: a change that breaks only an integration test is now caught after merge, not before. Use workflow_dispatch on the branch when a change touches pipeline wiring.

Test plan

• uv run pytest -m "not live": 1173 passed in 10s, 81% coverage
• -m "not integration and not live": 980 passed; -m "integration and not live": 193 passed
• Nothing under tests/integration/ is selected by -m "not integration"
• CI green on this PR shows no Integration Tests job

🤖 Generated with Claude Code

[luca-belli]

why not 3.13?

[luca-belli]

I copied 3.11 from the other two jobs, and it is also the lowest version we support: requires-python = ">=3.11", and Ruff targets py311. Testing on the lowest supported version catches code that only works on newer Pythons, such as except*/TaskGroup-era APIs or 3.12+ typing syntax.

You have a point, though. CI never runs the versions we actually use: local dev is on 3.13 and the Dockerfile builds on python:3.14-slim. My suggestion is a small follow-up rather than changing only this job, since the three jobs should stay on the same version. It would run the unit job as a matrix on ["3.11", "3.14"], the lowest supported version and the one we ship in Docker, and keep the integration and live jobs on a single version. The unit run takes about 10s, so the second version costs almost nothing.

[luca-belli]

Follow-up: we went with a single version instead of a matrix. #239 adds .python-version (3.14) as the one place the version is set. CI, requires-python, Ruff, and a Dockerfile check all follow it. This PR stays on 3.11 like the other jobs until #239 lands on top.

[jgieringer]

LGTM