ci: run live smoke tests on merge to main and weekly - #236
Merged
Merged
Conversation
luca-belli
added this pull request to stack #237
September 28, 2026 22:22
3 of 4 tasks
luca-belli
force-pushed
the
ci/live-tests-job
branch
from
September 28, 2026 23:01
a5a8cad to
a0fc65f
Compare
luca-belli
changed the title
A separate `live.yml` workflow holds the API-key secrets, so the PR workflow never sees them. Every push to main and a nightly schedule run the cheap live checks: model availability and the 4-turn `vera pipeline` smoke test. The full live suite, including the costly legacy test_scoring.py and the clinician-ratings judge check, runs only by manual dispatch with scope `all`, in line with the live-test consolidation plan in TODO. A guard step fails the job when the required secrets are unset: live tests skip themselves without keys, so the job would otherwise pass having tested nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luca-belli
force-pushed
the
ci/live-tests-job
branch
from
September 28, 2026 23:02
a0fc65f to
1847085
Compare
Weekly instead of nightly: the schedule only needs to catch a provider retiring a model in a week with no merge. Cancel an in-flight live run when a newer one of the same trigger starts. Providers bill per call, so cancelling saves the calls not yet made. Grouping by trigger keeps a merge from cancelling a manual `all` run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luca-belli
commented
Oct 1, 2026
| push: | ||
| branches: [ main ] | ||
| schedule: | ||
| # Nightly, 06:00 UTC: catches a provider retiring a model on days with no |
Collaborator
Author
There was a problem hiding this comment.
make this weekly
Collaborator
Author
There was a problem hiding this comment.
Done in 35cea03: 0 6 * * 1, Monday 06:00 UTC, the same slot as the weekly Docker check. Since every merge already runs the smoke tests, the schedule only has to catch a provider retiring a model during a week with no merges. Weekly is enough for that.
| options: [ smoke, all ] | ||
| default: smoke | ||
|
|
||
| # Never cancel a live run halfway: its calls are already paid for. |
Collaborator
Author
There was a problem hiding this comment.
who said that? calls are paid on a call basis, so if we cancel we pay half
Collaborator
Author
There was a problem hiding this comment.
Fair point, my reasoning was wrong. Providers bill per call, so cancelling halfway saves the calls not made yet. It does not waste the ones already made. Changed in 35cea03 to cancel-in-progress: true, so a newer run supersedes an in-flight one. The group is now live-tests-${{ github.event_name }}, so a merge only cancels the previous merge run and never a manual all run you started yourself.
luca-belli
marked this pull request as ready for review
October 1, 2026 21:18
Mirror the team .env, where OPENAI_API_KEY, ANTHROPIC_API_KEY, and GOOGLE_API_KEY all expand to one gateway API_KEY and API_BASE_URL points at the gateway. The repo then needs two secrets, not four, and the required-secrets check now also covers API_BASE_URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luca-belli
changed the title
OPENAI_API_KEY, ANTHROPIC_API_KEY, and GOOGLE_API_KEY each read a same-named repository secret and fall back to the shared gateway API_KEY, as the team .env does. One provider can then move to a direct key without changing the others. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
luca-belli
added a commit
that referenced
this pull request
Oct 2, 2026
Resolve the live.yml conflict with the self-hosted runner change (#236). Keep main's runner comment, but drop the python-version: '3.11' that main added to setup-uv, so the live job uses .python-version like the rest of CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #235, which is stacked on #234. This PR's base is #235's branch, so the diff shows only these changes.
Draft until the gateway is reachable from GitHub-hosted runners. The secrets are set, but the first live run would time out connecting to the internal gateway.
Summary
This adds a separate
.github/workflows/live.yml, so the API-key secrets never reach the PR workflow. It never runs on pull requests.maintest_model_availability.pyand the live 4-turnvera pipelinetest intest_vera_pipeline_e2e.pyworkflow_dispatchsmoke, orallfor everylivetest, including the costly legacytest_scoring.pyand the clinician-ratings judge check.allis manual-only, per the live-test consolidation item inTODOOPENAI_API_KEYorANTHROPIC_API_KEYis unset. Live tests skip themselves without keys, so otherwise the job would pass having tested nothing.allrun.AGENTS.mdandCLAUDE.mddescribe the workflow and the secrets it needs.Before merging
.env:API_BASE_URL(the gateway) andAPI_KEY, which the workflow passes asOPENAI_API_KEY,ANTHROPIC_API_KEY, andGOOGLE_API_KEY.llm-gateway.dev.springtest.ussits behind an internal AWS load balancer, and even public DNS resolves it to private10.31.x.xaddresses, so a hosted runner can't connect to it.workflow_dispatchfor workflows already onmain, so this one can't be started manually from the branch. The first real run is the merge push, which runs the smoke tests, and it can be re-run manually from then on.🤖 Generated with Claude Code