ci: run evals weekly and record results on main

[angeloashmore]

Resolves:

Description

Runs the eval suite in CI: weekly on Monday 06:00 UTC, and on demand via workflow dispatch. Results land on main through a bot-opened PR the job squash-merges immediately, since main's ruleset requires changes to arrive by PR. The results PR body carries the run's report, so watching the repo's PRs doubles as a weekly report; run-over-run history is the git log of evals/results.json.

Eval failures do not fail the job; evals are measurements, not gates. The job fails only when evals/results.json was not rewritten (the commit is the gate), meaning the harness died before grading. The eval step never receives GH_TOKEN because the agent under test can read the step's environment.

Requires one new repository secret: ANTHROPIC_API_KEY (already set).

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA ¹

After this merges: Actions → evals → Run workflow. The run takes roughly 10–15 minutes and about $31; confirm it opens and squash-merges a test(eval): record eval run at … PR whose body shows the per-eval report.

🤖 Generated with Claude Code

---

Note

Medium Risk
Grants CI write/PR permissions and uses production API and e2e secrets, but scope is limited to eval automation with deliberate token isolation from the agent step.

Overview
Adds a new evals GitHub Actions workflow that runs the existing npm run evals suite on a schedule (Mondays 06:00 UTC), on workflow_dispatch, and temporarily on pushes to aa/evals-workflow for pre-merge testing.

The job uses a concurrency group so runs don’t overlap on the shared e2e Prismic account, checks out with persist-credentials: false so the agent under test can’t use the checkout token to push, and keeps GH_TOKEN out of the eval step (only the record step gets it). Eval failures are non-blocking (|| true); recording on main only opens a bot PR that commits evals/results.json, embeds the evals:report output in the PR body, and squash-merges immediately to satisfy branch protection.

^{Reviewed by Cursor Bugbot for commit 68bb4ab. Bugbot is set up for automated code reviews on this repo. Configure here.}

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1adbd64. Configure here.}

[lihbr]

LGTM, maybe it's over optimizing it, but I'm wondering if we should just skip the run if there hasn't been a new release/tag since the last run.

[angeloashmore]

@lihbr: LGTM, maybe it's over optimizing it, but I'm wondering if we should just skip the run if there hasn't been a new release/tag since the last run.

I considered this initially, but decided against it. Models seem to change over time, even when using the same version. A perpetual recurring check seems like the best way to ensure we are always evaluating current capabilities.

On the other hand, we'll probably make CLI changes frequently enough to keep up with model changes.

I think we can keep it as once a week for now. If we find the frequency is too expensive or unnecessary, we can add a version check to do what you're suggesting.