Repository navigation
CI: run the eval tests on Ubuntu and Windows, and check E2 on saved copies - #16
Open
06pratyush wants to merge 4 commits into
Open
06pratyush wants to merge 4 commits into
06pratyush wants to merge 4 commits into
Conversation
The workflow gains an eval job that runs pytest eval/tests with Python 3.13 on Ubuntu and on Windows. The two authors work on different systems, and path handling is where they differ: #15's new tests fail on Windows only, which CI could not show because it ran the app's tests alone. Steps run under bash on both systems, so a failing command stops the job. The privacy suite now keeps each copy it saves in the test's folder under tests/test-results/ (Playwright empties it before each run), as Shashwat asked in #15. After the suite, the test job runs eval/check_files.py on all of them together, the E2 check of docs/eval-protocol.md, part E. That script comes with #15, so the step runs only once the file is on the branch being tested. With #15's script, the 18 copies the suite saves locally (6 per browser) all pass, with one segment layout and no ICC profile, under Python 3.12 and 3.13. When the job fails, the saved copies are uploaded with the report. The deploy job still waits only for the app's tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflict in docs/ai-use.md, where both sides added a row: keeps both, Shashwat's row for check_files.py (#15, merged first) before this branch's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI ran only the app's tests, so none of the eval tests ran on a pull request. That is how #15's 12 Windows failures got through. This PR adds two things to
.github/workflows/pages.yml.Eval tests on Ubuntu and Windows
A new
evaljob runspytest eval/testswith Python 3.13 onubuntu-24.04andwindows-2025. It installseval/requirements.txtand Chromium's headless shell. Steps run under bash on both systems, so a failing command stops the job.actions/setup-pythonis pinned to its v7.1.0 commit, like the other actions.The deploy job still waits only for the app's tests.
Check E2 on the privacy suite's saved copies
As asked in #15:
saved/folder inside each test's folder undertests/test-results/. Playwright empties that folder before each run, and a repeated name is numbered rather than overwritten.eval/check_files.pyon all of them together, so the ICC rule compares the three browsers' copies with each other. On failure, the saved copies and the report are uploaded with the Playwright report.For Shashwat:
eval/check_files.pyarrives with #15, so the step hasif: hashFiles('eval/check_files.py') != ''and switches itself on once #15 is merged. Until then it is skipped. After #15 merges, I'll drop the condition in a small follow-up, so a renamed script fails loudly instead of being skipped.Checked locally (Windows)
check_files.pyunder Python 3.12 (as on Ubuntu runners) and 3.13, passes all 18. They share one segment layout (JFIF, DQT ×2, SOF0, DHT ×4, SOS) and carry no ICC profile.pytest eval/testsonmaingives 78 passed./code-review(high) found four issues; two are fixed, and the other two are the condition explained above and the choice to keep both changes in one PR.🤖 Generated with Claude Code