Repository navigation
Validator prompt version; regression runs and re-check data kept per version - #910
Merged
Merged
Conversation
…er version PROMPT_VERSION names the prompt (v1 = api#900). The regression runner saves its report with --save, and the re-check tool says which version judged. The audio audit's checker_id includes the version. Keeps the 25 Oct re-check of the top 300 past fixes and the v1 baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ws the word The 25 Oct re-check of past fixes found v1 demanding the translation's own agreement: Dutch "echt" -> "echte" because Dutch writes "een echte uitdaging", though the learner sees the French sentence. v3 exempts adjective endings and articles only the translation needs; tense, number and person still follow the word (Nutzer = gebruikers). 48/58 against v1's 46/58; v2 and v4, wider exemptions, were more lenient and scored lower. New cases from the re-check in the regression set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ArchLens - No architecturally relevant changes to the existing views |
… runs and the cases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mircealungu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No prompt change. Versions the validator prompt and keeps its evaluation data in the repo.
PROMPT_VERSIONinprompts/translation_validator.py:translation-validator-v1is the deployed prompt (Validator: the meaning must fit the sentence it is shown in #900). Bump it on every prompt change.tools/run_validator_regression.py --savewrites each run toregression/results/translation_validator/<version>--<date>.txt, with a README table of the versions.tools/recheck_validator_fixes.pysays which version judged.checker_idincludes the version, so code changes around the prompt also make old reports unusable for--deprecate.regression/recheck/translation_validator--2026-10-25--top300.csv(the first re-check of past fixes: 64 disagreements in the top 300) and 9 new regression cases from it.Tried and not shipped (runs kept): v2–v4 tuned how strictly the card must match the translation's own grammar (Dutch echt vs echte). The best, v3, scored 48/58 against v1's 46/58, which is within run-to-run noise, for a longer prompt with exceptions. Instead the plan is to keep the looked-up form on the bookmark and the learning unit on the user word (docs
future-work/bookmark-and-learning-unit.md). The adjective cases stay in the set as known v1 misses.🤖 Generated with Claude Code