Repository navigation
Regression set for the translation validator - #899
Merged
Merged
Conversation
50 cases with their expected verdict: 40 fixes sampled from validation_log (Danish -> English) and hand-made controls for verb form, plus the case that started this: "vil sige" = "mean" shown over "Jeg lærte, hvad det vil sige at gå" (validator fix of 13 Aug 2026, log 13793). The rule the labels encode: an exercise shows the translation above the word in this very sentence, so the card must read right there. A case may list several acceptable cards or verdicts, and UNRESOLVED (give up) is a label for broken sentences, ahead of the validator learning that verdict. tools/run_validator_regression.py scores the validator against it, --runs times per case: at temperature 0 it is still not deterministic. Current prompt, 3 runs: 104/150. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ArchLens - No architecturally relevant changes to the existing views |
…y default) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The labels let through the fragment the note calls wrong (mere ved) and a base form the sentence does not contain (gøre for gør). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hvad det vil sige = what it means, as for the will-say case; and a FIX that keeps at slå op = to look up as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The validator has had no kept eval data, so prompt changes were judged on cases put together for the occasion. This adds a set to keep, plus a runner.
zeeguu/core/llm_services/regression/translation_validator.yaml: 50 cases. 40 are fixes sampled fromvalidation_log(Danish → English) and the rest are hand-made controls for verb form. It includes the case that started this: "vil sige" = "mean" shown over Jeg lærte, hvad det vil sige at gå (validator fix of 13 Aug, log 13793).UNRESOLVED(give up) is the label for broken sentences; the validator learns that verdict in the next PR.tools/run_validator_regression.py --runs 3scores the validator against the set. Each case runs several times because the validator isn't deterministic, even at temperature 0.Current prompt, 3 runs: 104/150. Of the failures, 34 are wrong meanings and 12 are broken sentences let through.
Adds PyYAML to requirements; it was only in the venv through newspaper3k.
🤖 Generated with Claude Code