[mircealungu]

Builds on #899 (the regression set, merged).

Problem. An exercise showed "mean" above "vil sige" in Jeg lærte, hvad det vil sige at gå ("what it mean to walk"). The validator passes that pair in 3 of 3 runs ("a correct, tense-neutral gloss"), and on 13 Aug it fixed "will say" to that same "mean". It checks whether a translation holds in other sentences, never whether it reads right in this one.

The PR in three lines

• Prompt: fit this sentence first, and you may give up.
• Reply: the model writes the sentence with its meaning in place and says whether it reads right.
• Code: if it says no, ask once more, then give up. It never returns a meaning the model itself called wrong.

Changes

• Prompt (prompts/translation_validator.py). STEP 1 now runs: (1) is the sentence usable, (2) does the translation fit this sentence, form included (English may add "does"/"not"), (3) is it the word's own meaning, tested in another sentence (the noget = "one" example), expanding to the smallest expression worth learning, (4) the "to" rule, (5) give up. A corrected word must appear in the sentence as written. Correct but less common translations are not "improved".

• Two ways to give up:

  • BROKEN_SENTENCE: the sentence is cut off, ungrammatical or nonsense.
  • WORD_NOT_FIT: the sentence is fine, but no meaning of the word fits it.

  Both take only this learner's word out of study, with a log line giving the kind and the sentence. The meaning is not touched, nothing is written to validation_log (whose action says what happened to a meaning), and user_preference (the learner's choice) is not written either. When the learner acts on the word again (a new look-up, starring), it's judged again.

• Self-check in code (translation_validator.py). Four new reply fields hold the sentence with the meaning in place, and whether it reads right, for the meaning given and for any correction. _misfit acts on them:

  • certain (the model says it reads wrong, or a corrected word isn't in the sentence): one retry quoting the model's own sentence, then WORD_NOT_FIT;
  • uncertain (the translation isn't word for word in the model's sentence, which separable verbs do legitimately): kept.

  "/" alternatives and "(glosses)" are reduced to one option before the fit check, keeping the model's own judgement. A failed retry is no verdict and goes to Sentry, unless the first answer would have been kept anyway.

• One validator, two callers. The validator takes ask (who answers) and retry. The audio-lesson check (second_opinion_on_meaning, formerly judge_pair) is now the same validator asked of DeepSeek without retries, so the decision logic lives in one place. A give-up there counts as not confirmed (script_generator._meaning_confirmed_wrong), so it never marks a meaning INVALID. "true"/"false" sent as strings by DeepSeek count as booleans.

Regression set (3 runs per case)

	before	after
runs passing	104/150	126/147

"vil sige" = "mean" is no longer accepted in any run; from "will say" it comes out as "means". Still missed: Provencesanger (passes as "Provence singer"; it's a bird), the broken Kan jeg at få…, and mere ved. Results vary by about ±4 runs between identical runs. The noget case left the set when it became the prompt's example.

Cost: about 60 more output tokens per call, plus one retry when a meaning doesn't fit (worst case 3 calls per word, normally 1).

Deploy: no migration. After deploy, old audio-lesson audit reports can't be acted on (checker_id hashes the prompt); re-run the audit before any --deprecate.

Not in this PR (see docs/future-work/llm-tasks-roadmap.md):

• generated examples must never change the meaning, and a broken sentence gets another sentence: A generated example never changes the learner's meaning; a broken sentence gets another #902;
• re-checking past FIX results ("vil sige" itself);
• replacing user_word.user_preference with study_choice + unfit_reason.

🤖 Generated with Claude Code

[github-actions]

ArchLens detected architectural changes in the following views: