feat(cli): disclose what bulk structure acceptance spends, and diagnose #513 - #514
Merged
Conversation
#513 Refs #513. `evals/edge_typing/` grew a held-out pair (17 edges now) and every arm was re-measured on that one fixture, so the repository holds a single comparable generation. THE FINDING. Same prompt, same fixture, only the model changed: `gemma2:9b` 0.63, `qwen2.5:7b` 0.55, `llama3.1:8b` 0.45, `qwen3:8b` (the configured default) 0.44, `mistral:7b` 0.27. The default is next to last, and the +0.19 available from a config value is larger than any prompt change measured here. Not a safe swap -- the default is global and extraction was tuned on `qwen3:8b` through `extraction_cap` -- so it is recorded, not applied. Three prompt arms, none shippable. `evidence-first` (quote the supporting sentence first) drove `related_to` from 0.65 to 0.93 of emissions and accuracy to 0.29: naming the abstention more primed it, the same failure mode `edge_typing.py` already records. `less-priming` moved nothing. `few-shot` reads as +0.07 but the entire gain sits on 2 of 17 edges, both `member_of`, both phrased like the `member_of` example -- while the `member_of` pair written in different language does not move because baseline already answers it. Six examples, one confusion, only where the surface form matched: pattern-matching on phrasing the same author wrote on both sides. Rejected. WHAT SHIPS. Precision per emitted type puts specific types at 0.60 aggregate -- exactly what `--accept structure` writes unreviewed -- and `references` at 0.00 across 5 emissions. So an accepted Structure now prints one stderr advisory naming that types go in unreviewed and where the measurement lives. The flag stays; the operator asked for it. Going in blind does not. `related_to` sits at 0.30 precision: seven times in ten it is emitted where the documents state a specific relationship, which is the case the prompt's own tie-break (3) says must resolve to a specific type. That is under-claiming, not caution. Stability is 0.98 throughout, so the suggester is not guessing, and sampling-based mitigations are out: voting and self-consistency sample the same settled mistake, and #508 already measured and rejected a stated-confidence threshold. Suite 4050 -> 4052.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #513. The fixture grew a held-out pair (17 edges) and every arm was re-measured on that one fixture, so the repo holds a single comparable generation.
The finding: the model dominates every prompt change
Same prompt, same fixture, only the model swapped:
gemma2:9bqwen2.5:7bllama3.1:8bqwen3:8b(configured default)mistral:7bThe default is next to last, and +0.19 is available from a config value — larger than any prompt change measured here.
It is not a safe swap and this PR does not make it. The default is global and extraction was tuned on
qwen3:8bthroughextraction_cap; changing it would move a pipeline this harness does not score. Filed separately as the real fix.Three prompt arms, none shippable
evidence-firstasked for the supporting sentence before the type, and pushedrelated_tofrom 0.65 to 0.93 of emissions. Naming the abstention more primed it — the exact failureedge_typing.pyalready records from #388's era: "a clause forbidding a shape made that shape more frequent through priming".few-shotreads as a +0.07 win and I nearly shipped it. The entire gain sits on 2 of 17 edges, bothmember_of, and both phrased like themember_ofexample — "one of the … like the van and the truck" against "one of the scheduled maintenance jobs … each registered the same way". Themember_ofpair written in deliberately different language ("a roster whose entries are peers") does not move, because baseline already answers it correctly.Six worked examples, one confusion moved, only where the surface form matched — with the examples and the fixture written by the same author. That is pattern-matching on phrasing, not a model that learned the distinction. Rejected.
What ships
Precision per emitted type — "when it says T, how often is T right":
member_ofproduced_bypart_ofrelated_toreferencesSpecific types aggregate to 0.60 — exactly what
curate --accept structurewrites unreviewed. Roughly two in five bulk-applied relation types are wrong by the rubric, and a wrongpart_ofasserts something false that everything reading the graph then believes.So an accepted Structure now prints one stderr advisory naming that types go in unreviewed and where the measurement lives. The flag stays — the operator asked for it. Going in blind does not.
related_toat 0.30 is worth stating plainly: seven times in ten it is emitted where the documents state a specific relationship, which the prompt's own tie-break (3) says must resolve to a specific type. That is under-claiming, not caution.Stability is 0.98 throughout, so sampling-based mitigations are out: voting and self-consistency sample the same settled mistake, and #508 already measured and rejected a stated-confidence threshold.
Verification
ruffandmypyclean over the whole repo.Disclosure: no review receipt
Same lens defect as PRs #503, #506, #507, #509, #510, #511 and #512.