Skip to content

feat(cli): disclose what bulk structure acceptance spends, and diagnose #513 - #514

Merged
jasonssdev merged 1 commit into
mainfrom
feat/513-edge-typing-diagnosis
Aug 9, 2026
Merged

feat(cli): disclose what bulk structure acceptance spends, and diagnose #513#514
jasonssdev merged 1 commit into
mainfrom
feat/513-edge-typing-diagnosis

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

Refs #513. The fixture grew a held-out pair (17 edges) and every arm was re-measured on that one fixture, so the repo holds a single comparable generation.

The finding: the model dominates every prompt change

Same prompt, same fixture, only the model swapped:

model accuracy stability
gemma2:9b 0.63 0.92
qwen2.5:7b 0.55 0.80
llama3.1:8b 0.45 0.76
qwen3:8b (configured default) 0.44 0.98
mistral:7b 0.27 0.92

The default is next to last, and +0.19 is available from a config value — larger than any prompt change measured here.

It is not a safe swap and this PR does not make it. The default is global and extraction was tuned on qwen3:8b through extraction_cap; changing it would move a pipeline this harness does not score. Filed separately as the real fix.

Three prompt arms, none shippable

arm accuracy verdict
baseline 0.44 (0.41–0.45 over four runs)
few-shot 0.51 rejected — see below
less-priming 0.41 no effect
evidence-first 0.29 actively harmful

evidence-first asked for the supporting sentence before the type, and pushed related_to from 0.65 to 0.93 of emissions. Naming the abstention more primed it — the exact failure edge_typing.py already records from #388's era: "a clause forbidding a shape made that shape more frequent through priming".

few-shot reads as a +0.07 win and I nearly shipped it. The entire gain sits on 2 of 17 edges, both member_of, and both phrased like the member_of example — "one of the … like the van and the truck" against "one of the scheduled maintenance jobs … each registered the same way". The member_of pair written in deliberately different language ("a roster whose entries are peers") does not move, because baseline already answers it correctly.

Six worked examples, one confusion moved, only where the surface form matched — with the examples and the fixture written by the same author. That is pattern-matching on phrasing, not a model that learned the distinction. Rejected.

What ships

Precision per emitted type — "when it says T, how often is T right":

emitted correct emissions precision
member_of 6 6 1.00
produced_by 5 5 1.00
part_of 10 19 0.53
related_to 15 50 0.30
references 0 5 0.00

Specific types aggregate to 0.60 — exactly what curate --accept structure writes unreviewed. Roughly two in five bulk-applied relation types are wrong by the rubric, and a wrong part_of asserts something false that everything reading the graph then believes.

So an accepted Structure now prints one stderr advisory naming that types go in unreviewed and where the measurement lives. The flag stays — the operator asked for it. Going in blind does not.

related_to at 0.30 is worth stating plainly: seven times in ten it is emitted where the documents state a specific relationship, which the prompt's own tie-break (3) says must resolve to a specific type. That is under-claiming, not caution.

Stability is 0.98 throughout, so sampling-based mitigations are out: voting and self-consistency sample the same settled mistake, and #508 already measured and rejected a stated-confidence threshold.

Verification

  • Suite 4050 → 4052. ruff and mypy clean over the whole repo.
  • The advisory is mutation-verified, and a second test pins that it stays silent on a per-item walk so it cannot become noise.
  • One process note: an early attempt ran three "arms" through a broken heredoc that never applied the patch, so all three reported baseline. Caught because the numbers clustered at baseline; re-run from a script file. The reported figures are the corrected ones.

Disclosure: no review receipt

Same lens defect as PRs #503, #506, #507, #509, #510, #511 and #512.

#513

Refs #513.

`evals/edge_typing/` grew a held-out pair (17 edges now) and every arm was
re-measured on that one fixture, so the repository holds a single
comparable generation.

THE FINDING. Same prompt, same fixture, only the model changed:
`gemma2:9b` 0.63, `qwen2.5:7b` 0.55, `llama3.1:8b` 0.45, `qwen3:8b`
(the configured default) 0.44, `mistral:7b` 0.27. The default is next to
last, and the +0.19 available from a config value is larger than any
prompt change measured here. Not a safe swap -- the default is global and
extraction was tuned on `qwen3:8b` through `extraction_cap` -- so it is
recorded, not applied.

Three prompt arms, none shippable. `evidence-first` (quote the supporting
sentence first) drove `related_to` from 0.65 to 0.93 of emissions and
accuracy to 0.29: naming the abstention more primed it, the same failure
mode `edge_typing.py` already records. `less-priming` moved nothing.
`few-shot` reads as +0.07 but the entire gain sits on 2 of 17 edges, both
`member_of`, both phrased like the `member_of` example -- while the
`member_of` pair written in different language does not move because
baseline already answers it. Six examples, one confusion, only where the
surface form matched: pattern-matching on phrasing the same author wrote
on both sides. Rejected.

WHAT SHIPS. Precision per emitted type puts specific types at 0.60
aggregate -- exactly what `--accept structure` writes unreviewed -- and
`references` at 0.00 across 5 emissions. So an accepted Structure now
prints one stderr advisory naming that types go in unreviewed and where
the measurement lives. The flag stays; the operator asked for it. Going
in blind does not.

`related_to` sits at 0.30 precision: seven times in ten it is emitted
where the documents state a specific relationship, which is the case the
prompt's own tie-break (3) says must resolve to a specific type. That is
under-claiming, not caution.

Stability is 0.98 throughout, so the suggester is not guessing, and
sampling-based mitigations are out: voting and self-consistency sample
the same settled mistake, and #508 already measured and rejected a
stated-confidence threshold.

Suite 4050 -> 4052.
@jasonssdev
jasonssdev merged commit 5e53555 into main Aug 9, 2026
6 checks passed
@jasonssdev
jasonssdev deleted the feat/513-edge-typing-diagnosis branch August 9, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant