feat(cli): stop accepting related_to in bulk (#508) - #511
Merged
Conversation
First slice of #508, and the one that needs no prompt change. `--accept structure` was applying, unreviewed, exactly the suggestions where the model had declined to claim anything. `related_to` is not a wrong answer -- the rubric defines it as an answer, and an honest one beats a guessed `part_of` that asserts something false about how the knowledge fits together. What sets it apart is narrower and uncontested: applying it adds no claim to the graph beyond the untyped link that was already there. That makes it the cheapest place to spend a human glance and, at a measured 67% of accepted edges, also the largest. An accepted Structure now applies every specific type without asking and routes `related_to` to the operator. On a non-TTY run there is no channel to ask on, so it is counted as skipped rather than prompted -- reaching `typer.prompt` with no terminal would kill the walk mid-run, the same failure the Identity non-TTY guard exists to prevent. An unattended run therefore writes the confident suggestions and leaves the rest queued. The exemption is scoped to the ITEM, not the stage: the same run still applies every specific suggestion in bulk. The numeric-confidence route stays out of scope. `EdgeSuggestion` and `TierSuggestion` carry no confidence, adding it is a prompt change, and no eval under `evals/` scores either suggester -- so there is nothing to A/B against. Building that harness is its own piece of work and #508 stays open for it. Spec, docs and CHANGELOG updated. Suite 4048 -> 4050.
This was referenced Aug 9, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of #508 — the one that needs no prompt change. #508 stays open for the rest.
The hole this closes
PR #509 shipped
--accept structurehours ago. It was applying, unreviewed, exactly the suggestions where the model had declined to claim anything.related_tois not a wrong answer, and this is not a walk-back of the rubric.edge_typing.pyis explicit that it "is defined as an ANSWER, not as a shrug", and the prompt says an honestrelated_tobeats a guessedpart_of, which "asserts something false about how the knowledge fits together, and anything reading this graph will believe it".What sets it apart is narrower and uncontested: applying it adds no claim to the graph beyond the untyped link that was already there. That makes it the cheapest place to spend a human glance and — at 67% of accepted edges, measured on a real bundle (
edge_typing.py:146) — also the largest. Note that corrects #385's "roughly 80%" from memory.So
--accept structurenow applies every specific type without asking and routes this one to the operator. The exemption is scoped to the item, not the stage.Non-TTY
On a pipe there is nothing to prompt on, so the exempted item is skipped, not prompted. Without that, the walk would reach
typer.promptwith no terminal and die mid-run — the same failure the Identity non-TTY guard exists to prevent, and it exits 1 when mutated. An unattended run writes the confident suggestions and leaves the rest queued.Why the numeric-confidence route did not ship
Investigating first changed the shape of the work.
EdgeSuggestionandTierSuggestioncarry no confidence, adding it is a prompt change — and no eval underevals/scores either suggester. The cap harness measures extraction, a different pipeline. So there is nothing to A/B a prompt change against, and this project has already lost a prompt A/B once by adopting on intuition.Building that harness is its own piece of work. #508 stays open for it; this slice delivers the behavior it was actually after, deterministically.
Metadata gets no equivalent: its three tiers are all substantive answers with no abstention token, and its volume is per concept type rather than per edge, so the prompt-count problem is Structure's almost entirely.
Verification
ruffandmypyclean over the whole repo.Disclosure: no review receipt
Same lens defect as PRs #503, #506, #507, #509 and #510.