Skip to content

feat(cli): stop accepting related_to in bulk (#508) - #511

Merged
jasonssdev merged 1 commit into
mainfrom
feat/508-abstention-not-accepted-in-bulk
Aug 9, 2026
Merged

feat(cli): stop accepting related_to in bulk (#508)#511
jasonssdev merged 1 commit into
mainfrom
feat/508-abstention-not-accepted-in-bulk

Conversation

@jasonssdev

Copy link
Copy Markdown
Owner

First slice of #508 — the one that needs no prompt change. #508 stays open for the rest.

The hole this closes

PR #509 shipped --accept structure hours ago. It was applying, unreviewed, exactly the suggestions where the model had declined to claim anything.

related_to is not a wrong answer, and this is not a walk-back of the rubric. edge_typing.py is explicit that it "is defined as an ANSWER, not as a shrug", and the prompt says an honest related_to beats a guessed part_of, which "asserts something false about how the knowledge fits together, and anything reading this graph will believe it".

What sets it apart is narrower and uncontested: applying it adds no claim to the graph beyond the untyped link that was already there. That makes it the cheapest place to spend a human glance and — at 67% of accepted edges, measured on a real bundle (edge_typing.py:146) — also the largest. Note that corrects #385's "roughly 80%" from memory.

So --accept structure now applies every specific type without asking and routes this one to the operator. The exemption is scoped to the item, not the stage.

Non-TTY

On a pipe there is nothing to prompt on, so the exempted item is skipped, not prompted. Without that, the walk would reach typer.prompt with no terminal and die mid-run — the same failure the Identity non-TTY guard exists to prevent, and it exits 1 when mutated. An unattended run writes the confident suggestions and leaves the rest queued.

Why the numeric-confidence route did not ship

Investigating first changed the shape of the work. EdgeSuggestion and TierSuggestion carry no confidence, adding it is a prompt change — and no eval under evals/ scores either suggester. The cap harness measures extraction, a different pipeline. So there is nothing to A/B a prompt change against, and this project has already lost a prompt A/B once by adopting on intuition.

Building that harness is its own piece of work. #508 stays open for it; this slice delivers the behavior it was actually after, deterministically.

Metadata gets no equivalent: its three tiers are all substantive answers with no abstention token, and its volume is per concept type rather than per edge, so the prompt-count problem is Structure's almost entirely.

Verification

  • Suite 4048 → 4050. ruff and mypy clean over the whole repo.
  • Mutation-verified: re-accepting the abstention in bulk, and disabling the non-TTY branch, each fail their tests.

Disclosure: no review receipt

Same lens defect as PRs #503, #506, #507, #509 and #510.

First slice of #508, and the one that needs no prompt change.

`--accept structure` was applying, unreviewed, exactly the suggestions
where the model had declined to claim anything. `related_to` is not a
wrong answer -- the rubric defines it as an answer, and an honest one
beats a guessed `part_of` that asserts something false about how the
knowledge fits together. What sets it apart is narrower and uncontested:
applying it adds no claim to the graph beyond the untyped link that was
already there. That makes it the cheapest place to spend a human glance
and, at a measured 67% of accepted edges, also the largest.

An accepted Structure now applies every specific type without asking and
routes `related_to` to the operator. On a non-TTY run there is no channel
to ask on, so it is counted as skipped rather than prompted -- reaching
`typer.prompt` with no terminal would kill the walk mid-run, the same
failure the Identity non-TTY guard exists to prevent. An unattended run
therefore writes the confident suggestions and leaves the rest queued.

The exemption is scoped to the ITEM, not the stage: the same run still
applies every specific suggestion in bulk.

The numeric-confidence route stays out of scope. `EdgeSuggestion` and
`TierSuggestion` carry no confidence, adding it is a prompt change, and
no eval under `evals/` scores either suggester -- so there is nothing to
A/B against. Building that harness is its own piece of work and #508
stays open for it.

Spec, docs and CHANGELOG updated. Suite 4048 -> 4050.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant