docs: correct the edge-typing latency with a measured range (#513) - #519
Merged
Conversation
The docs said a 74-edge Structure stage was "roughly 9 minutes" on `gemma2:27b`. That number came from `evals/edge_typing/`, whose fixtures average 145 characters, and publishing it as how long a real run takes was the mistake: it is the optimistic end of a range, not a single value. Measured on one machine, same model, same day: | corpus | avg document | s/edge | | harness fixtures | ~145 chars | 6.8 | | examples/good-life-demo | ~1,220 chars | 11.5 | So 74 edges is 8.4 to 14.2 minutes depending on the corpus, and 200 edges is 22.7 to 38.4. The cause is structural rather than environmental. `_build_messages` puts the full body of BOTH documents into the prompt while the reply stays a short JSON object, so the input scales and the output does not -- an 8.4x larger input cost 1.70x the time, sublinear as that shape predicts. Re-running the harness on the same machine reproduced 6.8, which is what rules out load as the explanation; without that control the difference could equally have been the machine. Corrected in three places, plus the reason it varies: - `docs/cli.md` publishes the range, a table of both measured corpora, and projections at 20/74/200 edges. The model-comparison table's `s/edge` column is now labelled with the fixture size it was measured on. - `curate.py`'s `model_notice` docstring, which justified the disclosure line using the same optimistic figure. - `evals/edge_typing/README.md` gains a section on what `s/edge` does and does not tell you: it is for comparing models against each other, and publishing it as real-run latency without naming the document size is the error this commit fixes. No behavior change; documentation and one docstring only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #518. Documentation and one docstring only — no behavior change.
The error
I published "a 74-edge Structure stage is roughly 9 minutes on
gemma2:27b" indocs/cli.md, theCHANGELOG, and #518's body. That number came fromevals/edge_typing/, whose fixtures average 145 characters. Publishing a harness figure as how long a real run takes is the mistake — it is the optimistic end of a range, not a single value.Found by actually running it: the first end-to-end
curatewith the packaged default took 11.5 s/edge, 64% above the published figure.The measurement
Same machine, same model, same day:
examples/good-life-demoWhy, and how that was established rather than assumed
The cause is structural:
_build_messagesputs the full body of both documents into the prompt, while the reply is a short JSON object of roughly constant size. The input scales; the output does not. An 8.4× larger input cost 1.70× the time — sublinear, exactly the shape that explanation predicts.The competing explanation was machine load. Re-running the harness on the same machine reproduced 6.8 s/edge, matching the historical 7.0 from #516's sweep, which rules load out. Without that control the difference could equally have been the hardware, and the correction would have been a guess.
What changed
docs/cli.md— publishes the range, both measured corpora, projections at 20/74/200 edges, and the explanation of what makes it vary. The model-comparison table'ss/edgecolumn is now labelled with the fixture size it was measured on, so the same mistake is harder to repeat.src/openkos/cli/curate.py—model_notice's docstring justified the disclosure line with the same optimistic figure; now carries the range and the reason.evals/edge_typing/README.md— a new section on whats/edgedoes and does not tell you: it is for comparing models against each other, which is what the harness is for, and publishing it as real-run latency without naming the document size is the error this fixes.CHANGELOG— records the correction.Note on what is not claimed
The verification run scored 13/17 = 0.76 accuracy against #516's 0.81. That is a single run and accuracy is not what it measures; it is within expected variation and is not offered as evidence about accuracy either way. The harness artifacts from that run were deliberately not committed, so a stray n=1 number cannot be read as a result.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MraEQooNmnhKbUqaQ2xU3A