Repository navigation
fix: entity extraction quality + feat: LLM response caching - #3
Merged
Merged
Conversation
Entity names feed the graph, wiki, query and diff, so these two defects degraded every one of those views at once. On this repo's own README, 15 of 35 entity names were prose rather than concepts, and three of the metrics did not exist. Feature bullets are conventionally "Name -- description", but the splitter only knew ':' and the typographic dashes, missing the ASCII '--' this project actually uses. The whole bullet was therefore kept and merely truncated at 80 characters, which is why graph nodes and wiki headings appeared ellipsised mid-sentence. Splitting now covers the ASCII form, tries multi-character dashes before the bare hyphen so "Auto-PR" is not cut in half, and falls back to the first sentence and then a word-boundary trim when a bullet has no separator at all. The full text is still kept in the entity's description property, so trimming the name loses nothing. The metric pattern let the bare "s", "k" and "M" units match the first letter of any following word, so "12 selectable formats" and "0 sections added" became durations. Units are now ordered longest-first — "30 seconds" was also capturing its unit as a bare "s" — and a trailing lookahead requires the unit to end at a non-letter. Measured on this README: entity names over 60 characters 15 -> 0, and the bogus "12 s" / "0 s" / "5 s" metrics are gone, with real entity count essentially unchanged (35 -> 32), so this is precision gained rather than recall lost. Tests: 574 pass (34 new), including dogfooding assertions that this repo's own README yields no prose-length names and no phantom durations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
`--mode llm` re-issued every request on every run, so regenerating docs from an unchanged README paid the full API cost again. Responses are now cached on disk and reused. Caching sits in LLMProvider.chat/chat_json, the single chokepoint all five providers already share, so OpenAI, Anthropic, Google, Ollama and Azure are covered without touching any provider individually. The cache is process-wide, so LLMExtractor, LLMSummarizer and LLMContentEnhancer also benefit from each other's results within one run, not only across runs. The key covers provider, model, temperature, token ceiling, call kind and the exact prompts. Temperature and max_tokens are in there deliberately: replaying an answer produced under different sampling settings would be wrong, so changing either re-asks. This trades novelty for cost and reproducibility. With temperature > 0 the provider would have returned a different answer and you get the earlier one — which is the point for repeated builds, and why OPENDOCS_LLM_CACHE=0 and --no-cache both turn it off. --no-cache now covers artifacts and responses alike, since that is what "don't reuse anything" means. Two details: chat_json caches the raw provider text rather than the parsed object, so entries stay faithful to what was returned and later parsing improvements apply to already-cached responses; and a response that fails to parse is never stored, so a bad reply cannot become permanent. Verified across processes: a second invocation makes zero API calls and returns the identical answer. Tests: 614 pass (40 new). A counting fake provider asserts the property that matters — a cache hit is an API call not made. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2. 614 tests pass (was 540),
ruff checkandruff format --checkclean.Extraction quality
Entity names feed the graph, the wiki,
query, anddiff, so a naming defect degrades all four at once. Two were doing exactly that.Feature bullets became prose, not names. The convention is
Name -- description, but the splitter only knew:and the typographic dashes — it missed the ASCII--this project actually uses. The whole bullet was kept and truncated at 80 characters, which is why graph nodes and wiki headings appeared ellipsised mid-sentence:Splitting now covers the ASCII form, tries multi-character dashes before the bare hyphen so
Auto-PRisn't cut in half, and falls back to the first sentence then a word-boundary trim when a bullet has no separator. The full text is still kept in the entity'sdescriptionproperty.Any number followed by an s-word became a duration. The bare
s/k/Munits matched the first letter of the next word:Units are now ordered longest-first, and a trailing lookahead requires the unit to end at a non-letter.
Measured on this repo's own README: entity names over 60 characters 15 → 0, the phantom
12 s/0 s/5 smetrics gone, and total entities 35 → 32 — precision gained rather than recall lost.LLM response caching
--mode llmre-issued every request on every run, so regenerating docs from an unchanged README re-paid the full API cost.Caching sits in
LLMProvider.chat/chat_json— the chokepoint all five providers already share — so OpenAI, Anthropic, Google, Ollama and Azure are covered without touching any provider individually. It's process-wide, soLLMExtractor,LLMSummarizerandLLMContentEnhanceralso reuse each other's results within a single run.The key covers provider, model, temperature, token ceiling, call kind, and the exact prompts. Temperature and
max_tokensare in there deliberately — replaying an answer produced under different sampling settings would be wrong.This trades novelty for cost and reproducibility. With
temperature > 0the provider would have returned a different answer and you get the earlier one. That's the point for repeated builds, and whyOPENDOCS_LLM_CACHE=0and--no-cacheboth disable it.--no-cachenow covers artifacts and responses alike, since that's what "don't reuse anything" means.Two details worth recording:
chat_jsoncaches the raw provider text, not the parsed object, so entries stay faithful to what was returned and later parsing improvements apply to already-cached responses.Verified across processes: a second invocation makes zero API calls and returns the identical answer.
Testing
74 new tests. The LLM cache tests use a counting fake provider that records how often the transport layer would have been reached — a cache hit has to mean an API call not made. The extraction tests include dogfooding assertions that this repo's own README yields no prose-length names and no phantom durations.
Generated by Claude Code