Skip to content

fix: entity extraction quality + feat: LLM response caching - #3

Merged
Joshuaakaspace merged 2 commits into
mainfrom
claude/repo-review-vk08e5
Aug 16, 2026
Merged

Joshuaakaspace merged 2 commits into
mainfrom
claude/repo-review-vk08e5

Conversation

@Joshuaakaspace

Copy link
Copy Markdown
Contributor

Follow-up to #2. 614 tests pass (was 540), ruff check and ruff format --check clean.

Extraction quality

Entity names feed the graph, the wiki, query, and diff, so a naming defect degrades all four at once. Two were doing exactly that.

Feature bullets became prose, not names. The convention is Name -- description, but the splitter only knew : and the typographic dashes — it missed the ASCII -- this project actually uses. The whole bullet was kept and truncated at 80 characters, which is why graph nodes and wiki headings appeared ellipsised mid-sentence:

before: "15 Output Formats -- Word, PDF, PPTX, Blog Post, Jira Tickets, Changelog, L…"
after:  "15 Output Formats"

Splitting now covers the ASCII form, tries multi-character dashes before the bare hyphen so Auto-PR isn't cut in half, and falls back to the first sentence then a word-boundary trim when a bullet has no separator. The full text is still kept in the entity's description property.

Any number followed by an s-word became a duration. The bare s/k/M units matched the first letter of the next word:

"all 12 selectable output formats"  ->  metric "12 s"
"0 sections added"                  ->  metric "0 s"
"takes 30 seconds"                  ->  metric "30 s"   (unit truncated)

Units are now ordered longest-first, and a trailing lookahead requires the unit to end at a non-letter.

Measured on this repo's own README: entity names over 60 characters 15 → 0, the phantom 12 s / 0 s / 5 s metrics gone, and total entities 35 → 32 — precision gained rather than recall lost.

LLM response caching

--mode llm re-issued every request on every run, so regenerating docs from an unchanged README re-paid the full API cost.

Caching sits in LLMProvider.chat / chat_json — the chokepoint all five providers already share — so OpenAI, Anthropic, Google, Ollama and Azure are covered without touching any provider individually. It's process-wide, so LLMExtractor, LLMSummarizer and LLMContentEnhancer also reuse each other's results within a single run.

The key covers provider, model, temperature, token ceiling, call kind, and the exact prompts. Temperature and max_tokens are in there deliberately — replaying an answer produced under different sampling settings would be wrong.

This trades novelty for cost and reproducibility. With temperature > 0 the provider would have returned a different answer and you get the earlier one. That's the point for repeated builds, and why OPENDOCS_LLM_CACHE=0 and --no-cache both disable it. --no-cache now covers artifacts and responses alike, since that's what "don't reuse anything" means.

Two details worth recording:

  • chat_json caches the raw provider text, not the parsed object, so entries stay faithful to what was returned and later parsing improvements apply to already-cached responses.
  • A response that fails to parse is never stored, so a bad reply can't become permanent.

Verified across processes: a second invocation makes zero API calls and returns the identical answer.

Testing

74 new tests. The LLM cache tests use a counting fake provider that records how often the transport layer would have been reached — a cache hit has to mean an API call not made. The extraction tests include dogfooding assertions that this repo's own README yields no prose-length names and no phantom durations.


Generated by Claude Code

claude added 2 commits August 16, 2026 09:08
Entity names feed the graph, wiki, query and diff, so these two defects
degraded every one of those views at once. On this repo's own README, 15 of
35 entity names were prose rather than concepts, and three of the metrics did
not exist.

Feature bullets are conventionally "Name -- description", but the splitter
only knew ':' and the typographic dashes, missing the ASCII '--' this project
actually uses. The whole bullet was therefore kept and merely truncated at 80
characters, which is why graph nodes and wiki headings appeared ellipsised
mid-sentence. Splitting now covers the ASCII form, tries multi-character
dashes before the bare hyphen so "Auto-PR" is not cut in half, and falls back
to the first sentence and then a word-boundary trim when a bullet has no
separator at all. The full text is still kept in the entity's description
property, so trimming the name loses nothing.

The metric pattern let the bare "s", "k" and "M" units match the first letter
of any following word, so "12 selectable formats" and "0 sections added"
became durations. Units are now ordered longest-first — "30 seconds" was also
capturing its unit as a bare "s" — and a trailing lookahead requires the unit
to end at a non-letter.

Measured on this README: entity names over 60 characters 15 -> 0, and the
bogus "12 s" / "0 s" / "5 s" metrics are gone, with real entity count
essentially unchanged (35 -> 32), so this is precision gained rather than
recall lost.

Tests: 574 pass (34 new), including dogfooding assertions that this repo's
own README yields no prose-length names and no phantom durations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
`--mode llm` re-issued every request on every run, so regenerating docs from
an unchanged README paid the full API cost again. Responses are now cached on
disk and reused.

Caching sits in LLMProvider.chat/chat_json, the single chokepoint all five
providers already share, so OpenAI, Anthropic, Google, Ollama and Azure are
covered without touching any provider individually. The cache is process-wide,
so LLMExtractor, LLMSummarizer and LLMContentEnhancer also benefit from each
other's results within one run, not only across runs.

The key covers provider, model, temperature, token ceiling, call kind and the
exact prompts. Temperature and max_tokens are in there deliberately: replaying
an answer produced under different sampling settings would be wrong, so
changing either re-asks.

This trades novelty for cost and reproducibility. With temperature > 0 the
provider would have returned a different answer and you get the earlier one —
which is the point for repeated builds, and why OPENDOCS_LLM_CACHE=0 and
--no-cache both turn it off. --no-cache now covers artifacts and responses
alike, since that is what "don't reuse anything" means.

Two details: chat_json caches the raw provider text rather than the parsed
object, so entries stay faithful to what was returned and later parsing
improvements apply to already-cached responses; and a response that fails to
parse is never stored, so a bad reply cannot become permanent.

Verified across processes: a second invocation makes zero API calls and
returns the identical answer.

Tests: 614 pass (40 new). A counting fake provider asserts the property that
matters — a cache hit is an API call not made.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
@Joshuaakaspace
Joshuaakaspace merged commit 6df9a13 into main Aug 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants