Repository navigation
fix: repo review defects + four new features (query, lint, diff, build cache) - #2
Merged
Merged
Conversation
Fixes found while reviewing the repository. Each fix has a regression test that fails against the previous code. Correctness: - cli: `opendocs inspect <notebook.ipynb>` crashed with UnboundLocalError because `name` was only assigned on the Markdown branch. - cli: `mindmap` was present in FORMAT_MAP and in the `--format all` set but missing from the click Choice, so `-f mindmap` was rejected. The choice list and the all-formats list are now derived from FORMAT_MAP, which is what let them drift apart across three commands. - cli/pipeline: `--no-outputs` was parsed but never forwarded, so notebook cell outputs were always included. - latex: `_escape` never escaped the backslash, so any README containing one produced a .tex that would not compile (`50% \& x` became `50\% \\& x`, where `\\` is a line break). - watcher: `_create_docs_pr` stashed the user's uncommitted work but returned without popping it on three failure paths, including the common failed-push case. It also popped unconditionally, discarding a stash the user had made themselves when there was nothing of ours to stash. - watcher: the PR base branch was hardcoded to `main`, breaking any repo that uses another default. It is now resolved from the remote. - knowledge_graph: label propagation shuffled with the global unseeded RNG, so community IDs in graph.json and the wiki changed between runs. The shuffle is now seeded (overridable via `seed=`). Injection: - interactive_graph: entity names reach the page unsanitised. They are untrusted — mermaid node labels are lifted verbatim from README content, and in LLM mode names come from model output. `json.dumps` does not escape `/`, so a name containing `</script>` closed the script element early, and four panels then wrote names straight to `innerHTML`. Serialisation now escapes `</`, `<!--` and U+2028/9, and every innerHTML sink is escaped. Reproducible end to end in basic mode with no API key. - generate-docs workflow: `github.event.inputs.*` was interpolated directly into a `run:` block; the values now pass through `env:`. Packaging: - The docagent Markdown templates were dropped from the wheel, so `get_template()` would raise FileNotFoundError once installed from PyPI. Added the corresponding package-data entry. Test isolation: - MermaidRenderer's "auto" backend probes `npx @mermaid-js/mermaid-cli`, which downloads mermaid-cli and Chromium on a cold machine. The suite was making real network calls despite a CI comment claiming otherwise (the env var it set, PYTHONDONTWRITEBYTECODE, is unrelated to networking). Added an OPENDOCS_MERMAID_BACKEND override with a "none" value, cached the probe, and set the override in CI. Suite time drops from 4m30s to 8s. Docs: - Corrected the stale v0.5.0 heading, the inconsistent format counts, the "self-contained" claim for a page that loads vis.js from a CDN, and the `--branch` help text, which described a prefix as a base branch. Tests: 349 pass (up from 294). cli.py and watcher.py previously had no coverage at all, which is where most of these defects lived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
The generated graph loaded vis-network from unpkg.com, so the page did not render at all without network access — despite the README calling it "self-contained". Adds `--embed-graph-assets`, which inlines the library into the HTML. The bundle is resolved cheapest-first: an `OPENDOCS_VIS_NETWORK_JS` path (for air-gapped builds that cannot download anything), then a local cache, then a one-time download that is written to the cache. If none succeed it falls back to the CDN tag rather than emitting a page that cannot work. Inlined library source is escaped so a `</script` sequence inside it cannot terminate the script element early. Verified in Chromium with all network requests blocked: the embedded page renders 32 nodes, a canvas, and 21 panels with zero JS errors, while the CDN page fails with "vis is not defined". Cost is ~28 KB -> ~716 KB. Also documents OPENDOCS_MERMAID_BACKEND, OPENDOCS_VIS_NETWORK_JS and OPENDOCS_CACHE_DIR in a new offline/air-gapped section of the README, and makes generate_interactive_graph create its output directory. Tests: 364 pass (15 new). None of them touch the network. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
The README already promised that graph.json lets you "query weeks later
without re-processing", but nothing could actually read the file back. This
adds the missing half.
`opendocs.core.graph_query.GraphQuery` loads an exported graph and answers
structural questions entirely offline — no LLM, no API key, no network:
--stats / --list-types / --god-nodes / --questions
--search / --entity / --type / --community / --provenance
--neighbors / --dependents / --dependencies
--path A B shortest connection, traversing edges either direction
Plain-English questions are routed onto those same primitives by shape
("what depends on X", "how are X and Y connected", "which databases"), so
answers stay reproducible and need no model. Unrecognised questions fall back
to a name search rather than failing.
Resolution is forgiving where it helps and strict where it matters: a term is
matched by id, then exact name, then a unique search hit, and an ambiguous
term reports its candidates instead of silently picking one.
Scriptable: --json for machine-readable output, and exit codes 0 success,
1 query matched nothing, 2 graph file unreadable.
Tests: 408 pass (44 new), including a round trip that generates a graph with
the real exporter and queries it back, so the export format and the query
layer cannot drift apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
A CI-friendly linter over the same parsed DocumentModel the generators consume, so it sees exactly what will be rendered. Exits non-zero when documentation regresses, letting a PR fail on a broken README the way it fails on a broken test. Twelve rules across three severities: missing title or prose, unreplaced template placeholders, absent conventional sections, thin content, unfinished markers, ragged tables, images without alt text, heading-level jumps, duplicate headings, and unlabelled code fences. `--check-links` adds link checking and is the only rule that touches the network. Two rules needed more care than they first appeared: - ragged-table works from the raw source, not the parsed blocks. Mistune rejects a table whose rows do not match its header, so the block never reaches the model and a rule over TableBlock could never fire. The failure it catches is exactly that: the table silently renders as plain text. - Markers and placeholders treat code inversely. A `TODO` in code is documentation *about* markers, so fenced and inline code are both stripped before matching. A placeholder in a fenced block is the opposite — that is the command a reader copies and runs — so only inline spans are stripped there. Without this split, any README documenting the linter fails it. Wired into the Quality workflow as a separate job that gates this repo's own README, with README.md added to the workflow path filters so a docs-only change actually triggers it. Link checking is left off in CI so a third-party outage cannot fail the build. Scriptable: --fail-on error|warning|info|never, --json, and exit codes 0 clean, 1 findings at or above the threshold, 2 source unreadable. Tests: 473 pass (65 new), including a dogfooding test asserting this repo's README stays clean, and one asserting lint makes no network call without --check-links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
Reports what changed between two versions of documentation — sections, concepts, and relationships — and renders the delta as release notes. Deterministic and offline. Sources can be two files, two exported graph.json files, or two git revisions of the same path (`--git . --path README.md`), which is the "what changed since the last release" case. Output as a terminal summary, Markdown release notes, or JSON; `--fail-on-change` turns it into a CI drift gate. The summary also reports which output formats the change actually invalidates, so a docs pipeline can rebuild only those. Built on core.doc_diff rather than the agents/ package. That package was the obvious home, but it does not work: DiffAgent._compute_diff returns an empty DiffSummary regardless of input, RegenerationAgent._regenerate regenerates nothing, and ImpactAgent indexes entities by a source_file property the extractor never sets. Measured against three real commits, git reports 7 changed files while the agent chain reports 0, with 0 of 35 entities carrying a file path. Exposing it would have shipped commands that silently do nothing, so its docstring now states that plainly and points at the working module. One implementation bug worth noting, caught while testing against real history: naming each snapshot after its git ref made the PROJECT entity — which every other entity links to — appear added on one side and removed on the other, inventing 21 phantom relation changes for a two-section edit. Display label and parsed identity are now separate, and a regression test asserts no ref labels leak into entity names. Tests: 508 pass (35 new), including diffs across real git revisions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
Generated artifacts are cached by content fingerprint, so re-running the pipeline rebuilds only what changed. `opendocs watch` is the main beneficiary — it re-ran every format on every save, and most formats are usually unaffected by a given edit. Measured on a 260 KB document: a warm single-format build drops from 4.0s to 1.1s, and the restored artifact is byte-identical to a freshly generated one. Correctness rests on the fingerprint covering every input, since a missed input means silently serving a stale artifact. The key folds in the document content, the knowledge graph, the theme, the template variables, the output format, the sort/mode options, and the installed opendocs version. Most of the new tests exercise the *miss* path for exactly this reason. Two details worth recording: - metadata.generated_at is a fresh timestamp on every parse, so including it made the key unique per run: the cache never hit and grew an entry per build. Volatile metadata is now excluded, which is what lets identical inputs produce identical output. The trade-off is that a hit keeps the timestamp from when the artifact was first built. - Restoring uses the filename recorded in the manifest rather than asking the generator where its output would go. Generators compute their own paths inside generate(), so the alternative was adding a path-prediction method to all twelve of them. Damage never breaks a build: a corrupt entry, unreadable manifest, or failed write degrades to a miss. Adds `opendocs cache` to inspect or clear the store, plus --no-cache and --cache-dir on generate. Tests: 540 pass (32 new), including an end-to-end check that a changed source is not served from cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A repository review, the fixes it produced, and four features built on top.
540 tests pass, up from 294.
ruff checkandruff format --checkclean. Verified in both network and offline modes. Every fix has a regression test that was confirmed to fail against the previous code before the fix was applied.Part 1 — Fixes
Correctness
opendocs inspect <notebook.ipynb>crashed.namewas only assigned on the Markdown branch, so the notebook path raisedUnboundLocalError.-f mindmapwas rejected. It was inFORMAT_MAPand in--format all, but missing from the clickChoice. The choice list is now derived fromFORMAT_MAP; it was duplicated across three commands, which is how it drifted.--no-outputsdid nothing. Parsed but never forwarded, so notebook cell outputs were always included._escapenever escaped the backslash, so any README containing one produced a.texthat would not compile —50% \& xbecame50\% \\& x, where\\is a line break._create_docs_prstashes the working tree but returned without popping on three failure paths, including the common failed-push case. It also popped unconditionally, discarding a stash the user had made themselves when there was nothing of ours to stash — a bug the tests surfaced that reading had missed.main, breaking any repo using another default. Now resolved from the remote.graph.jsonchanged between runs on identical input.Injection
The interactive graph executed untrusted markup. Entity names are not trusted input — mermaid node labels are lifted verbatim from README content.
json.dumpsdoes not escape/, so a name containing</script>closed the script element early, and four panels then wrote names straight toinnerHTML. Reproducible end-to-end in basic mode with no API key.Verified fixed in Chromium: the payload neither executes nor enters the DOM, and renders as escaped text while the page stays functional.
generate-docs.ymlinterpolatedgithub.event.inputs.*into arun:block. Values now pass throughenv:.Packaging
package-dataentry, soget_template()would raiseFileNotFoundErroron any PyPI install. Confirmed by building the wheel before and after.Test isolation
MermaidRenderer'sautobackend probesnpx @mermaid-js/mermaid-cli, downloading mermaid-cli and Chromium on a cold runner. The suite made real network calls despite a CI comment claiming otherwise — the variable it set,PYTHONDONTWRITEBYTECODE, is unrelated to networking.Added an
OPENDOCS_MERMAID_BACKENDoverride with anonevalue, cached the probe, and set it in CI. Suite time: 4m30s → 8s.Part 2 — Features
opendocs query— explore an exported graphThe README promised
graph.jsoncould be queried "weeks later without re-processing", but nothing could read it back.Plain-English questions are routed onto structural primitives by shape, so answers are reproducible and need no model. Offline, no API key.
opendocs lint— documentation quality gateTwelve rules over the same parsed model the generators consume. Wired into the Quality workflow, gating this repo's own README.
Two rules needed more care than they first appeared:
ragged-tableworks from raw source, because mistune drops a malformed table entirely — a rule overTableBlockcould never fire. The failure it catches is exactly that silent drop.TODOin code is documentation about markers; a placeholder in a fenced block is the command a reader copies and runs. Without this split, any README documenting the linter fails it.opendocs diff— what changed since the last releaseCompares two files, two
graph.jsonexports, or two git revisions, and renders release notes.This is not built on
agents/. That package looked like the obvious home, but it does not work:DiffAgent._compute_diffreturns an empty summary regardless of input,RegenerationAgent._regenerateregenerates nothing, andImpactAgentindexes entities by a property the extractor never sets. Measured against three real commits: git reports 7 changed files, the agent chain reports 0, with 0 of 35 entities carrying a file path. Exposing it would have shipped commands that silently do nothing, so its docstring now says so and points at the working module.Incremental build cache
Artifacts cached by content fingerprint;
opendocs watchwas the motivating case. On a 260 KB document a warm build drops 4.0s → 1.1s, byte-identical to fresh output.Correctness rests on the fingerprint covering every input, so most of the new tests exercise the miss path. One detail worth recording:
metadata.generated_atis a fresh timestamp per parse, so including it made the key unique per run — the cache never hit and grew an entry per build.Test coverage
cli.py(993 lines) andwatcher.py(508 lines) previously had no tests at all, which is where most of the defects lived. Added seven test modules; none touch the network.