Repository navigation
feat: add opendocs coverage — report what the docs don't cover - #4
Merged
Merged
Conversation
`opendocs lint` judges the quality of what documentation says. This reports what it misses, by comparing the real API surface — which code_analyzer already derives — against what the prose actually mentions. Four dimensions are scored, each picked because it has an objective answer: docstrings on public symbols, environment variables the code reads, CLI flags it defines, and detected technologies. Any judgement of whether prose is *good* is deliberately excluded; that is not objectively measurable, and a score nobody can verify is worse than no score. Two decisions determine whether the number can be trusted: Test files are excluded. Counting them made the metric actively misleading — on this repo, test classes were most of the "undocumented" classes, reporting 61.4% where the shipped code was at 98.9%. CodebaseModel has a test_files field but the analyser never populates it, so classification is by path convention, with --include-tests to opt back in. Private helpers are excluded for the same reason, since they carry no promise to a reader. The overall figure is weighted by item count rather than averaging the per-dimension percentages, so one missed flag out of two cannot outweigh hundreds of documented symbols. Dimensions with nothing to measure are left out entirely rather than banking a vacuous 100%. Dogfooded: run against this repo it found one undocumented function, two undocumented classes, and three undocumented environment variables (AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_VERSION, OPENDOCS_MODEL_CACHE), all genuine. Fixing them takes docstrings and env-vars to 100% and the overall figure from 86.3% to 89.2%. The remaining gaps — 30 undocumented CLI flags and 10 undocumented technologies — are real and left for a follow-up. Scriptable: --fail-under for CI, --json, --show-missing, and exit codes 0 pass, 1 below threshold, 2 analysis failed. Tests: 671 pass (57 new), weighted mostly toward what must *not* be counted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2mrMSRs3cHBdPgHbRzVDX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
671 tests pass (was 614),
ruff checkandruff format --checkclean.opendocs lintjudges the quality of what documentation says. This reports what it misses, by comparing the real API surface — whichcode_analyzeralready derives — against what the prose actually mentions. No other docs tool answers this question, and both halves already existed in the repo.Two decisions that determine whether the number is trustworthy
Test files are excluded. Counting them made the metric actively misleading — on this repo, test classes were most of the "undocumented" classes, reporting 61.4% where the shipped code was actually at 98.9%. Nobody documents
TestMicroservicesAgent.CodebaseModelhas atest_filesfield but the analyser never populates it, so classification is by path convention, with--include-teststo opt back in. Private helpers are excluded for the same reason — they carry no promise to a reader.The overall figure is weighted by item count, not an average of per-dimension percentages, so one missed flag out of two cannot outweigh hundreds of documented symbols. Dimensions with nothing to measure are dropped entirely rather than banking a vacuous 100%.
Scope
Four dimensions, each chosen because it has an objective answer: docstrings on public symbols, environment variables the code reads, CLI flags it defines, and detected technologies.
Deliberately not scored: whether prose is any good. That isn't objectively measurable, and a score nobody can verify is worse than no score at all.
Dogfooding
Run against this repo it found real gaps — one undocumented function, two undocumented classes, and three undocumented environment variables (
AZURE_OPENAI_ENDPOINT,AZURE_OPENAI_API_VERSION,OPENDOCS_MODEL_CACHE). All genuine; all fixed in this PR.The remaining gaps — 30 undocumented CLI flags and 10 undocumented technologies — are real README gaps, left for a follow-up rather than padded out here.
Testing
57 new tests, weighted toward what must not be counted: test paths, private helpers, standard environment variables, and flags merely mentioned in help text rather than declared. Everything runs offline.
Generated by Claude Code