Documentation Quality Agent

An AI agent that audits developer documentation against its reference repository using Claude's tool use, flagging code-related claims that no longer match the code.

What It Does

Given a Markdown doc and a reference repository, the agent reads the doc, uses Claude to identify checkable code-related claims, searches and reads the actual source to verify them, and reports every claim that's inaccurate: what the doc says, what the code actually shows, and a file/line reference.

Installation

git clone https://github.com/eshaben/documentation-quality-agent.git
cd documentation-quality-agent
python3 -m venv venv
source venv/bin/activate
pip install -e .

Setup

Copy .env.example to .env and add your Claude API key:

cp .env.example .env

ANTHROPIC_API_KEY=your-key-here

Usage

docs-audit -p path/to/doc.md -r https://github.com/org/repo.git

• -p / --path: the Markdown doc to check
• -r / --repository: the GitHub URL of the reference repo to check it against

The repository is cloned into a temporary directory for the duration of the audit after which it is automatically deleted.

Example

docs-audit -p examples/anthropic_sdk_doc.md -r https://github.com/anthropics/anthropic-sdk-python.git

examples/anthropic_sdk_doc.md has three inaccuracies planted on purpose, to give the agent something real to catch:

• It claims max_tokens defaults to 1024 if omitted. It's actually a required argument, with no default.
• It claims failed requests are retried up to 5 times by default. The SDK's actual default is 2 retries.
• It claims streaming uses client.messages.stream_response(). The real method is client.messages.stream().

How It Works

The agent (agent.py) gives Claude two tools: search_code, which searches the repo via git grep, and read_file, which reads a specific doc file. Claude reads the doc, decides what's worth checking, calls those tools as many times as it needs across an agentic loop, and produces a final report.

Requirements

• Python 3.12+
• git
• A Claude API key