Install uv and use the committed lockfile:
uv sync --frozen
uv run ruff format .
uv run ruff format --check .
uv run ruff check .
uv run mypy src/ocr_toolkit
uv run bandit -r src/ocr_toolkit --severity-level medium --confidence-level medium
uv run pytest --cov=ocr_toolkit --cov-report=term-missing --cov-fail-under=85
uv run python -m build
uv run twine check dist/*For routine agent and contributor checks, run focused tests for each logical change and scripts/quality.sh check once on the completed handoff head. After changing Python, run scripts/quality.sh format before self-review so Ruff's formatter is applied immediately; before every Python commit, require a clean repository-wide uv run --frozen ruff format --check . result. The full wrapper remains the final owner and repeats that check, so a formatting failure discovered only after push is a missed local commit gate rather than a missing quality control. It replaces the selected mode's prior log, captures current output under ignored .quality-logs/, and prints only a short status; on failure it prints the last 80 lines. Individual modes are format, lint, test, coverage, types, and security. The coverage and check modes reuse that single branch-aware test run, then enforce scoped floors for result/preflight and GitLab posting transactions at 80%, plus review/context/DLP/approval and MCP/provider/policy/result contracts at 85%; a high combined result cannot hide a weak risk group. Hosted pull requests still run the complete suite on all five supported OS/Python combinations. The three Linux jobs are release-blocking, with Ubuntu on the newest supported Python as the sole coverage owner. The two macOS endpoint jobs are best-effort compatibility diagnostics: they continue to run and remain visible for follow-up, but a platform-only failure does not block a Linux-priority release. The Bandit gate scans only the supported runtime package at medium-or-higher severity and confidence; tests and synthetic fixtures are intentionally outside that bounded gate.
Runtime code must remain compatible with the Python range and runtime dependencies declared in pyproject.toml; uv.lock owns the resolved dependency set. Tests must use synthetic data; public examples must use safe placeholder hosts and credentials while describing the real operating behavior rather than labelling the feature itself as synthetic. User-visible changes require a fragment in changelog.d/.
Repository-only qualification tools and evidence live under scripts/ and compatibility/; they are excluded from both published distributions. Validate the manifest with PYTHONPATH=src python scripts/ocr_compat.py validate.
For package, executable-integration, or release-machinery changes, install the wheel and sdist into separate temporary virtual environments, run pip check, and exercise the changed installed runtime entry point outside the checkout with repository paths removed from PYTHONPATH. ocr-ci --help proves console entry-point availability only; it does not prove dependency completeness or runtime execution. The official recipe and registry smoke share the hash-locked binary runtime installation and installed MCP/schema checks; dependency metadata changes must preserve that clean-install contract. Ordinary runtime changes rely on the pull request's single Build artifacts owner instead of rebuilding packages inside both CI jobs. Repeat deterministic builds locally only when reproducibility or release machinery is in scope. Before every commit, stage the intended files and run scripts/gitleaks.sh --staged to scan the exact candidate index for secrets and concrete local paths. Before every push, run scripts/gitleaks.sh --tree and the history gate below. The tree mode includes tracked and non-ignored untracked files without following symlinks; neither mode replaces the history scan. Failed or incomplete scanning blocks the commit or push. Generic secret scanning uses Gitleaks; dependency auditing uses pip-audit. Install the exact Gitleaks version printed by scripts/gitleaks.sh --version, then run scripts/gitleaks.sh before pushing and scripts/quality.sh check for the Python quality matrix. The wrapper fails closed when the scanner version or base ref is unavailable, scans the complete first-parent feature history, and is also the single source for the hosted security job's version pin. TestPyPI and stable-release workflows do not duplicate that dedicated security job.
tests/test_installed_policy_e2e.py builds both the direct wheel path and the sdist-to-wheel path, installs each into a clean environment, and exercises target decisions and nested guidance through the real stdio MCP. It runs with a hostile repository shadow package, restricted PATH, owner-only artifacts, and the installed console entry point; keep package-boundary changes inside that test rather than replacing it with editable-install mocks.
GitHub Actions storage is repository-owned infrastructure. Pull-request CI restores setup-uv caches but does not save branch-specific entries; protected-main publication may refresh shared dependency state. CodeQL TRAP caching and the separately controlled v4 overlay-database mode are disabled, so the small repository receives a full analysis without per-run CodeQL cache writes. Workflow artifacts use a seven-day handoff window. The weekly Actions storage maintenance workflow grants actions: write only to its cleanup job and deletes all CodeQL caches, non-main or superseded setup-uv caches, superseded Gitleaks caches, artifacts older than seven days, TestPyPI preview logs and runs after seven days, ordinary and TestPyPI development logs and completed runs after 14 days, and stable Release logs and completed runs after 30 days. Deleting a run removes that run's metadata, logs, and check metadata, so a workflow run is never removed before its separately promised log window. Active and newer runs remain untouched. The scheduled collector reads a closed 44-day UTC window in daily shards, retaining a fail-closed ten-page limit per day instead of applying that limit to the aggregate run history. Scheduled log cleanup uses a bounded 14-day retry window so immutable run history does not get scanned and retried forever. Manual dispatch is a dry run unless execute is selected; the same plan is available locally with python scripts/actions_cleanup.py, requires --execute for deletion, and accepts --include-all-old-logs for a deliberate one-time historical log cleanup.
For documentation-only or bookkeeping changes that do not alter runtime, packaging, dependencies, or CI behavior, run the existing contracts for the affected documents and review the instructions against representative tasks. The full local Python matrix is not required for that scope. Preserve applicable results for unchanged inputs; rerun affected checks after a material edit or failure. Hosted checks and the pinned pre-push secret gate remain unchanged.
During an authorized implementation task, run the scoped local tests that use synthetic fixtures and temporary resources, fix failures caused by the change, and rerun affected tests without asking for approval at each step. This does not authorize live provider calls, OCR execution, production access, or publication; follow the user's current constraints for those actions.
Current operator guidance describes the qualified OCR contract without repeating
release numbers or asset hashes; link to compatibility/ocr-support.json for the
exact recommendation of the same toolkit revision. Keep exact versions in
executable pins, binary-identity checks, release-specific plans/notes and historical
evidence. Generic fixtures derive the current identity from its runtime owner;
frozen historical fixtures must not follow the current pin. A version-neutral
description still needs semantic review when the consumed contract changes.
The live suite in scripts/ocr_compat.py runs the same forward-only probes for
every candidate, then validates observations against the contract epoch owned by
that candidate. Do not add old parser fallbacks or patch-specific fixture copies
to live probes. Version checks belong to binary identity, release ordering,
classification, promotion, and the frozen-versus-current contract boundary—not
to choosing which behavioral checks execute. Change a probe only when a consumed
upstream contract actually changes, and document the concrete before/after
behavior.
scripts/ocr_compat_history.py owns frozen validation of evidence from before the
forward-only suite boundary. It never launches OCR and must not depend on current
numeric defaults, language inventories or live fixtures. Preserve historical
evidence bytes. Generic promotion tests use a frozen baseline; only current-pin
tests assert the latest supported version. Current promotion validates all required
contract evidence before writing any pin or evidence file.
When a consumed upstream contract changes, first freeze the last unaffected
evidence epoch exactly, then advance the cutoff. The same version-selected
validator must be used by manifest readback, prepare-update, and the live
runner; a candidate cannot bypass a newly required probe merely because an
older release remains readable. Evidence through OCR 1.12.7 is frozen in its
original epoch. From OCR 1.12.8, the current contract adds F# signature/script
selection and its built-in Rules pattern, plus expanded dependency and build-output
exclusions. The Rego and other previously consumed language contracts remain
required.
Use exact-tag dispatch for one candidate. When several unseen releases must be
qualified as one adjacent chain, set through_tag to the authorized upper bound;
never use an unbounded discovery run for a release-scoped task. Manual dispatch remains available.
Qualification uses one existing deterministic gateway and bounded observations. Do not copy OCR's implementation, add another configuration framework, or turn unconsumed upstream wording/features into mandatory checks. Prove preservation with real no-LLM OCR runs and adversarial verifier tests. Separate those proofs from model-quality qualification in a configured external environment.
PLANS.md contains active or blocked repository work. For a small change without runtime or trust-boundary impact, a short entry with scope, release classification, validation, and completion or resume state is sufficient; omit inapplicable fields. Multi-step implementation, release work, and trust-boundary changes need the relevant decisions, service boundaries, trust inputs, ordered work, validation, and recovery state. Read-only work needs no entry. Reuse current context and update only changed plan facts.
Before a logical commit, update the plan and directly affected status-bearing documents to describe post-commit truth. A milestone closes only after current implementation and tests prove its own outcome; update roadmap, backlog, strategy, or README only when their recorded facts change. For completed no-release work, remove the entry at handoff once its outcome and validation are retained in the commit or PR; an open PR alone does not imply pending implementation when PR creation is the requested endpoint. Keep genuinely unfinished work with its exact resume action. The stable-release archive remains owned by docs/release.md; generic skills do not introduce another plan schema, archive, or closure gate.
Historical wording is evidence, not a specification. Rebuild a capability matrix from current code, tests, and published behavior before retaining backlog work or dependency edges. Keep implementation, safety, and rollout dependencies distinct so conditional future work does not block an independently safe capability. Completed stable plans follow the archive lifecycle owned by the release guide.
When a failure recurs, classify its cause before changing guidance: add or repair the canonical requirement when it is missing or conflicting, correct startup selection when it was not loaded, and otherwise add or repair the concrete subsystem control. Add a pitfalls entry only for a distinct reusable incident class with historical evidence; the catalogue itself does not own the correction.
The root maintains current intent, accepted decisions, implementation/integration state, blockers and the next safe action. Delegate only independent bounded work with a self-contained packet and accessible evidence; keep tightly coupled changes with one owner. Workers return focused results and artifact references; only root updates shared plan/backlog state. After context loss, recover from the current plan, Git state and relevant durable artifacts rather than reconstructing transcripts or repeating side effects. Preserve full private logs only when needed for diagnosis or durable validation; transient waiter output is not a substitute for required artifacts.
Before a search or diagnostic call, identify the decision it will inform and select the smallest useful output: filenames for discovery, focused ranges for inspection, or status and counts for a structured report. Inspect headings before loading a large owner document. Read additional sections when the changed boundary requires them; a route to a document does not require repeatedly loading its entire contents.
For potentially large audit, export, build, and test output, preserve the complete result in an ignored private artifact and return a bounded summary with exit status, errors, truncation state, and the artifact location. Reduce known JSON schemas before sending them to the model; do not print a full report and then wrap or serialize that same output again. If a result is truncated, narrow the next query or inspect a relevant range of the preserved artifact instead of repeating the broad call with a larger output limit. Inventory tracked and non-ignored source files; keep ignored logs, caches, virtual environments, and generated dependency trees out of ordinary source discovery. Explicitly scoped incident diagnosis may inspect an ignored artifact.
Reuse evidence only while its relevant inputs remain unchanged. Repeat a successful check after an affected code, dependency, environment, or contract change, a prior incomplete result, or a required release/publication boundary; state that reason. Full quality checks remain required for implementation/release scope, while documentation-only checks follow the scoped rule above. Avoid adding bookkeeping files, wrapper scripts, or prose-presence tests merely to enforce an agent habit that bounded semantic review can assess.
When auditing session efficiency, distinguish tool-call counts, output volume, cached input, and fresh input. Do not sum overlapping resumed histories or interpret cumulative token counters as subscription charges. Repeated calls require contextual review before being classified as unnecessary. Keep raw sessions and private identifiers out of public PRs and issues.
For a PR-only task, finish the scoped local checks, review security-relevant changes and run the pinned secret gate before push, create the PR, and read its initial check state once. Report pending CI honestly. Wait for completion when merge, release, CI repair, or explicit monitoring is part of the task; otherwise PR creation is the handoff boundary. When waiting is needed, keep repeated watch output in the execution layer and return only meaningful changes or the terminal result.
Wait for a long-running local command through one completion-driven waiter when the execution environment exposes a persistent process or session identity. The waiter stays attached until exit, cancellation, or the command's overall deadline and returns as soon as that terminal state occurs. Internal infrastructure waits must not yield periodic empty observations back to the model: repeated model-driven status, terminal, or process polls add tool results and task history without changing a decision. The waiter is a completion signal, never the only copy of command output or a required result: its transport or model-facing output can be truncated, discarded after completion, or lost with the session.
Keep the wait bounded and diagnosable:
- assign an overall command deadline and preserve a cancellation path; terminate and clean up owned child processes when either is reached;
- before starting the process, choose explicit ignored owner-only paths outside any transient directory that the command cleans up, remove or reject stale owned artifacts, and have the process atomically publish every result required for later validation;
- redirect potentially large stdout and stderr to ignored private files and preserve them until their focused inspection is complete; return from the waiter only the exit status, elapsed time, artifact paths, digests, and a bounded diagnostic tail or structured summary;
- validate the persisted artifact's type, ownership/permissions, size, completion marker or exact schema, and digest before trusting it; a missing or invalid artifact is a failed/inconclusive run, not permission to reconstruct the result from a truncated waiter transcript;
- retain enough process or session identity to distinguish completion from a lost waiter, but do not expose credentials, inherited environment, unbounded logs, or result bodies through the waiter;
- let the outer wait return immediately on process completion even when its maximum deadline is much longer; and
- inspect full logs only after a concrete failure makes a focused range relevant.
Use periodic polling only when no completion notification or persistent waiter exists, when the process can require interactive input, or when intermediate state can change an authorized operational decision. In that fallback, choose an interval proportionate to expected duration, suppress unchanged observations, and increase the interval for stable work. Delete preserved private artifacts only after the relevant evidence has been extracted and verified. This discipline reduces redundant model turns and context growth; it does not waive required monitoring, validation evidence, timeouts, cleanup, or the cost of analyzing the eventual result, and it makes no exact subscription-billing claim.
Select checks from the changed boundary rather than from an ever-growing generic prohibition list. Start with the narrowest reproducer, then run the applicable contract tests and the complete quality gate before release handoff. In particular:
- parser changes exercise the semantic grammar and bounded degradation;
- repository, persistence, subprocess, network, provider-write, and report changes use the cross-cutting trust invariants and the checklist below;
- package or executable-integration changes include clean wheel and sdist validation rather than mocks alone;
- public-source changes keep private audit material untracked and run the pinned complete-range Gitleaks wrapper before push; and
- release changes run the release authorization, receipt, workflow, artifact, and documentation suites owned by
docs/release.md. - OCR pin changes run production-equivalent behavioral probes for every numeric CLI option the toolkit example passes. Cover omitted/default, each sentinel, invalid-below-boundary, minimum minus one, minimum, a representative value, and maximum plus one when a maximum exists. Record only closed exit, diagnostic, normalization, ownership, and effective-value facts; help text alone is not compatibility evidence.
Safe bounded read-only diagnostics are allowed. A boundary rule prohibits the unsafe acquisition, trust transition, or mutation mechanism, not HTTP, subprocesses, provider APIs, file cleanup, or debugging as whole categories.
For every boundary or integration claim, write down the production owner, the entry point exercised, the observable result, and any external collaborator replaced by a test double. The double must sit beyond the claimed boundary: do not mock the Git reader to prove Git isolation, the HTTP adapter to prove redirect or byte limits, the store loader to prove hostile readback, the subprocess launcher to prove argv or descriptor behavior, or the MCP dispatcher to prove stdio protocol behavior. A mocked-owner test may prove orchestration only and must be paired with a production-path test before the broader claim is accepted. Prefer real temporary repositories, local HTTP peers, child processes, persisted files, stdio clients, and clean installed wheel/sdist environments. Make hostile cases traverse the same owner and assert the intended rejection branch rather than an earlier mock-selected failure. The maintained test evidence matrix records those owners, entry points, external qualifications, and non-claims across the complete suite. Update it when a new boundary claim is introduced or when a test double moves across an existing owner.
Treat one confirmed boundary or parser defect as a risk class: inspect sibling implementations, make negative tests reach the intended rejection or degradation branch, and assert that contract rather than an unrelated earlier failure. Before implementing a new parser or trust boundary, record its grammar, normalization, degradation, budget units, inherited-process state, and adversarial fixtures in the active plan or focused tests.
New runtime modules, classes, and functions need purpose-focused docstrings. Comments at non-obvious security, compatibility, ownership, and state-transition boundaries explain why the constraint exists rather than narrating the code. Do not add legacy namespace shims or historical integrations outside the public contract.
Apply the cohesive-module invariant during self-review. Prefer an extract-and-delegate refactor that moves already characterized functions or classes intact, preserves the intentional package facade, and reruns the same contract suite before and after each move. Split on distinct responsibility and dependency direction, not an arbitrary line count; do not rewrite a working algorithm merely to make a file shorter. Architecture tests should protect required owners and forbidden upward dependencies without freezing every future helper-module name.
reporting/ owns provider-neutral report data and pure presentation: model.py
snapshots admitted results and execution-owner facts, outcome.py describes
review health separately from finding delivery, result.py normalizes coverage
and warnings, and metadata.py, usage.py, dlp.py, sections.py, and text.py
own their respective pure calculations. Shared reporting must not import posting,
forge acquisition, forge receipts, environment configuration or provider actions.
Posting facades preserve existing imports while GitLab retains its publication
transactions, discussion anchors, suppression and approval policy.
providers/local.py publishes private Markdown artifacts and console output from that shared report, with every
admitted finding and no posting cap, remote badge or HTML disclosure. It does not
acquire forge data or accept discussion commands. The runner, not the formatter,
must establish immutable identity, mandatory completed evidence use and DLP
admission before constructing a successful report. Passing a model-authored
receipt to a formatter is not evidence of successful execution.
New shared reporting tests live in tests/reporting/; keep GitLab transaction
regressions with their existing owners. Dependency tests enforce the downward
boundary, while outcome and rendering tests prove the pure report contract.
Provider I/O and executable CLI claims require separate production-boundary tests.
Normalized source adapters live under src/ocr_toolkit/evidence/ecosystems/. Shared parser result contracts belong in ecosystems/contracts.py; Python, JavaScript, Go, and PHP package metadata each have one adapter module. Ansible keeps Galaxy requirements and topology/inventory analysis as separate modules under ecosystems/ansible/. These adapters consume text or already bounded metadata and return normalized facts: they do not own Git or filesystem reads, subprocesses, network access, framework derivation, persistence, or MCP lifecycle.
The evidence/collectors/ package is the bounded immutable acquisition boundary. registry.py owns path-to-adapter registration, sources.py owns small cross-ecosystem CI/container source projections, graphs.py owns local include-graph reads, projections.py owns record/coverage/delta projection, and orchestration.py coordinates one immutable ref. Pure helper modules must not import orchestration, persistence, MCP, or higher policy/framework lifecycles. The package __init__.py is the intentional runtime facade; do not recreate a flat compatibility module.
Do not add a flat compatibility module when moving or adding an adapter. Parser changes need semantic-variant fixtures, explicit item/include bounds, malformed-input behavior, redaction checks, and collector/delta/MCP coverage where applicable. A new framework that interprets those normalized facts belongs in evidence/frameworks/, not in the source adapter.
Pure policy contracts, accepted-decision parsing, safe scope matching, and guidance applicability live under src/ocr_toolkit/evidence/policy/. Register providers statically; do not use entry points or repository-controlled imports. Policy code consumes bounded immutable text and normalized changed paths only. Git/tree/blob reads remain in evidence.collectors, compact hints remain in evidence.project, and transport remains in the single built-in evidence MCP.
The evidence/store/ package is the persistence boundary. contracts.py owns versions, kinds, limits, and errors; values.py owns recursive redaction and value normalization; core.py owns in-memory admission, ordering, and serialization; atomic.py owns owner-only replacement; and readback.py owns hostile envelope decoding and cross-reference reconstruction. EvidenceStore.read() and write() stay thin delegates, and the package facade preserves the supported EvidenceStore, EvidenceStoreError, and EvidenceStoreLimits imports. Readback must not import the concrete core implementation back or bypass its admission and policy-binding controls.
Parser changes need legacy-format, duplicate-ID, malformed-field, unknown-field, scope, date, applicability, precedence, rename, unsafe-object, multibyte-boundary, and redaction fixtures. New policy values require exact kind-specific persisted schemas, snapshot/provenance correlation, and impossible-state rejection. Select applicable guidance before content reads and isolate policy truncation from unrelated evidence domains. Repository guidance is untrusted evidence and must never become executable instructions or an authorization channel; bootstrap renderers must use shared delimiter-aware Markdown helpers for repository-derived values.
src/ocr_toolkit/context/ is independent from ocr_toolkit.evidence: contracts and normalization point downward; protected policy and fixed recognizers are pure; broker.py composes forge-context authorization, DLP, limits, and admission; store.py owns separate atomic hostile-read persistence and handle binding; mcp.py serves only the committed local store. providers/gitlab_discussions.py owns forge pagination, stable repeated snapshots, provider account classification, and run-local pseudonyms. Optional model-directed external tools belong to the separate governed federation/ package, never the context store. review_runner.py is the sole orchestration owner that combines repository evidence, bounded forge context and federation into one OCR execution.
A new forge-context source must keep provider acquisition before OCR and enter the same broker/store/handle/MCP/publication/cleanup lifecycle. Do not restore executable adapters or policy-selected endpoints. A model-directed external read belongs in operator registry v2 behind the federation gateway, with an explicit allowlist, schema/origin/DLP/budget controls and content-free accounting. Neither path may add arbitrary link retrieval, a second store budget charged to evidence, or a second OCR/model pass. Schema discriminators on policy, store, registry, status and receipt boundaries prevent cross-contract interpretation; ephemeral M5/M7 state has no migration path.
Boundary evidence must cross the production owner with a real Git repository, child process, local TLS peer, atomic file, stdio MCP client, installed artifact, or actual OCR as appropriate. Keep unavailable outcomes uniform across denial/not-found/foreign-tenant cases, and preserve explicit non-claims for adapter truth, broader service credentials, host compromise, semantic paraphrase, and model judgment. Update the public bounded-context contract, threat model, test-evidence matrix, strategy, roadmap status, and Towncrier fragments whenever this lifecycle changes.
Framework support lives under src/ocr_toolkit/evidence/frameworks/. Add an ecosystem declaration under frameworks/providers/ and register it explicitly in frameworks/registry.py; keep Jinja2 first in the bounded priority order. Reuse the generic package detector where its direct-declaration and resolution semantics fit. Extend the closed schema and generic detector deliberately when a demonstrated provider needs different normalized semantics. Do not add entry-point discovery, compatibility shims, repository reads, filesystem access, subprocesses, network calls, mutation, or another MCP lifecycle to this package. Git/tree/manifest collection, storage, and serving remain core-owned boundaries.
Every provider change needs synthetic tests for direct activation, lock-only non-activation, component ownership, malformed and bounded source degradation, fact/configuration limits, schema reload, base/head deltas, and the existing MCP projection as applicable. Template engines also need explicit OCR include/rule fixtures because evidence collection does not alter OCR file selection. Update the strategy and changelog when the supported public behavior changes.
Provider results are admitted atomically: facts, coverage observations, and notices must all satisfy their package limits and immutable contracts before any of them reach shared registry output. Use . for a declaration manifest at the repository root; never overload a valid path such as repository as a root sentinel. Keep identifier/path bounds separate from longer manifest-derived scalar bounds, and validate plugin records only after the store applies its persistence redaction and total-value budget.
Before closing a parser, repository reader, persisted schema, network helper, provider mutation, subprocess, or report-rendering change, apply the canonical trust-boundary invariants and add the applicable tests:
- exercise byte limits with multibyte input and prove reading or writing stops at the boundary instead of checking only after full capture;
- reload persisted artifacts as hostile input and verify schema, size, redaction, control-character, and cross-reference invariants again;
- vary valid parser syntax, including key order, indentation, scalar versus mapping forms, optional fields, markers, URLs, digests, and Git status letters;
- clear Git process, global/system, repository, object-store, and replacement-ref controls in subprocess tests, then verify immutable refs remain bound to the validated work tree;
- parse Git path-bearing output with NUL-delimited records and transfer raw file-descriptor ownership exactly once;
- run installed wheel and sdist entrypoints with a restricted environment and a repository-local shadow package; and
- assert mandatory fields for skipped, clean, warning, error, and finding summaries through one shared outcome matrix.
Performance evidence must separate cold-start validation from steady-state requests. Profile realistic bounded stores and report wall time plus dominant cumulative functions; optimize the measured bottleneck.