diff --git a/TRACKER.md b/TRACKER.md index 8394a4a..ec5e1d7 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -8,9 +8,9 @@ > Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience. > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`. -**Last updated:** 2026-05-24 +**Last updated:** 2026-05-25 **Current phase:** Phase 1 — Ingestion + Knowledge Store -**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring. +**Next action:** Phase 1 Step 1.5 — Structure-aware chunker: heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization. Consumes `ParsedDocument.blocks` from Step 1.3 and OCR regions from Step 1.4 as natural chunk boundaries. > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md]. @@ -32,14 +32,14 @@ | Phase | Title | Steps | ✅ Done | Remaining | |-------|-------|------:|-------:|----------:| | 0 | Foundation | 13 | **13** | 0 | -| 1 | Ingestion + Knowledge Store | 16 | **11** | 5 | +| 1 | Ingestion + Knowledge Store | 16 | **10** | 6 | | 2 | Retrieval Engine | 11 | 0 | 11 | | 3 | Gateway & Agent Runtime | 11 | 0 | 11 | | 4 | Reliability | 6 | 0 | 6 | | 5 | Eval & Observability | 7 | 0 | 7 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **84** | **24** | **60** | +| **Total** | | **84** | **23** | **61** | --- @@ -76,7 +76,7 @@ | 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). | | 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). | | 1.3 | Document parsers | ✅ | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse ` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). | -| 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring | +| 1.4 | OCR pipeline | ✅ | `build/phase-1/step-1.4-ocr-pipeline` | [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | New `rag-ocr` package (`packages/ocr/`, v0.1.0): `TesseractOCR` (per-word regions, ISO-639-3 lang codes, `pytesseract` wrapper) + `PaddleOCRBackend` (per-line regions, quadrilateral → axis-aligned bbox, lazy engine init); both behind optional extras `[tesseract]`/`[paddle]` so default install stays pure-Python. OCR SPI evolved: `BoundingBox(x0,y0,x1,y1)` + `OCRRegion(text,confidence,bbox,page_number)` + `OCRResult(text,confidence,regions,page_number,engine)` Pydantic v2 frozen models in `rag_core.types`; aggregate confidence is text-length-weighted mean (1.0 on empty regions so "no text" doesn't read as failure); `NoopOCR` migrated; 5 contract tests updated. `ragctl ocr ` smoke command (`--engine tesseract\|paddleocr`, `--lang`, `--regions N`) — 5 CLI tests. 22 unit tests under `tests/ocr/` stub both engines at the `sys.modules` boundary so CI runs without Tesseract binary or PaddlePaddle wheels. Schemas regenerated (`BoundingBox.json`, `OCRRegion.json`, `OCRResult.json`). `rag-core` 0.8.0 → 0.9.0. New docs: [reference/ocr.md](docs/reference/ocr.md), [architecture/ocr.md](docs/architecture/ocr.md). | | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization | | 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level | | 1.7 | PII detection | ⏳ | — | — | Presidio integration; per-tenant policy (block/redact/allow); `pii.detected` event; CI gate | @@ -225,6 +225,9 @@ | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | feat(core,backends): connectors framework — SPI evolution + filesystem/S3/GCS (Step 1.2) | `build/phase-1/step-1.2-connectors-framework` | ✅ Merged | 2026-05-24 | | [#56](https://github.com/officialCodeWork/AgentContextOS/pull/56) | chore(tracker): sync Step 1.2 → ✅ and log PR #55 | `chore/tracker-sync-pr55` | ✅ Merged | 2026-05-24 | | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | feat(core,parsers): document parsers — PDF/DOCX/PPTX/XLSX/HTML/MD/text/JSON/CSV/YAML (Step 1.3) | `build/phase-1/step-1.3-document-parsers` | ✅ Merged | 2026-05-24 | +| [#58](https://github.com/officialCodeWork/AgentContextOS/pull/58) | chore(tracker): sync Step 1.3 → ✅ and log PRs #56, #57 | `chore/tracker-sync-pr57` | ✅ Merged | 2026-05-25 | +| [#59](https://github.com/officialCodeWork/AgentContextOS/pull/59) | fix(parsers): normalize CRLF/CR to LF in text & markdown parsers | `fix/parsers-windows-crlf` | ✅ Merged | 2026-05-25 | +| [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | feat(core,ocr): OCR pipeline — region-aware SPI + Tesseract + PaddleOCR (Step 1.4) | `build/phase-1/step-1.4-ocr-pipeline` | ✅ Merged | 2026-05-25 | ---