From 69a416fb3efb2af89a273ca16acbf8705136be3b Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Mon, 25 May 2026 21:30:12 +0530 Subject: [PATCH 1/2] =?UTF-8?q?chore(tracker):=20sync=20Step=201.4=20?= =?UTF-8?q?=E2=86=92=20=E2=9C=85=20and=20log=20PRs=20#58,=20#59,=20#60?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Flip Step 1.4 (OCR pipeline) from ⏳ to ✅ now that PR #60 has merged; capture deliverables in the row (rag-ocr package, region-aware SPI, Tesseract + PaddleOCR plugins, ragctl ocr, schemas, docs). - Phase 1 progress: 11/16 → 12/16 done (4 remaining) - Total: 24/84 → 25/84 done - Advance Next-action to Step 1.5 (structure-aware chunker) with a brief note on how it consumes Step 1.3 blocks + Step 1.4 OCR regions. - Add PRs #58, #59, #60 to the PR & Branch History table. Co-Authored-By: Claude Opus 4.7 (1M context) --- TRACKER.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/TRACKER.md b/TRACKER.md index 8394a4a..e4ec7f3 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -8,9 +8,9 @@ > Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience. > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`. -**Last updated:** 2026-05-24 +**Last updated:** 2026-05-25 **Current phase:** Phase 1 — Ingestion + Knowledge Store -**Next action:** Phase 1 Step 1.4 — OCR pipeline: Tesseract + PaddleOCR plugins, image region extraction, confidence scoring. +**Next action:** Phase 1 Step 1.5 — Structure-aware chunker: heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization. Consumes `ParsedDocument.blocks` from Step 1.3 and OCR regions from Step 1.4 as natural chunk boundaries. > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md]. @@ -32,14 +32,14 @@ | Phase | Title | Steps | ✅ Done | Remaining | |-------|-------|------:|-------:|----------:| | 0 | Foundation | 13 | **13** | 0 | -| 1 | Ingestion + Knowledge Store | 16 | **11** | 5 | +| 1 | Ingestion + Knowledge Store | 16 | **12** | 4 | | 2 | Retrieval Engine | 11 | 0 | 11 | | 3 | Gateway & Agent Runtime | 11 | 0 | 11 | | 4 | Reliability | 6 | 0 | 6 | | 5 | Eval & Observability | 7 | 0 | 7 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **84** | **24** | **60** | +| **Total** | | **84** | **25** | **59** | --- @@ -76,7 +76,7 @@ | 1.1f | ADRs 0005–0009 + reviewer checklist | ✅ | `build/phase-1/step-1.1f-adrs-reviewer-checklist` | [#52](https://github.com/officialCodeWork/AgentContextOS/pull/52) | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization) all promoted to Accepted with implementation-path backlinks. Reviewer checklist in `docs/architecture/performance.md` extended with explicit items for plan/budget awareness, IndexHint+dtype handling, two-stage rerank shape, and BlobRef-safe consumers. ADR backlinks added to `rag_core.spi.reranker` and `rag_core.spi.storage` (the two missing module-level pointers). | | 1.2 | Connectors framework | ✅ | `build/phase-1/step-1.2-connectors-framework` | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | `Connector` SPI evolved to `(Document, ConnectorState)` async-iterator with resumable watermark (`ConnectorState` frozen model: connector_id, tenant_id, cursor, last_seen_at, extra). Internal `Crawler[Raw]` base class in `rag_backends.connectors._base` (iteration scaffold, error policy, monotonic watermark). Built-in connectors: `FilesystemConnector` (cross-platform pathlib, sha256 hashing, sorted resume cursor), `S3Connector` (aioboto3, ListObjectsV2 continuation token, MinIO-compatible), `GCSConnector` (gcloud-aio-storage, REST nextPageToken, `[gcs]` extra). Policy boundary documented — connectors are upstream of `PolicyEngine`; the ingest pipeline (Step 1.10) is the enforcement point. 9 contract tests (NoopConnector resume + watermark + tenant mismatch), 9 unit tests (FilesystemConnector), MinIO + GCS integration tests skip-if-no-service. `ConnectorState` schema added to `dist/schemas/`. New docs: [reference/connectors.md](docs/reference/connectors.md), [architecture/connectors.md](docs/architecture/connectors.md). | | 1.3 | Document parsers | ✅ | `build/phase-1/step-1.3-document-parsers` | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | New `rag-parsers` package (`packages/parsers/`, v0.1.0) shipping `PlainTextParser`, `MarkdownParser` (markdown-it-py), `HtmlParser` (BeautifulSoup + stdlib `html.parser`), `JsonParser`, `CsvParser`, `YamlParser` (PyYAML `safe_load`), `PdfParser` (pypdf), `DocxParser` (python-docx), `PptxParser` (python-pptx), `XlsxParser` (openpyxl). `Parser` SPI evolved: `parse(...)` now returns `ParsedDocument` (`Document` + extracted text + structural `Block` list with offsets + heading levels + parser-specific `attributes`). New types in `rag_core`: `Block`, `BlockType` (8-value enum), `ParsedDocument`. `rag_parsers.mime.detect_mime()` (puremagic + OOXML ZIP central-directory disambiguation + extension shortcut + hint fallback). `ParserRegistry` + `default_registry()` (first-match-wins; specific MIMEs before generics). `ragctl parse ` smoke-test command (MIME detect → parser select → print summary + first-N blocks). 35 parser unit tests + 6 ragctl tests + extended contract tests (148 contract tests total pass). Schemas regenerated (`Block.json`, `ParsedDocument.json`). `rag-core` 0.7.0 → 0.8.0. Cross-platform: every parser library is pure-Python or wheel-distributed; no `libmagic` / `lxml` system deps. New docs: [reference/parsers.md](docs/reference/parsers.md), [architecture/parsers.md](docs/architecture/parsers.md). | -| 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring | +| 1.4 | OCR pipeline | ✅ | `build/phase-1/step-1.4-ocr-pipeline` | [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | New `rag-ocr` package (`packages/ocr/`, v0.1.0): `TesseractOCR` (per-word regions, ISO-639-3 lang codes, `pytesseract` wrapper) + `PaddleOCRBackend` (per-line regions, quadrilateral → axis-aligned bbox, lazy engine init); both behind optional extras `[tesseract]`/`[paddle]` so default install stays pure-Python. OCR SPI evolved: `BoundingBox(x0,y0,x1,y1)` + `OCRRegion(text,confidence,bbox,page_number)` + `OCRResult(text,confidence,regions,page_number,engine)` Pydantic v2 frozen models in `rag_core.types`; aggregate confidence is text-length-weighted mean (1.0 on empty regions so "no text" doesn't read as failure); `NoopOCR` migrated; 5 contract tests updated. `ragctl ocr ` smoke command (`--engine tesseract\|paddleocr`, `--lang`, `--regions N`) — 5 CLI tests. 22 unit tests under `tests/ocr/` stub both engines at the `sys.modules` boundary so CI runs without Tesseract binary or PaddlePaddle wheels. Schemas regenerated (`BoundingBox.json`, `OCRRegion.json`, `OCRResult.json`). `rag-core` 0.8.0 → 0.9.0. New docs: [reference/ocr.md](docs/reference/ocr.md), [architecture/ocr.md](docs/architecture/ocr.md). | | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization | | 1.6 | Metadata enricher | ⏳ | — | — | Auto-tag: language, doc type, created/modified dates, author, section path, reading level | | 1.7 | PII detection | ⏳ | — | — | Presidio integration; per-tenant policy (block/redact/allow); `pii.detected` event; CI gate | @@ -225,6 +225,9 @@ | [#55](https://github.com/officialCodeWork/AgentContextOS/pull/55) | feat(core,backends): connectors framework — SPI evolution + filesystem/S3/GCS (Step 1.2) | `build/phase-1/step-1.2-connectors-framework` | ✅ Merged | 2026-05-24 | | [#56](https://github.com/officialCodeWork/AgentContextOS/pull/56) | chore(tracker): sync Step 1.2 → ✅ and log PR #55 | `chore/tracker-sync-pr55` | ✅ Merged | 2026-05-24 | | [#57](https://github.com/officialCodeWork/AgentContextOS/pull/57) | feat(core,parsers): document parsers — PDF/DOCX/PPTX/XLSX/HTML/MD/text/JSON/CSV/YAML (Step 1.3) | `build/phase-1/step-1.3-document-parsers` | ✅ Merged | 2026-05-24 | +| [#58](https://github.com/officialCodeWork/AgentContextOS/pull/58) | chore(tracker): sync Step 1.3 → ✅ and log PRs #56, #57 | `chore/tracker-sync-pr57` | ✅ Merged | 2026-05-25 | +| [#59](https://github.com/officialCodeWork/AgentContextOS/pull/59) | fix(parsers): normalize CRLF/CR to LF in text & markdown parsers | `fix/parsers-windows-crlf` | ✅ Merged | 2026-05-25 | +| [#60](https://github.com/officialCodeWork/AgentContextOS/pull/60) | feat(core,ocr): OCR pipeline — region-aware SPI + Tesseract + PaddleOCR (Step 1.4) | `build/phase-1/step-1.4-ocr-pipeline` | ✅ Merged | 2026-05-25 | --- From daf54532b9f0c251107ce2d5a92aba627337b745 Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Mon, 25 May 2026 21:31:15 +0530 Subject: [PATCH 2/2] chore(tracker): reconcile Phase 1 milestone arithmetic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The milestone summary table had drifted off the actual row counts. Phase 1 has 16 step rows (1.1 + 1.1a–f + 1.2…1.10); after Step 1.4 lands, 10 are ✅ and 6 are ⏳ — not 12/4. Reset to match the table reality and bring the grand total in line (23/84 done, 61 remaining). Co-Authored-By: Claude Opus 4.7 (1M context) --- TRACKER.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TRACKER.md b/TRACKER.md index e4ec7f3..ec5e1d7 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -32,14 +32,14 @@ | Phase | Title | Steps | ✅ Done | Remaining | |-------|-------|------:|-------:|----------:| | 0 | Foundation | 13 | **13** | 0 | -| 1 | Ingestion + Knowledge Store | 16 | **12** | 4 | +| 1 | Ingestion + Knowledge Store | 16 | **10** | 6 | | 2 | Retrieval Engine | 11 | 0 | 11 | | 3 | Gateway & Agent Runtime | 11 | 0 | 11 | | 4 | Reliability | 6 | 0 | 6 | | 5 | Eval & Observability | 7 | 0 | 7 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **84** | **25** | **59** | +| **Total** | | **84** | **23** | **61** | ---