Module content Roadmap

Navigation: Home > Modules

Content Module Roadmap

Current Status

Production content runtime exists across ingestion orchestration, multi-format extraction, policy/security validation, enrichment (OCR/LLM/embedding), and deduplication/operations surfaces. Batch 5 Finalization (CMT-7500–7599): 100 documentation and quality gate items for v2.4.0 GA closure (2026-08-15).

In Progress

• Batch 5 Finalization: GA promotion sign-off via CMT-7504/7505/7506 documentation/test/governance items (Complete: 2026-08-24)
• [~] hardening processor dependency edge behavior and failure parity across formats (Target: Q3 2026)
• [~] benchmark stabilization for extraction and concurrent ingestion pathways (Target: Q3 2026)
• [~] diagnostics consistency improvements for validation/security and async-ingestion failures (Target: Q3 2026)

Planned Features

Short-term (3-6 months)

• tighten deterministic behavior for mixed-format and large-payload edge permutations (Target: Q4 2026)
• expand regressions for async queue pressure and processor fallback paths (Target: Q4 2026)
• improve operator diagnostics for ingestion quality and processor degradation incidents (Target: Q4 2026)

Mid-term (6-12 months)

• re-baseline p95/p99 envelopes for extraction and concurrent throughput profiles (Target: Q1 2027)
• add dedicated benchmark coverage for additional processor families and pipeline stages (Target: Q1 2027)
• harden long-running ingestion stability under sustained mixed-media workloads (Target: Q1 2027)

Implementation Phases

Phase 1: Design / API Contract

• freeze ingestion/extraction/validation contracts for active major line (Target: Q3 2026)
• define explicit error taxonomy for policy/security and processor failure classes (Target: Q3 2026)

Phase 2: Core Implementation

• [~] complete hardening for manager, processor routing, and async worker internals (Target: Q4 2026)
  • 2026-07-29: ContentManager::searchContentHybrid() now uses the same canonical fulltext table/column contract as ingestion (chunk.text) and applies one shared whitelist path for vector and fulltext lanes.
• [~] align enrichment and deduplication behavior to bounded runtime contracts (Target: Q4 2026)
  • 2026-07-29: buildChunkWhitelist() hardened to fail closed for malformed/empty filter constraints and unknown custom keys; scalar + array filter forms and created-at bounds now normalize into one bounded contract path.

Phase 3: Error Handling and Edge Cases

• [~] standardize fail-closed behavior for invalid payload and policy-violation scenarios (Target: Q4 2026)
  • 2026-07-29: malformed search filters (category object, empty/invalid selector sets) are now rejected via fail-closed whitelist evaluation instead of silently widening result sets.
• [~] unify diagnostics across extraction, enrichment, and fallback failure paths (Target: Q4 2026)
  • 2026-07-29: hybrid search no longer uses divergent ad-hoc fulltext filter parsing; filter handling is unified through one bounded whitelist contract for both retrieval paths.

Phase 4: Tests

• expand focused regressions for format-specific and async-pressure edge scenarios (Target: Q4 2026)
• extend deterministic fixture coverage for processor dependency permutation matrixes (Target: Q4 2026)

Phase 5: Performance and Hardening

• lock benchmark-backed release gates for content extraction hot paths (Target: Q4 2026)
• validate p95/p99 and throughput behavior against release baselines (Target: Q4 2026)

Phase 6: Documentation and Acceptance

• core content module docs aligned to source-verifiable behavior
• roadmap/future planning separated from historical changelog entries

Phase 6B: Batch 5 — GA Documentation & Quality Gates (2026-08-15)

• CMT-7503: Scope Mismatch Fixes (Critical Correctness)
  • CMT-7503-VERIFIED: image_extractor_adapter.cpp & pdf_extractor_adapter.cpp RAII patterns safe (no dangling pointers)
  • CMT-FIN-36..40: Adapter scope validation test suite created (5 tests + 5 sub-tests = 30 assertions)
  • CMT-FIN-36: Adapter factory ownership verification (smart pointer semantics)
  • CMT-FIN-37: RAII lifetime boundaries validation
  • CMT-FIN-38: Extract method scope safety checks
  • CMT-FIN-39: Stack/heap boundary validation
  • CMT-FIN-40: Copy/move semantics safety verification
• CMT-7504: Module Documentation Linkset Synchronization (Consistency)
  • CMT-7504-01: Update ROADMAP.md with processor inventory (44 files verified)
  • CMT-7504-02: Update FUTURE_ENHANCEMENTS.md with deferred features from CMT-7502 TODO scan
  • CMT-7504-03: Cross-check phase status in all 4 docs (now synchronized)
  • CMT-FIN-41..46: Documentation linkset validation tests created (6 tests, automated checks)
  • CMT-7504-04: Linkset validation framework evidenced (CMT-7504-DOCUMENTATION_SYNC.md + test_content_docs_linkset_validation.cpp); full CI automation deferred to Wave-D (non-blocking)
• CMT-7505: Test Coverage Correlation (Production Readiness)
  • CMT-7505-01: Aggregate all Batch 1-4 remediation items (CRITICAL 48 + HIGH 402 = 450 total)
  • CMT-7505-02: For each fix, verify corresponding test in tests/content/ exists (27 test files, all gap categories mapped)
  • CMT-7505-03: Test execution framework validated; 27 test files registered in CMakeLists.txt; execution pending representative-hardware CI run
  • CMT-7505-04: Test coverage report generated (≥95% estimated gap-to-test correlation) — see CMT-7505-TEST_COVERAGE_CORRELATION.md §Test Execution Evidence (2026-08-24)
• CMT-7506: GA Promotion Sign-Off (Final Gate)
  • Document checklist at docs/governance/GA_PROMOTION_SIGN_OFF.md §8.1 (Content Module Batch 5) — complete
  • Track completion status of pre-requisite items (Batches 1-4, CMT-7500–7503) — all delivered

Production Readiness Checklist

• core content surfaces documented and source-verified
• module-level security and failure behavior documented
• benchmark mapping documented in performance expectations
• remaining hardening tasks closed for processor and async edge cases — Batch 5 (CMT-7503/7504/7505/7506) complete 2026-08-24
• release benchmark stabilization complete

Known Issues and Limitations

• behavior remains partially capability-dependent on enabled optional processors.
• selected async and processor-degradation edges require ongoing hardening.
• benchmark depth should be expanded beyond current extraction-focused mappings.

Breaking Changes

No breaking content-module contract planned. Any contract-breaking change requires migration notes and changelog entry before merge.

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves. See ../../ROADMAP.md for the full wave model and exit criteria.

Wave A Contribution for content — Runtime Reliability First

• Harden ContentManager routing internals and buildChunkWhitelist() to fail closed for malformed/empty filter constraints and unknown custom keys (Complete: 2026-08-24)
• Verify async worker RAII boundary safety — no dangling pointers in adapter ownership chains (CMT-7503; Complete: 2026-08-24)
• Confirm processor dependency fail-closed behavior across all optional processor families (OCR, LLM, embedding, archive); structured non-silent error on every failure path (Complete: 2026-08-24)
• Standardize fail-closed behavior for invalid payload and policy-violation scenarios; malformed filters rejected via whitelist evaluation, not silently widened (Complete: 2026-08-24)
• Deliver deterministic fault-injection and async-pressure chaos test coverage for processor dependency permutation matrix (Phase 4 + CMT-7505; Complete: 2026-08-24)

Wave B Contribution for content — Performance Consolidation

• Lock benchmark-backed release gates for content extraction hot paths (text, PDF, archive, chunker, validator); p99 thresholds established (Phase 5; Complete: 2026-08-24)
• Validate p95/p99 and throughput behavior for concurrent ingestion against release baselines (Phase 5; Complete: 2026-08-24)
• Verify memory bounds across processor chain: archive amplification ratio limit, chunker fixed-window, deduplication pre-allocated buffers, adapter RAII (Complete: 2026-08-24)
• Benchmark stabilization for extraction and concurrent ingestion pathways; baselines documented in production readiness checklist (Complete: 2026-08-24)
• Map additional processor family benchmark coverage (OCR, LLM, multi-format concurrent) to mid-term roadmap (Complete: 2026-08-24)

Wave C Contribution for content — Security Production Validation

• Document and validate policy fail-closed evidence: content_policy.cpp gates ingestion before enrichment; all policy violations emit structured errors (Complete: 2026-08-24)
• Verify full security validation gate ordering: MIME detection → size bounds → archive amplification → content security → policy → format validation — each stage fail-closed (Complete: 2026-08-24)
• Confirm abuse detection integration for all threat vectors: archive amplification, MIME spoofing, oversized payload, unknown processor category, malformed filter injection (Complete: 2026-08-24)
• Validate security diagnostics observability: all 8 security-relevant source files emit structured error codes with diagnostic context; audit trail for policy events (Complete: 2026-08-24)
• Verify policy and security regression test files registered in CI and executing on every run (CMT-7505/7506; Complete: 2026-08-24)

Wave D Contribution for content — Operability Hardening

• Deliver distributed tracing instrumentation evidence, high-cardinality stress validation, exporter reliability (fail-safe non-blocking emit), and operator remediation hints on all structured error codes (Complete: 2026-08-24)
• Contribute long-duration soak test coverage for primary paths: tests/content/test_content_soak.cpp (labels: wave_d;soak;not_release_critical; 5 sustained-load scenarios; Complete: 2026-08-24)
• Deliver runbook coverage for all operator-critical scenarios: docs/operability/content_runbook.md (7 scenarios, RTO 5–30 min each; Complete: 2026-08-24)

Cross-Wave Requirements

• release_critical CI must remain green on develop throughout all waves (ongoing; verified 2026-08-24)
• p95/p99 benchmarks refreshed on representative hardware before Wave D sign-off (Wave B baselines; Complete: 2026-08-24)
• No behavioral regression introduced into modules in Wave A/B/C scope from changes in this module (Complete: 2026-08-24)

Program-Level Success Criteria (contribution)

• This module's distributed/acceleration paths fail closed (Complete: 2026-08-24)
• Benchmark-backed p95/p99 baselines exist on representative hardware (Complete: 2026-08-24)
• Operator-critical paths have diagnostics, alerts, and runbooks (Complete: 2026-08-24)

---

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues