Module llm Roadmap

Navigation: Home > Modules

LLM Module Roadmap

Current Status

The module provides production-grade LLM runtime surfaces across async inference, enhanced multi-model orchestration, adapter/plugin management, routing, streaming, and safety/policy controls.

Wave Alignment (see root ROADMAP.md § Program Execution Model):

Wave A (Q3–Q4 2026): Distributed end-to-end optimization (SpeculativeDecoder + cross-node inference hardening)
Wave B (Q3–Q4 2026): Wiki Phase B (RocksDB retrieval, cache hit-rate, query-latency gates)
Wave A Exit Criteria: Deterministic chaos evidence + fail-closed verification + release-critical CI GREEN
Wave B Exit Criteria: Full 4-layer retrieval chain with stable p95/p99 on representative hardware

Module Evidence Summary (2026-07-19)

Doxygen Documentation:

All 90 .cpp source files have Doxygen @file headers with maturity/quality metadata
Canonical Doxygen format applied to final_layer_orchestrator.cpp and model_switch_workflow.cpp

Build & Test Evidence:

Build preset: windows-release
Test target: module_llm_test_active_vram_allocator_focused
Latest validation: 2026-07-19
Result: PASS (exit 0, [ PASSED ] 45 tests)
Test timeout budget: 120s

Wiki Secondary Index — Phase A (2026-07-27)

Implementation units delivered

wiki_index_store.cpp — WikiIndexStore (BM25 + HNSW + RRF) + JsonWikiIndexReader
wiki_chunk_splitter.cpp — WikiChunkSplitter (heading-aware, sliding-window)
wiki_rag_source.cpp — WikiRagSource (RAGStageHandler, fail-closed default with explicit fail-open compatibility mode)

Test coverage

tests/llm/test_wiki_index_store.cpp — WIS-01..17 unit tests
tests/llm/test_wiki_rag_quality.cpp — WISQ-01..05 quality gate tests (Recall@5 ≥ 80 %, latency < 200 ms/10 queries on 100 chunks)
tests/llm/test_llm_doku_rag.cpp — RAG-01..12 doku.db CI test suite (2026-08-24)
- RAG-01..07: existing BM25/HNSW/RRF/phase/AdaLoRA/community/latency gates
- RAG-08: Golden dataset keyword gate (≥ 1 expected keyword per entry in Top-5)
- RAG-09: Golden dataset Recall@5 ≥ 80 % across all curated entries
- RAG-10: Source-hint presence gate (expected doc source in Top-5 results)
- RAG-11: Median query latency < 500 ms across all golden-dataset queries
- RAG-12: Golden dataset schema/size/distribution guard:
  - minimum dataset size ≥ 110 entries
  - distribution 20 % general / 30 % specific / 50 % specialized (±3 % tolerance)
  - specialized entries must be tagged as rare/non-generic knowledge points
tests/llm/data/themisdb_rag_golden_dataset.yaml — 110 curated Q/A entries (2026-08-24)
- Levels: 22 general (20 %), 33 specific (30 %), 55 specialized (50 %)
- Categories: architecture, roadmap, AdaLoRA/training, branch/release, build, API contracts
- Generated bootstrap script: scripts/generate_rag_golden_dataset.py

Phase B (2026-07-27, partial delivery)

Delivered:

Fix ingestion::BaseEntity → themis::BaseEntity type mismatch in wiki_index_store.cpp — writeChunk() and writeBatch() now build storage-compatible entities via chunkToEntity() that uses themis::BaseEntity::setField(). The entityToChunk() dead-code stub (which referenced the non-existent getProperty() API) was removed.
tests/llm/test_wiki_index_store_phase_b.cpp — WIS-B-01..16 RocksDB-backed integration tests covering: construction, empty query, BM25-only retrieval, KNN-only retrieval, hybrid RRF, writeBatch(), top_k, min_score, pre-embedded chunks, concurrent reads, writer+reader concurrency, score monotonicity, flush() no-op, query embedding cache, and multi-doc-id ingestion.
Thread-safety hardening: removed const_cast<EmbeddedLLM&>(llm_) from query(); added mutable EmbeddedLLM* llm_ptr_ + mutable std::mutex query_embed_mutex_ + mutable std::unordered_map<...> query_embed_cache_ so query-embedding caching is race-free under concurrent readers.

Remaining (Target: Q3–Q4 2026):

Persistent embedding cache keyed by chunk_id in RocksDB (2026-08-09: WikiIndexConfig::enable_persistent_cache; lazy RocksDB lookup via fetchPersistedEmbedding + persistEmbedding)
Embedding dimension auto-probe on first embed() call (2026-08-09: WikiIndexConfig::auto_probe_dim; probeEmbeddingDim() called on first write; dim_probed_ atomic guard)
Streaming ingest via WriteBatch with configurable batch size (2026-08-09: WikiIndexConfig::batch_size, default=32; writeBatch() uses config batch size instead of hardcoded 32)

In Progress

Planned Features

End-to-end distributed draft/verify optimization in speculative decoding paths (COMPLETE 2026-08-16) — SpeculativeDecoder remote-draft shard wiring and distributed end-to-end optimization completed in Wave A-8. Batch aggregation ≥8× speedup verified.
Stronger operational isolation for multi-tenant adapter lifecycle and cache surfaces (Target: Q4 2026)
Extended operator diagnostics for model routing, queue pressure, and policy-deny causes (Target: Q4 2026)
[~] Wave B B3: multi-task LoRA shared-base/domain-gating/joint-loss rollout (Target: Q1–Q2 2027) — core impl + ablation/benchmark tests done

Wave 2-B: RAII & Resource Safety (Target: Q4 2026)

Source: MODULE_GAP_ANALYSIS_WAVE2.md §Wave 2-B, gap scanner verified 2026-08-25
Gap count: 192 db_connection_leak (CRITICAL), 108 resource_leaked_in_exception, 118 pointer_arithmetic_unbounded

[~] Implement ScopedDbConnection RAII wrapper — replace all 192 raw DB-connection acquires in ml_model_manager.cpp, lora_storage_service_themisdb.cpp, inference_engine_enhanced.cpp (Target: Q4 2026)
- include/llm/scoped_db_connection.h created (2026-08-26)
- inference_engine_enhanced.cpp: replaced std::shared_ptr<void> RAII hack with ScopedDbConnection for model-plugin acquisition guard (Wave-B L2, 2026-08-26)
- Inputs: raw getConnection() call sites; bounded pool size config
- Outputs: RAII-wrapped connections released on scope exit or exception
- Constraints: zero new db_connection_leak findings post-fix; valgrind --leak-check=full clean
- Errors: pool exhaustion → ErrorCode::LLM_RESOURCE_EXHAUSTED; test: tests/llm/test_llm_raii_db_connections.cpp
- Perf: no throughput regression (benchmark: bench_llm_hotpaths LLM-01..LLM-08)
Fix resource_leaked_in_exception — distributed_training_coordinator.cpp: saveCheckpoint now writes to a .tmp file and renames atomically; partial-write on exception no longer corrupts checkpoint (Wave-B L3, 2026-08-26)
Bounds-check all pointer arithmetic in gpu_memory_manager.cpp — 5 pointer_arithmetic_unbounded sites in GPU/CPU defrag paths guarded with explicit offset + bytes > total check before every memcpy/cudaMemcpy (Wave-B L4, 2026-08-26)

Wave 2-C: LLM Stub Replacement (Target: Q4 2026)

Source: Semantic analysis 2026-08-25 — inference_engine_enhanced.cpp (8 stubs), inline_training_engine.cpp (5 stubs)

Replace the remaining speculative-decode fallback stubs in inference_engine_enhanced.cpp after the Wave-7 bridge work (Target: Q4 2026)
- Source revalidation 2026-08-31: TokenizerFn bridging is wired, production llama.cpp paths now auto-reuse live tokenizer state via LlamaWrapper::tokenizeForBridge(), generic non-llama draft paths now fail closed without byte-modulo token fabrication, target-model verification now auto-uses native llama-backed target logits instead of a fabricated peaked matrix, and speculative generation now installs a scoped per-request TARG entropy override so downstream RAG gates can reuse those target-logit rows without global callback leakage. Remaining follow-up is representative-hardware/acceptance evidence.
- Inputs: draft-model logits + verify-model logits; KV-cache capacity config
- Outputs: accepted token count, cache hit/miss metrics
- Tests: tests/llm/test_wave7_llm_kvcache_lru_checkpoint.cpp (LRU-01..LRU-10)
[~] Complete inline_training_engine.cpp training loop (5 stubs → production): SGD/Adam gradient update, loss tracking, model-checkpoint persistence to RocksDB, cancellation/timeout support (Target: Q4 2026)
- Persistent model_params_ vector added to Impl; training loop now updates real parameters across steps instead of a per-step zero-initialised dummy (Wave-B L5, 2026-08-26)
- SGD, Adam, AdamW, AdaGrad, RMSProp optimizers fully implemented and wired
- Stop flag (stop_flag) checked at epoch and batch boundaries
- Loss tracked per step, logged via spdlog
- Checkpoint persistence to RocksDB — setCheckpointDb() wired; dual-write (RocksDB + filesystem JSON) in saveCheckpoint(); RocksDB-first load with filesystem fallback in loadCheckpoint() (Wave-7, 2026-08-26)
- Constraints: loss must decrease over 10 epochs on synthetic data (test criterion)
- Errors: checkpoint write failure, cancellation mid-epoch
- Tests: tests/llm/test_inline_training_production.cpp

Wave 2-D: Thread-Safety Hardening — L7 Class (Target: Q4 2026)

Source: Thread-safety audit — shared state in inference handlers; top-20 std::atomic/mutex additions
Gap count: 13 sites across ml_model_manager.h/.cpp, llm_plugin_manager.h/.cpp

Wave A-8 Distributed Optimization Closure (2026-08-16)

Distributed End-to-End Inference Implementation ✅ COMPLETE

Objective: Close all 13 LLM distributed optimization gaps for production readiness.

Deliverables:

Remote draft shard integration in SpeculativeDecoder::Config::remote_draft_shard_id
Batch request aggregation in FederatedInferenceCoordinator (≥8× speedup on batch size 32)
Load balancing across shards (least-loaded prioritized, round-robin failover)
Cross-shard communication error handling (500ms timeout, 2 retries, exponential backoff)
Exception-safe RAII guards (strong guarantee on failure)
Thread-safe batch accumulation (verified with TSan)
8 focused distributed inference tests (LLM-DI-01..08)
28 exception safety tests (EXS-01..28)
24 memory safety tests (MEM-01..24)

Performance Verified:

Batch throughput: ≥8× vs sequential (batch size 32)
Draft-verify pipeline latency: <50ms
Cross-shard RPC timeout: <500ms (p99)
Load balancing fairness: >80% across shards
Fallback activation latency: <10ms

Test Coverage:

LLM-DI-01: Sharded inference coordination (3 shards)
LLM-DI-02: Draft-verify pipeline (100 draft, 95 verified)
LLM-DI-03: Cross-shard communication
LLM-DI-04: Speculative decode acceptance (partial token set)
LLM-DI-05: Inference failure recovery
LLM-DI-06: Load balancing across shards (9 ops, 3 shards)
LLM-DI-07: Shard failure handling (2/3 healthy)
LLM-DI-08: End-to-end distributed inference (3 workers, 10 tokens each)

Hardening & Production Readiness:

Exception safety (EXS-01..28) — model load/unload, adapter, plugin cleanup
Memory safety (MEM-01..24) — quota, cache, batch lifecycle, 1000-cycle stress
Concurrency safety (RC-01..08) — atomic ops, mutex, lock-free reads, memory ordering

Evidence: WAVE_A8_CLOSURE_EVIDENCE.md

Implementation Phases

Phase 1: Design / API Contract

Define and freeze non-breaking API contracts for inference, streaming, and routing extension points — include/llm/llm_api_contract.h (§1 Inference API, §4 Streaming, §5 Cancellation, §8 Concurrency) (Target: Q3 2026)
Document ownership/lifecycle boundaries for plugin, model, and adapter resources — include/llm/llm_api_contract.h (§3 Plugin/Adapter Lifecycle, §6 Resource Contracts VRAM/RAM) (Target: Q3 2026)

Phase 2: Core Implementation

[~] Implement pending distributed inference and speculative decode integration items (Target: Q4 2026)
[~] Complete runtime wiring for queue/load telemetry propagation in all configured execution paths (Target: Q4 2026)
- Fan-out merge path now emits per-request telemetry metadata (fan_out_success_count, fan_out_failure_count, fan_out_total_attempts, fan_out_max_dispatch_time_ms) in InferenceEngineEnhanced.
- Fan-out success path now preserves partial-failure envelope metadata for operator diagnostics.

Phase 3: Error Handling and Edge Cases

[~] Standardize failure envelopes for timeouts, cancellation, backend unavailability, and partial fan-out errors (Target: Q4 2026)
Harden fallback behavior when optional acceleration/runtime features are unavailable (Target: Q4 2026)
- FanOutInstanceResult now carries stable error_code + dispatch_time_ms; coordinator emits canonical classes for unknown-instance, timeout, backend-unavailable, permanent-remote, and retry-exhausted errors.

Phase 4: Tests

Expand focused tests for distributed orchestration, adapter hot-swap races, and stream abort handling (Target: Q4 2026)
- CBS-H-01..08: ContinuousBatchScheduler backpressure + quota tests (tests/llm/test_llm_hardening_phase4.cpp)
- TQM-H-01..04: TokenQuotaManager sliding-window semantics
- LAC-01..20: LLM API contract hardening (inference, batch, stream, plugin lifecycle, embed) — tests/llm/test_llm_api_contract_hardening_focused.cpp
- [~] Distributed orchestration / remote-shard failure paths (pending)
Add deterministic regression suites for routing and policy enforcement under load (Target: Q4 2026)
- PCL-H-01..06: PromptPolicy concurrent access + hot-swap safety
- SHD-H-01..04: Engine + scheduler shutdown-under-load teardown

Phase 5: Performance and Hardening

P5-L01: Exception safety audit + RAII wrapper hardening — 28 EXS-* tests delivered (tests/llm/test_llm_phase5_hardening.cpp)
P5-L02: Memory leak fixes (model loading, cache cleanup) — 24 MEM-* tests delivered (tests/llm/test_llm_phase5_hardening.cpp)
Lock performance gates to benchmark-backed thresholds and release baselines — 8 release-gate benchmarks LLM-01..LLM-08 in benchmarks/llm/bench_llm_hotpaths.cpp (GATE-LLM-01..GATE-LLM-08) (Target: Q4 2026)
[~] Validate memory-pressure and VRAM-recovery behavior under sustained multi-model load (Target: Q4 2026)

Recently Delivered — Phase 5 Hardening (2026-07-20)

EXS-01..28: Exception safety + RAII wrapper tests covering model load failures, move-only handle semantics, concurrent exception isolation, strong/basic guarantee proofs, shutdown under exception, and gate score (tests/llm/test_llm_phase5_hardening.cpp)
MEM-01..24: Memory leak simulation tests covering load/unload cycles, cache eviction, KV-cache/stream/batch teardown, LoRA cleanup, quota reset, concurrent load, 1 000-cycle stress, and gate score (tests/llm/test_llm_phase5_hardening.cpp)
52 new deterministic GTest cases; CTest labels: llm;hardening;phase5; TIMEOUT 120

Phase 1: Top-Risk Module Hardening (Exception-Safety/RAII/Memory-Leak/Race-Condition)

Fixed Exception-Safety Violations (Target: Q3 2026)
- LLM-EXC-01..08: Exception-safety during model load/unload (8 tests) ✓
- LLM-EXC-01: Model load success (no exception)
- LLM-EXC-02: Load throws, cleanup on exception
- LLM-EXC-03: Unload success (no exception)
- LLM-EXC-04: Double unload idempotent
- LLM-EXC-05: Exception during destruction (no throw)
- LLM-EXC-06: Strong exception guarantee (state unchanged)
- LLM-EXC-07: Basic exception guarantee (consistent state)
- LLM-EXC-08: Adapter load/unload sequence validation
Fixed Memory-Leak & Race-Condition Gaps (Target: Q3 2026)
- Audited include/llm/, src/llm/ (190 files) for non-RAII patterns
- LLM-RAII-01..08: RAII lifecycle and cleanup validation (8 tests) ✓
- LLM-RAII-01: UniquePtr automatic cleanup
- LLM-RAII-02: SharedPtr ref-counted cleanup
- LLM-RAII-03: SimAllocGuard move semantics
- LLM-RAII-04: Guard transfer ownership
- LLM-RAII-05: Multiple scopes cleanup
- LLM-RAII-06: Nested resource management
- LLM-RAII-07: Exception unwinding cleanup
- LLM-RAII-08: Cache lifecycle cleanup
Race-Condition & Concurrency Hardening (Target: Q3 2026)
- LLM-RC-01..08: Race-condition & concurrency scenarios (8 tests) ✓
- LLM-RC-01: Atomic increment thread-safe (10 threads × 100 ops = 1000)
- LLM-RC-02: Mutex-protected access validation
- LLM-RC-03: Concurrent model loading (3 threads)
- LLM-RC-04: Producer-consumer pattern validation
- LLM-RC-05: Read-write lock pattern
- LLM-RC-06: Memory ordering constraints (release/acquire)
- LLM-RC-07: Double-checked locking (std::once_flag)
- LLM-RC-08: Deadlock prevention (consistent lock order)
Multi-Tenant Operational Isolation (Target: Q4 2026)
- LLM-MT-01..08: Multi-tenant isolation (8 tests) ✓
- LLM-MT-01: Tenant isolation (no data leakage)
- LLM-MT-02: Per-tenant quota enforcement
- LLM-MT-03: Concurrent tenant access
- LLM-MT-04: Tenant cache isolation
- LLM-MT-05: Tenant resource cleanup
- LLM-MT-06: Cross-tenant contamination check
- LLM-MT-07: Tenant metadata consistency
- LLM-MT-08: Multi-tenant shutdown coordination
Distributed Inference & Speculative Decode (Target: Q4 2026)
- LLM-DI-01..08: Distributed inference edge cases (8 tests) ✓
- LLM-DI-01: Sharded inference coordination (3 shards)
- LLM-DI-02: Draft-verify pipeline (100 draft, 95 verified)
- LLM-DI-03: Cross-shard communication
- LLM-DI-04: Speculative decode acceptance (partial token set)
- LLM-DI-05: Inference failure recovery
- LLM-DI-06: Load balancing across shards (9 ops, 3 shards)
- LLM-DI-07: Shard failure handling (2/3 healthy)
- LLM-DI-08: End-to-end distributed inference (3 workers, 10 tokens each)
Created 40 Focused LLM Tests (Target: Q3 2026)
- LLM-EXC-01..08: Exception-safety (8 tests)
- LLM-RAII-01..08: RAII lifecycle (8 tests)
- LLM-RC-01..08: Race-condition/concurrency (8 tests)
- LLM-MT-01..08: Multi-tenant isolation (8 tests)
- LLM-DI-01..08: Distributed inference (8 tests)
- All tests: Use themis_register_module_focused_test(), tier unit/integration, timeout 120s
- Registered with label: release_critical;llm;phase1
P5-L01/P5-L02: Comprehensive Hardening Test Suite (Target: 2026-08-02)
- Extended include/llm/llm_memory_safety_utils.h with concurrency utilities (QuotaGuard, BatchGuard, ThreadSafeCounter)
- MEM-01..16: Memory safety lifecycle tests (quota/batch/resource management)
- MEM-17..28: Concurrency and backpressure tests (concurrent quota, batch scheduling, stress tests)
- EXS-01..25: Exception safety in model lifecycle (load/unload, adapter, plugin, cleanup guarantee)
- All 53 tests registered as module_llm_test_llm_memory_safety_hardening_focused
- File: tests/llm/test_llm_memory_safety_hardening.cpp
- Labels: release_critical llm batch1 memory_safety, TIMEOUT 120s each
- Deterministic (seed 42), ASan/TSan validated
Phase 1 Exit Criteria (2026-08-02)
- 0 new CRITICAL findings in CodeQL
- 93 focused tests created and passing (40 legacy + 53 comprehensive hardening)
- Exception-safety audits complete with documented contracts
- Memory-leak and race-condition fixes validated with sanitizers (ASan/TSan)
- Sanitizer evidence archived in docs/security/GA_SANITIZER_EVIDENCE_BUNDLE.md
- All module-level ROADMAP.md updated with closure status

Phase 6: Documentation and Acceptance

Synchronize operator docs/runbooks with implemented runtime behavior and metrics — include/llm/llm_api_contract.h documents all inference/embedding/plugin/streaming/cancellation/resource and error contracts for v1.x (Target: Q4 2026)
Publish acceptance checklist evidence for release sign-off — Phase 1 contract header, Phase 4 LAC-01..LAC-20 tests, Phase 5 LLM-01..LLM-08 benchmarks all delivered and referenced in this ROADMAP (Target: Q4 2026)

Production Readiness Checklist

API contracts for inference and streaming verified against tests and docs — include/llm/llm_api_contract.h + tests/llm/test_llm_api_contract_hardening_focused.cpp
Security and policy checks verified on all externally reachable LLM entry points — Phase 4 security review complete; pentest evidence in security/pentest/GA_PENTEST_EVIDENCE_BUNDLE.md
Performance expectations validated by reproducible release-profile benchmarks — benchmarks/llm/bench_llm_hotpaths.cpp (LLM-01..LLM-08)
Failure handling validated for cancellation, timeout, and backend degradation cases — P5-L01 EXS tests cover exception propagation, RAII teardown, and timeout paths
Audit and changelog documentation synchronized with implementation delta — CHANGELOG.md P5-L01/P5-L02 entries present; src/llm/ROADMAP.md up to date

Known Issues and Limitations

Some advanced distributed/remote execution optimizations depend on deployment wiring and are not universal defaults.
Runtime behavior can vary with enabled backend/plugin combinations and available hardware acceleration.
Not all benchmark targets currently represent transport- or topology-specific production mixes.

Wave 3 Gap-Closure Tracking (2026-08-31)

[~] ssm_state_rocksdb_store.cpp:261 — SSM snapshot serialization uses JSON; binary/protobuf serialization not yet implemented. Serialization is functionally correct but has higher storage overhead. Tracking comment added in source. Target Q2 2027.
[I] ssm_stub_plugin.cpp — SyntheticSSMStub STUB/SIMULATION NOTE block added to constructor; stub retained for Phase 1 PoC dataflow validation only.
[I] llm_plugin_manager.cpp:668 — THEMIS_LLAMA_CPP_STUB_MODE block documented with full STUB/SIMULATION NOTE template.

Wave B (Q1–Q2 2027) Tracking — B3 Multi-Task LoRA Fine-Tuning

Scope

Shared LoRA base with task-specific projections
Domain-gating mechanism
Joint loss with configurable task weighting
3-task benchmark evaluation and robustness checks

Validation

Unit tests MTL-01..10
Ablation study: shared multi-task training vs per-task single-task baselines

Acceptance Gates

Average task performance ≥ +8% vs single-task baseline
Training-time increase ≤ 15%
Robustness across task configurations

Dependencies

[~] Wave A deployment complete (Speculative Decoding, DPR, Fairness)
[~] Stable adapter lifecycle and benchmark baselines in LLM module

References

Detail tracker: ../ai/FUTURE_ENHANCEMENTS.md
Shared bibliography: ../../docs/research/ml_enhancements_bibliography.md
Issue scope: https://github.com/makr-code/ThemisDB/issues/5039

Breaking Changes

No breaking changes planned at roadmap level; any required API break must be explicitly documented in CHANGELOG and migration notes before merge.

Wave 3-LLM — Security Gap Triage & Closure (2026-08-25)

Triage Summary

Raw CRITICALs scanned: 155 across 113 source files
Real gaps confirmed: 5 (2 CRITICAL, 3 HIGH)
False positives removed: 150 (96.8% FP rate, consistent with prior waves)

False-Positive Patterns (Wave 3)

braces_imbalance (29): 100% FP — raw string literals (R"({...})") and #ifdef-gated brace blocks cause scanner heuristic to misfire; state-machine tokenizer confirms all files terminate at depth 0
circular_lock_ordering (108): 100% FP — scanner fires on any file with >1 mutex name; actual code uses consistent documented hierarchies and release-before-acquire patterns
data_race (11): 100% FP — all counters are std::atomic; collections protected by std::mutex/std::shared_mutex
sql_injection (7): 100% FP — RPC envelope strings named rpc_query, not SQL; mock strings in test/stub mode

Confirmed Real Gaps — All Fixed

ID	Severity	Pattern	Location	Fix
W3-SEC-01	HIGH	`insecure_model_url`	`model_downloader.cpp` `validateOllamaUrl`	Non-local HTTP rejected by default; `ModelDownloadConfig::allow_insecure_http` for explicit opt-in
W3-SEC-02	HIGH	`path_traversal`	`model_downloader.cpp` lines 150, 239	`sanitizeModelName()` rejects `..`, `/`, `\`, null bytes in `downloadFromOllama`/`pullFromOllama`
W3-SEC-03	CRITICAL	`deadlock_risk`	`ai_orchestrator.cpp` `PluginAdapterApplyService::applyAdapter`	Capture state under lock; release `mutex_` before `unloadLoRA`, `path_resolver_`, `loadLoRA`; re-lock to write
W3-SEC-04	CRITICAL	`prompt_injection`	`docs_assistant.cpp` lines 678, 683	`getConfigHelp`/`getTroubleshootingHelp` now apply `sanitizePromptWithSharedPolicy` with length caps (128 / 512 chars)
W3-SEC-05	HIGH	`hardcoded_path`	`llm_prefix_cache.cpp` line 46	`LLMPrefixCache::Config::cache_dir` field added; impl uses configured path with `/tmp/themis_llm_prefix_cache` as fallback

Tests Added

tests/llm/test_llm_wave3_gap_fixes.cpp — 15 regression tests (W3_01..W3_15)
tests/llm/test_model_downloader_url_validation.cpp — URL_VAL_05 updated; URL_VAL_10 added

Wave 3 Status

Triage 155 raw CRITICALs — triage report: ai_working/gap_verifier_report_llm.md
W3-SEC-01: insecure_model_url fixed
W3-SEC-02: path_traversal fixed
W3-SEC-03: deadlock_risk fixed
W3-SEC-04: prompt_injection fixed
W3-SEC-05: hardcoded_path fixed
Regression tests added (W3_01..W3_15)
ROADMAP and MODULE_GAPS updated

Wave 9 Block 5 — LLM CRITICAL Closure + Speculative Decode Wiring (2026-08-26)

Summary

CRITICAL residual after W9-16: 135 (20 braces_imbalance scanner FPs closed)
Speculative decode bridges: TokenizerFn wired; TargetLogitsFn initially exposed as an injection point
Tests: tests/llm/test_wave9_speculative_decode_bridges.cpp (SD-BRG-01..07)

W9-16: Batch-close `braces_imbalance` false positives

Verified all 20 braces_imbalance CRITICAL entries using a C++ state-machine parser that skips raw string literals — all 20 confirmed structurally balanced
grafana_metrics.cpp raw count −3 explained by R"()" JSON payloads
MODULE_GAPS.md updated with verification table (155 → 135 residual)

W9-17: Speculative decode bridges

TokenizerFn type + setTokenizerFn() / clearTokenizerFn() added to InferenceEngineEnhanced public API (include/llm/inference_engine_enhanced.h)
setTokenizerFn / clearTokenizerFn implemented in src/llm/inference_engine_enhanced.cpp (mutex-guarded, same pattern as TargetLogitsFn)
Remote draft path in trySpeculativeGeneration() updated to call TokenizerFn before byte-modulo fallback; fail-closed on exception
STUB #263 note updated: "Removal Plan" → "Production Injection Point"
STUB #262 note updated: "Removal Plan" → "Production Injection Point" (TargetLogitsFn was already fully wired; note corrected)
Tests SD-BRG-01..SD-BRG-07 added
STUB_INVENTORY.md entries 322/323 marked resolved

Wave 10 Block 8 — Native Target-Logit Bridge (2026-08-31)

Summary

trySpeculativeGeneration() no longer fabricates peaked target-logit matrices for unsupported paths
Native llama-backed target plugins now provide exact K+1 verification rows through computeTargetLogitsForTokens()
Generic non-llama target plugins now fail closed back to normal target generation when no TargetLogitsFn override is available

W10-8: STUB #262 closure

LlamaWrapper::computeTargetLogitsForTokens() added for exact speculative verification rows
LlamaCppPlugin::computeTargetLogitsForTokens() forwards the same bridge through the plugin boundary
InferenceEngineEnhanced::trySpeculativeGeneration() now prefers injected TargetLogitsFn, then native llama-backed target logits, and otherwise fails closed
Active + legacy speculative bridge tests updated to reflect target-generation fallback for unsupported generic targets

Wave 10 — Local Draft Plugin Bridge (2026-08-27)

W10-D: Bridge ILLMPlugin::setDraftTokensFn() for local draft path

trySpeculativeGeneration() local draft path wired: when TokenizerFn is set on the engine, a GenerateDraftTokensFn lambda is injected into ILLMPlugin::setDefaultGenerateDraftTokensFn() before calling draft_plugin->generateDraftTokens().
Lambda captures draft_plugin by value (shared_ptr copy) — no this capture across thread boundaries.
Bridge cleared (setDefaultGenerateDraftTokensFn(nullptr)) after the call, including on exception, to prevent global state pollution.
Byte-modulo heuristic retained as documented fallback when TokenizerFn is absent or returns empty / throws.
STUB #261 comment updated in include/llm/llm_plugin_interface.h: "Production Injection Point (wired by InferenceEngineEnhanced::trySpeculativeGeneration, 2026-08-27)".
Tests SD-LOCAL-01 and SD-LOCAL-02 added in tests/llm/test_w10d_local_draft_bridge.cpp.

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues

ThemisDB 1.9.0-beta · Home · Wiki-Index · Module-Index · FAQ · Quick-Reference · GitHub · Issues · Discussions · License

Module llm Roadmap

LLM Module Roadmap

Current Status

Module Evidence Summary (2026-07-19)

Wiki Secondary Index — Phase A (2026-07-27)

Implementation units delivered

Test coverage

Phase B (2026-07-27, partial delivery)

In Progress

Planned Features

Wave 2-B: RAII & Resource Safety (Target: Q4 2026)

Wave 2-C: LLM Stub Replacement (Target: Q4 2026)

Wave 2-D: Thread-Safety Hardening — L7 Class (Target: Q4 2026)

Wave A-8 Distributed Optimization Closure (2026-08-16)

Distributed End-to-End Inference Implementation ✅ COMPLETE

Implementation Phases

Phase 1: Design / API Contract

Phase 2: Core Implementation

Phase 3: Error Handling and Edge Cases

Phase 4: Tests

Phase 5: Performance and Hardening

Recently Delivered — Phase 5 Hardening (2026-07-20)

Phase 1: Top-Risk Module Hardening (Exception-Safety/RAII/Memory-Leak/Race-Condition)

Phase 6: Documentation and Acceptance

Production Readiness Checklist

Known Issues and Limitations

Wave 3 Gap-Closure Tracking (2026-08-31)

Wave B (Q1–Q2 2027) Tracking — B3 Multi-Task LoRA Fine-Tuning

Scope

Validation

Acceptance Gates

Dependencies

References

Breaking Changes

Wave 3-LLM — Security Gap Triage & Closure (2026-08-25)

Triage Summary

False-Positive Patterns (Wave 3)

Confirmed Real Gaps — All Fixed

Tests Added

Wave 3 Status

Wave 9 Block 5 — LLM CRITICAL Closure + Speculative Decode Wiring (2026-08-26)

Summary

W9-16: Batch-close braces_imbalance false positives

W9-17: Speculative decode bridges

Wave 10 Block 8 — Native Target-Logit Bridge (2026-08-31)

Summary

W10-8: STUB #262 closure

Wave 10 — Local Draft Plugin Bridge (2026-08-27)

W10-D: Bridge ILLMPlugin::setDraftTokensFn() for local draft path

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

W9-16: Batch-close `braces_imbalance` false positives