Module sharding Future

Navigation: Home > Modules

Sharding Module - Future Enhancements

Scope

• complete the GA hardening pass for routing, topology rebalancing, and GSI behavior already present in source
• verify production semantics under quorum loss, migration churn, and multi-DC latency variability
• convert current runtime logic into formal release-gate evidence on representative hardware
• keep non-production paths fail-closed and explicitly documented

Design Constraints

• sharding contracts remain backward compatible within major release line.
• routing/transaction outcomes remain explicit and deterministic.
• degraded quorum and operations paths remain observable and non-silent.
• migration/repair behavior remains bounded and diagnosable.
• all non-production paths must be explicitly marked and documented.
• no fallback may silently route to an arbitrary shard when valid latency evidence is absent.

Required Interfaces

Interface	Requirement
routing interfaces	deterministic key-to-shard and adaptive routing semantics
coordination interfaces	explicit distributed decision and quorum behavior
transaction interfaces	stable cross-shard commit/abort contracts
operations interfaces	bounded repair/rebalance/migration with observable states

Implementation Notes

• tighten parity between topology updates and routing consistency diagnostics.
• standardize incident taxonomy for transaction and repair/rebalance classes.
• expand resilience tests for prolonged migration and write contention traffic.
• broaden benchmark depth for multi-DC and topology-failure scenarios.
• keep the latency fallback semantics fail-closed when no measured RTT is valid under the timeout budget.
• GSI stale-entry invalidation must remain deterministic after shard migration or shard removal.

Test Strategy

• unit and integration suites for routing, coordination, transaction, and operational paths.
• regressions for quorum loss, migration faults, and repair edge failures.
• deterministic stress runs for sustained distributed write and rebalance workloads.
• release-profile benchmark runs for mapped sharding targets.
• GA proof: 3-DC latency-routing test, topology churn rebalancing test, and GSI stale-entry test must all pass in tests/sharding/test_sharding_core.cpp.

Performance Targets

• routing p95/p99 should remain stable within the release budget under degraded quorum and migration pressure.
• rebalancing under topology churn must preserve ≥80% steady-state throughput and finish within configured bounds.
• GSI equality and range queries must remain deterministic and bounded under shard churn.
• mapped benchmark manifests reach no-missing-case status for release gating.

Security / Reliability

• maintain strict bounded behavior for topology and routing transitions.
• preserve explicit failure signaling for transaction and operational faults.
• enforce predictable degradation under shard and quorum failure conditions.
• keep diagnostics actionable for production sharding incidents.
• any route decision made without valid RTT evidence must remain fail-closed rather than silently redirecting to an arbitrary shard.

---

Stub/Simulation Path Remediation (2026-07-19)

This section documents improvements made to stub and simulation code paths in the sharding module to ensure they are production-ready and properly documented.

Summary of Changes

All identified stub/simulation paths have been reviewed and improved with:

1. NON-PRODUCTION PATH markers - replaced generic STUB/SIMULATION NOTE with explicit NON-PRODUCTION PATH comments
2. Enhanced diagnostics - added THEMIS_WARN/THEMIS_INFO logging to identify when stubs are used
3. Error handling - improved error messages and validation for edge cases
4. Documentation - clarified production deltas and removal plans

Path-by-Path Improvements

1. gRPC Fallback Mode (shard_rpc_client.cpp:127)

• Status: ✅ IMPROVED
• Type: Graceful fallback (test-only path)
• Change: Added THEMIS_WARN log on initialization to warn operators that gRPC is disabled
• Production readiness: Acceptable for test/single-node only; production requires THEMIS_HAS_SHARD_GRPC=1

2. In-Process RPC Handler (shard_rpc_client.cpp:924)

• Status: ✅ IMPROVED
• Type: Simulation with injected response handler
• Changes:
  • Added one-time warning log when simulation is first used
  • Improved error message for unknown RPC methods
  • Added error logging with diagnostic hints
• Production readiness: Acceptable for loopback/single-node only; multi-node requires gRPC

3. mTLS Preparation Callback (stream_protocol.cpp:635)

• Status: ✅ IMPROVED
• Type: In-process simulation (test-only path)
• Change: Added THEMIS_WARN log with explicit diagnostic information
• Production readiness: Test-only; production requires real mTLS transport callback injection

4. Connection Factory (mtls_connection_pool.cpp:232)

• Status: ✅ COMPLETED (v2.0)
• Type: Factory pattern for connection lifecycle management
• Changes:
  • Created IEndpointConnectionFactory abstract interface (include/sharding/mtls_connection_factory.h)
  • Implemented MTLSConnectionFactory concrete class (src/sharding/mtls_connection_factory.cpp)
  • Added EndpointConnectionPool constructor overload accepting factory
  • Updated createNewConnection() stub comment to mark COMPLETED
  • Added comprehensive factory-based integration tests (9 new tests)
  • Added deprecation warnings to MTLSClient methods
• Production readiness: COMPLETED - Factory pattern fully implemented and tested for v2.0

5. GPU VRAM Fallback (shard_resource_manager.cpp:666)

• Status: ✅ VERIFIED
• Type: Graceful degradation (CPU-only systems)
• Change: Renamed marker from STUB/SIMULATION to NON-PRODUCTION PATH (Graceful Degradation)
• Production readiness: Acceptable; expected behavior on systems without CUDA/HIP

6. ONNX Model Predictor (predictive_detector.cpp:45)

• Status: ✅ IMPROVED
• Type: Heuristic-based simulation (acceptable fallback)
• Changes:
  • Clarified that sigmoid-calibrated heuristics are conservative and acceptable
  • Added THEMIS_INFO log to loadModel() explaining heuristic-based operation
  • Improved documentation noting that this is a valid early warning system
• Production readiness: Acceptable for current release; upgrade to ONNX in v1.5+

Migration Checklist

For operators deploying these modules in production:

• Single-node deployments: All in-process/simulation paths are acceptable
• Multi-node deployments: MUST enable THEMIS_HAS_SHARD_GRPC=1 and deploy real shard peers
• mTLS deployments: MUST inject real transport callbacks via setPrepareTransferCallback()
• GPU systems: MUST enable THEMIS_ENABLE_CUDA or THEMIS_ENABLE_HIP for resource-aware scheduling
• Failure prediction: Current heuristic-based detector is conservative; upgrade to ONNX for improved accuracy

Verification

All changes maintain backward compatibility and do not alter API contracts. Diagnostic logging uses appropriate levels (WARN for issues, INFO for informational, DEBUG for deep diagnostics).

Next Steps

1. v1.5 roadmap: Add ONNX Runtime support and replace heuristic predictor
2. v2.0 roadmap (COMPLETED Phase 2):
   • ✅ IEndpointConnectionFactory interface created
   • ✅ MTLSConnectionFactory concrete implementation
   • ✅ EndpointConnectionPool factory-based constructor
   • ✅ Factory-based integration tests (9 comprehensive tests)
   • ✅ MTLSClient deprecation warnings added
   • Phase 3 (Next): Wire factory injection into MTLSClient for production use
3. Ongoing: Monitor diagnostic logs from production deployments to validate graceful fallbacks

---

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues