feat(org-memory): AI knowledge base plugin for Mattermost

agent-memories/org-memory-chunking-engine.md

@@ -0,0 +1,36 @@
+# Org-Memory Chunking Engine (Step 005)
+
+## File Location
+- `server/plugins/org-memory/server/chunker.go` — main implementation
+- `server/plugins/org-memory/server/chunker_test.go` — comprehensive tests
+
+## Key Types
+- `ChunkResult` — contains Text, Index (0-based), StartByte, EndByte
+- `ChunkConfig` — TargetChunkTokens, OverlapPercent, MaxChunks, Logger
+- `segment` — internal type for pipeline processing with byte offsets
+
+## Constants
+- `DefaultTargetChunkTokens = 500` (~2000 chars)
+- `DefaultOverlapPercent = 10`
+- `DefaultMaxChunksPerPost = 32`
+- `runesPerToken = 4` (heuristic: 1 token ≈ 4 runes)
+
+## Public API
+- `ChunkText(text string, config ChunkConfig) []ChunkResult` — main entry point, pure function
+- `EstimateTokens(text string) int` — rune-based token count approximation
+
+## Algorithm Pipeline
+1. Strip markdown (regex-based, no goldmark dependency)
+2. Split at paragraph boundaries (double newlines)
+3. Split oversized paragraphs at sentence boundaries
+4. Merge small segments up to target size
+5. Add 10% overlap between adjacent chunks (word-boundary-aware)
+6. Truncate to MaxChunks with logging
+7. Assign sequential 0-based indices
+
+## Important Notes
+- The plugin does NOT use goldmark — markdown stripping is regex-based
+- Code blocks (indented content) are treated as atomic segments, never sentence-split
+- Sentence boundary: `[.!?]\s+[A-Z\p{Lu}]` — splits on "Dr. Smith" but not "e.g. test"
+- The function is source-agnostic — accepts any text, caller determines SourceType
+- All text processing uses rune counts, not byte counts (Unicode-aware)