diff --git a/agent-memories/org-memory-chunking-engine.md b/agent-memories/org-memory-chunking-engine.md new file mode 100644 index 000000000000..6159008279a3 --- /dev/null +++ b/agent-memories/org-memory-chunking-engine.md @@ -0,0 +1,36 @@ +# Org-Memory Chunking Engine (Step 005) + +## File Location +- `server/plugins/org-memory/server/chunker.go` — main implementation +- `server/plugins/org-memory/server/chunker_test.go` — comprehensive tests + +## Key Types +- `ChunkResult` — contains Text, Index (0-based), StartByte, EndByte +- `ChunkConfig` — TargetChunkTokens, OverlapPercent, MaxChunks, Logger +- `segment` — internal type for pipeline processing with byte offsets + +## Constants +- `DefaultTargetChunkTokens = 500` (~2000 chars) +- `DefaultOverlapPercent = 10` +- `DefaultMaxChunksPerPost = 32` +- `runesPerToken = 4` (heuristic: 1 token ≈ 4 runes) + +## Public API +- `ChunkText(text string, config ChunkConfig) []ChunkResult` — main entry point, pure function +- `EstimateTokens(text string) int` — rune-based token count approximation + +## Algorithm Pipeline +1. Strip markdown (regex-based, no goldmark dependency) +2. Split at paragraph boundaries (double newlines) +3. Split oversized paragraphs at sentence boundaries +4. Merge small segments up to target size +5. Add 10% overlap between adjacent chunks (word-boundary-aware) +6. Truncate to MaxChunks with logging +7. Assign sequential 0-based indices + +## Important Notes +- The plugin does NOT use goldmark — markdown stripping is regex-based +- Code blocks (indented content) are treated as atomic segments, never sentence-split +- Sentence boundary: `[.!?]\s+[A-Z\p{Lu}]` — splits on "Dr. Smith" but not "e.g. test" +- The function is source-agnostic — accepts any text, caller determines SourceType +- All text processing uses rune counts, not byte counts (Unicode-aware) diff --git a/agent-memories/org-memory-comprehensive-codebase-overview.md b/agent-memories/org-memory-comprehensive-codebase-overview.md new file mode 100644 index 000000000000..4730fe9ccd52 --- /dev/null +++ b/agent-memories/org-memory-comprehensive-codebase-overview.md @@ -0,0 +1,518 @@ +# Org-Memory Plugin - Comprehensive Codebase Overview + +## 1. Plugin.go - Main Plugin Struct and Lifecycle + +### Plugin Struct Fields +```go +type Plugin struct { + plugin.MattermostPlugin // Embeds Mattermost plugin interface + client *pluginapi.Client // Pluginapi client for Mattermost services + config *configuration // Current config (protected by configLock) + configLock sync.RWMutex // Protects concurrent config access + db *sql.DB // Master database handle + + // AI/Processing Components + geminiClient GeminiClient // Gemini AI client (embedding, extraction, summarization) + circuitBreaker *CircuitBreaker // Circuit breaker for Gemini API (KV store backed) + rateLimiter *RateLimiter // Exponential backoff rate limiter (KV store backed) + + // Storage + store Store // Knowledge store interface + enabledTeams map[string]bool // Cached enabled team IDs (nil = all enabled) + + // Job Scheduling + jobOnceScheduler *cluster.JobOnceScheduler // One-time job scheduler (debounce, backfill) + processingJob interface{ Close() error } // Background worker poll-and-process job + + // Metrics & Future Extensions + workerMetrics WorkerMetrics // Worker operation metrics (atomic counters) + summaryScheduler interface{} // Placeholder for step 011 (channel summaries) + metricsCollector interface{} // Placeholder for step 014 (Prometheus) +} +``` + +### Lifecycle Methods + +**OnActivate()** - Initialization sequence: +1. Create pluginapi.Client +2. Load & validate configuration +3. Obtain master DB handle +4. Run schema migrations (pgvector + 7 tables) +5. Obtain replica DB handle (fallback to master if unavailable) +6. Create SQLStore (master + replica) +7. Initialize CircuitBreaker (KV store backed) +8. Initialize RateLimiter (KV store backed) +9. Create GeminiClient +10. Register /ask slash command +11. Cache enabled teams set +12. Initialize JobOnceScheduler (singleton, may reuse existing) +13. Start processing worker with cluster.Schedule (1-second interval) + +**OnDeactivate()** - Graceful shutdown: +1. Cancel all pending jobs via jobOnceScheduler +2. Close processing worker (blocks until current item finishes) +3. Close database via store service +4. Log clean shutdown + +**OnConfigurationChange()** - Config updates: +1. Load new configuration +2. Validate configuration +3. Check if Gemini client needs reinitialization (API key, models, dimensions) +4. Reinitialize Gemini client if needed (circuit breaker/rate limiter state preserved) +5. Store config and rebuild cached team set + +--- + +## 2. Configuration Struct + +```go +type configuration struct { + GeminiAPIKey string // Required: Google AI API key + GeminiModel string // LLM model for summarization/extraction/RAG + EmbeddingModel string // Model for vector embeddings + EmbeddingDimensions int // Dimensionality (must match pgvector column) + + WorkerPollMs int // Background worker poll frequency + ThreadDebounceSeconds int // Inactivity threshold before summarization + MaxChunksPerPost int // Max text chunks per post + SkipBotPosts bool // Filter bot messages from processing + EnableForTeams string // Comma-separated team IDs (empty = all) + DailySummaryHour int // Hour of day (0-23) for channel summaries (step 011) +} + +// Validate() - Ensures GeminiAPIKey is present +// TeamIDSet() - Parses EnableForTeams into map for O(1) lookup +// IsTeamEnabled(teamID) - Checks if team is enabled +``` + +--- + +## 3. Store Interface & SQLStore Implementation + +### Top-Level Store Interface Groups Subsystems +```go +type Store interface { + Embedding() EmbeddingStore + Extraction() ExtractionStore + ThreadSummary() ThreadSummaryStore + ChannelSummary() ChannelSummaryStore + EntityMention() EntityMentionStore + ProcessingLog() ProcessingLogStore + QueryLog() QueryLogStore + DataRetention() DataRetentionStore +} +``` + +### Key Store Methods by Domain + +**EmbeddingStore** +- Save(embeddings []*Embedding) - Batch insert +- DeleteByPostID(postID) - Hard delete +- SoftDeleteByPostID(postID, deleteAt) - Soft delete +- SearchByVector(vector []float32, channelIDs []string, limit int) - pgvector similarity search + +**ExtractionStore** +- Save(extractions []*Extraction) - Batch insert +- GetByID(id) - Single extraction +- List(channelID, extractionType, offset, limit) - Paginated query +- UpdateContent(id, content, editorUserID, updateAt) - Manual edits +- SoftDeleteByPostID/DeleteByPostID + +**ThreadSummaryStore** +- Upsert(summary *ThreadSummary) - Insert or update per RootId +- GetByRootID(rootID) - Single thread summary +- ListByChannelIDInWindow(channelID, windowStart, windowEnd) - Time-windowed queries + +**ChannelSummaryStore** +- Save(summary *ChannelSummary) - Insert +- GetLatest(channelID, summaryType) - Most recent +- List(channelID, summaryType) - All summaries for channel + +**EntityMentionStore** +- Save(mentions []*EntityMention) - Batch insert +- GetEntityTimeline(entityName, channelIDs, offset, limit) - Cross-channel entity links +- DeleteByPostID + +**ProcessingLogStore** - Work queue with retry logic +- Create(entry) - Idempotent insert (skip if exists) +- Upsert(entry) - Insert or reset existing to "pending" +- UpdateStatus(id, status, errorMessage, completeAt) - Status transitions +- IncrementRetry(id) - Increment retry counter +- ListPending(limit) - Get pending items ordered by ScheduleAt +- FindStale(thresholdMs) - Items stuck in "processing" > threshold +- ResetStale(thresholdMs) - Reset stale items to "pending" +- MarkCompleteByPostID(postID, completeAt) - Mark entries complete (post deletion) +- GetStats() - Status counts + +**QueryLogStore** - Audit trail +- Insert(entry) - Log query execution +- GetStats() - Total query count and average latency + +**DataRetentionStore** - Batch cleanup +- DeleteBefore(beforeTimestamp, batchSize) - Delete aged data from all tables + +### SQLStore Implementation +- Uses Squirrel query builder (SQL injection safe) +- Master DB for writes, Replica DB for reads +- Atomic operations via KV store with SetAtomicWithRetries for state management + +--- + +## 4. Gemini Client Interface & Implementation + +### GeminiClient Interface +```go +type GeminiClient interface { + // Embedding generation with L2 normalization + GenerateEmbedding(ctx, text) ([]float32, error) + + // Knowledge extraction from post/conversation + ExtractKnowledge(ctx, conversationText) (*ExtractionResult, error) + + // Combined thread summarization + extraction + SummarizeAndExtract(ctx, threadText) (*ThreadSummaryResult, error) + + // RAG answer synthesis with citations + SynthesizeAnswer(ctx, query, passages, conversationHistory) (*SynthesisResult, error) +} +``` + +### Gemini Client Internals +```go +type geminiClient struct { + sdkClient *genai.Client // google.golang.org/genai SDK + embeddingModel string // e.g., "gemini-embedding-001" + llmModel string // e.g., "gemini-2.5-flash" + embeddingDimensions int + circuitBreaker *CircuitBreaker + rateLimiter *RateLimiter +} +``` + +### Key Methods & Patterns + +**GenerateEmbedding()** +- Checks resilience gates (circuit breaker + rate limiter) +- 30-second timeout +- Calls Models.EmbedContent with OutputDimensionality +- L2-normalizes the output vector +- Records success/failure with resilience components + +**ExtractKnowledge()** +- Extracts decisions, action items, topics from conversation +- Returns JSON with items array +- Each item has: type, content, confidence, assignee (optional), source_post_id (optional) +- Filters valid extraction types + +**SummarizeAndExtract()** +- Combined operation for thread summarization +- Returns summary text, key_takeaways array, and items array +- Handles empty/unparseable responses gracefully + +**SynthesizeAnswer()** +- RAG synthesis: combines query with context passages + conversation history +- Builds system prompt emphasizing citation format [N] +- Adds previous turns to multi-turn conversation context +- Returns answer with inline citations + +### Error Handling +- classifyError() categorizes errors: RateLimit, Retryable, NonRetryable +- handleError() updates resilience state accordingly +- handleSuccess() resets rate limiter and circuit breaker state + +### Timeouts +- embeddingTimeout = 30 seconds +- llmTimeout = 60 seconds (for extraction, summarization, synthesis) + +--- + +## 5. Processing Worker (worker.go) + +### Pattern: cluster.Schedule +```go +processingJob, err := cluster.Schedule( + p.API, + "org_memory_processing_worker", // Unique job name + cluster.MakeWaitForInterval(1*time.Second), // Poll frequency + p.workerCallback, // Callback function +) +``` +- Only one instance runs across the cluster at a time +- Callback invoked on each poll cycle +- Gracefully shutdown with Close() + +### Worker Metrics (Atomic Counters) +```go +type WorkerMetrics struct { + EmbedProcessed int64 + ExtractProcessed int64 + EmbedFailed int64 + ExtractFailed int64 + EmbedPermanentErrors int64 + ExtractPermanentErrors int64 + StaleRecovered int64 + LastQueueDepth int64 + TotalEmbedProcessingTimeMs int64 + TotalExtractProcessingTimeMs int64 +} +``` + +### workerCallback() Main Loop +1. Check circuit breaker (skip cycle if open) +2. Query pending items (batch size = 10) +3. Track queue depth metric +4. Return immediately if empty (also check for stale recovery) +5. Process each item sequentially + - Return if circuit breaker trips mid-batch +6. Recover stale items after batch + +### processItem() Per-Item Flow +1. Panic recovery (don't crash worker loop) +2. Acquire per-post distributed mutex (2-second timeout via KV store) +3. Transition status to "processing" +4. Route by job type: + - "embed" → handleEmbedJob() + - "extract" → handleExtractJob() +5. Handle result: + - If ErrCircuitOpen: return to pending, signal stop + - If other error: call handleItemFailure() + - If success: mark complete, update metrics +6. Log completion + +### handleItemFailure() Retry Logic +- Increment retry count +- If forceError OR retryCount >= 3 (max retries): mark as "error" (permanent) +- Otherwise: return to "pending" for retry + +### handleEmbedJob() - Vector Embedding +1. Determine content source: + - Post: fetch post.Message + - File: fetch file text content (3 sources: FileInfo.Content, KV pre-extracted, raw bytes) + - Summary: fetch ThreadSummary.Summary +2. Chunk content (MaxChunksPerPost, TargetChunkTokens, OverlapPercent) +3. Generate embeddings for each chunk +4. Delete previous embeddings (for reprocessing after edits) +5. Store new embeddings +6. Return nil on success + +### handleExtractJob() - Knowledge Extraction +1. Fetch post from Mattermost +2. Prepare conversation context: + - Include thread context (up to 5 recent messages before current post) +3. Call Gemini extraction +4. Delete previous extractions and entity mentions +5. Store new extractions +6. Extract named entities (person, topic) +7. Store entity mentions + +### recoverStaleItems() - Detect & Recover Stuck Items +- Find items in "processing" status > 5 minutes old +- For each stale item: + - Increment retry count + - If >= max retries: mark as permanent error + - Otherwise: reset to "pending" + - Update metric + +### Helper: clusterMutex (Custom Implementation) +- KV store atomic CAS-based mutex (instead of cluster.Mutex) +- LockWithContext() with context deadline support +- Refresh goroutine (7.5-second refresh, 15-second TTL) +- Unlock() stops refresh and deletes key + +--- + +## 6. Debounce Handler (debounce.go) + +### Pattern: JobOnceScheduler for Thread Debounce +- Key prefix: "debounce:" +- Scheduled via scheduleDebounce() after every message/edit in a thread +- One-time job per thread root (cancel & reschedule on new message) + +### handleDebounceJob() - Thread Summarization & Extraction +1. Parse job key/props (rootID, channelID, retryCount) +2. Validate root post still exists +3. Fetch full thread (PostList) +4. Filter eligible posts (no bots, no system messages) +5. Check thread size (skip if < 2 eligible posts) +6. Enrich posts with usernames (cached within call) +7. Route by thread size: + - < 100 posts: Direct SummarizeAndExtract() call + - >= 100 posts: Map-reduce approach (split into 50-post groups, summarize each, synthesize final) +8. Handle Gemini errors: reschedule with incremented retry (max 3 retries, 60-second delay) +9. Store thread summary +10. Store thread extractions and entity mentions +11. Queue unprocessed individual post embeddings (idempotent create) +12. Create summary processing entry (source type "summary") +13. Publish WebSocket event "thread_summary_ready" + +### Key Constants +- largeThreadThreshold = 100 posts +- mapReduceGroupSize = 50 posts per group +- debounceMaxRetries = 3 +- debounceRetryDelaySeconds = 60 + +### Map-Reduce for Large Threads +1. Map phase: Summarize each group (parallel possible, currently sequential) +2. Reduce phase: Synthesize group summaries into final summary +3. Merge extraction items from all groups + +--- + +## 7. Circuit Breaker Pattern + +### Implementation +```go +type CircuitBreaker struct { + kv *pluginapi.KVService // KV store for cluster-wide state +} + +type CircuitBreakerState struct { + State CircuitState // closed, open, half_open + ConsecutiveFailures int + LastStateChangeAt int64 // Unix millis +} +``` + +### Constants +- circuitBreakerFailureThreshold = 5 consecutive failures +- circuitBreakerOpenDuration = 30 seconds + +### State Machine +1. **Closed** (normal): Requests pass through + - On failure: increment counter, open if >= 5 + - On success: reset counter to 0 + +2. **Open** (tripped): Reject all requests immediately (return ErrCircuitOpen) + - After 30 seconds: transition to half-open + - On success (testing): close + - On failure (testing): re-open + +3. **Half-Open** (recovery testing): Allow one request through + - On success: transition to closed + - On failure: re-open immediately + +### Methods +- AllowRequest() - Check if request should proceed +- RecordSuccess() - Reset to closed +- RecordFailure() - Increment counter, potentially open/re-open +- GetState() - Read current state + +--- + +## 8. Rate Limiter Pattern + +### Implementation +```go +type RateLimiter struct { + kv *pluginapi.KVService +} + +type RateLimiterState struct { + NextAllowedAt int64 // Unix millis until next request allowed + CurrentBackoffMs int64 // Current backoff duration +} +``` + +### Exponential Backoff Strategy +- Initial backoff: 1 second +- Doubling on each consecutive 429: 1s → 2s → 4s → ... → 60s (max) +- Jitter: ±20% to prevent thundering herd + +### Methods +- AllowRequest() - Check if within backoff window (return ErrRateLimited if yes) +- RecordRateLimit() - Double backoff, calculate NextAllowedAt with jitter +- RecordSuccess() - Reset state (NextAllowedAt=0, CurrentBackoffMs=0) +- GetState() - Read current state + +--- + +## 9. Event Hooks (hooks.go) + +### MessageHasBeenPosted +1. Eligibility check (ShouldProcessMessage) +2. Team enablement check +3. Create processing log entry (embed, post source type) +4. Trigger thread debounce (scheduleDebounce) +5. Process file attachments + +### MessageHasBeenUpdated +1. Eligibility check +2. Team enablement check +3. Check if message content actually changed +4. Upsert processing log (resets complete → pending) +5. Re-trigger thread debounce + +### MessageHasBeenDeleted +1. Manage debounce timers: + - Root post deleted: cancel debounce + - Reply deleted: reschedule debounce for parent thread +2. Clean up embeddings, extractions, entity mentions +3. Mark processing log entries as complete +4. Clean up file-related data + +### FileWillBeUploaded +1. Size guard (max 10MB) +2. Extension guard (only supported text types) +3. Read file content +4. Validate UTF-8 +5. Store in KV store (10-minute TTL) for later use +6. Never reject/modify upload + +### UserHasBeenDeactivated +1. Delete all conversation sessions (multi-turn) for user +2. Preserve all knowledge base content + +### dispatchScheduledJob() +- Routes scheduled jobs by key prefix: + - "debounce:" → handleDebounceJob() + - "backfill:" → (step 017, not implemented yet) + +--- + +## 10. Models.go - Data Structures + +### Embedding +- PostID, ChannelID, SourceType (post/file/summary) +- ChunkIndex, ContentSnapshot +- Embedding ([]float32), CreateAt, DeleteAt + +### Extraction +- PostID, ChannelID, ExtractionType (decision/action_item/topic) +- Content, Metadata (JSON with assignee, confidence, source_post_id) +- IsEdited, EditorUserID, CreateAt, UpdateAt, DeleteAt + +### ThreadSummary +- RootID, ChannelID +- Summary, KeyTakeaways (JSON array) +- ModelVersion, LastActivityAt, UpdateAt + +### ChannelSummary +- ID, ChannelID, SummaryType +- Summary, WindowStart, WindowEnd +- CreateAt + +### EntityMention +- EntityName, EntityType (person/topic) +- PostID, ChannelID, ExtractionID +- CreateAt + +### ProcessingLogEntry +- PostID, JobType (embed/extract), SourceType (post/file/summary) +- Status (pending/processing/complete/error) +- ScheduleAt, ErrorMessage, RetryCount +- CreateAt, CompleteAt + +--- + +## Key Architectural Insights + +1. **Cluster-Wide Resilience**: Circuit breaker and rate limiter state stored in KV store +2. **One-Time Jobs**: Thread debounce uses JobOnceScheduler (singleton), cancel & reschedule on activity +3. **Work Queue Pattern**: Processing log with status tracking, retries, stale recovery +4. **Thread-Safe Config**: RWMutex protects config and team set during live updates +5. **Graceful Degradation**: Fail-open for circuit breaker/rate limiter if KV read fails +6. **Panic Recovery**: Both worker loop and debounce handler recover from panics +7. **Map-Reduce for Scale**: Large threads (>100 posts) use group summarization +8. **Source-Aware Processing**: Embeddings can be from posts, files, or summaries +9. **Entity Linking**: Extractions create entity mentions for cross-channel search +10. **Mutex Pattern**: Custom KV-based distributed mutex with refresh goroutine diff --git a/agent-memories/org-memory-e2e-tests.md b/agent-memories/org-memory-e2e-tests.md new file mode 100644 index 000000000000..acfa274d7e42 --- /dev/null +++ b/agent-memories/org-memory-e2e-tests.md @@ -0,0 +1,52 @@ +# Org-Memory Plugin E2E Tests + +## E2E Test Location +`e2e-tests/playwright/specs/functional/plugins/org_memory/` + +## Test Files +- `org_memory_installation.spec.ts` - Plugin install/activation, app bar icon visibility +- `org_memory_commands.spec.ts` - /ask slash command, WebSocket thread summary events +- `org_memory_rhs.spec.ts` - RHS panel open/close, query submission, loading states, citations, follow-up queries, new conversation reset + +## Helper Files +- `helpers/test-setup.ts` - `setupOrgMemoryTest()`, `openOrgMemoryRhs()`, `submitQueryInRhs()` +- `helpers/mock-org-memory-api.ts` - Route interception mocks for all plugin API endpoints + +## How to Run +```bash +cd e2e-tests/playwright +npm install # Install dependencies (includes @playwright/test) +npx playwright install # Install browser binaries +# Requires a running Mattermost server with org-memory plugin installed +npx playwright test specs/functional/plugins/org_memory/ --project=chrome +``` + +## Prerequisites +1. Running Mattermost server on localhost:8065 (or configured PW_BASE_URL) +2. org-memory plugin built and installed/enabled on the server +3. Playwright dependencies installed (`npm install` in e2e-tests/playwright/) +4. Playwright browsers installed (`npx playwright install`) + +## Key Design Pattern +All tests use Playwright's `page.route()` for API interception - no real Gemini API key needed. The `setupOrgMemoryTest()` helper handles server config, plugin activation, and page navigation. Tests that need the plugin but it's not installed will `test.skip()` gracefully. + +## Tags +- `@smoke` - Installation test only +- `@org-memory` - All org-memory tests + +## Selective Run Examples +```bash +npx playwright test --grep @org-memory --project=chrome +npx playwright test specs/functional/plugins/org_memory/org_memory_rhs.spec.ts --project=chrome +``` + +## Critical Bug Found: CSRF Token Missing +The webapp bundle (`webapp/dist/main.js`) was missing the `X-CSRF-Token` header in its `fetch()` POST requests to the plugin API. This caused Mattermost's CSRF protection to reject the requests as unauthenticated. Fixed by adding `"X-CSRF-Token": (document.cookie.match(/MMCSRF=([^;]+)/)||[])[1]||""` to the fetch headers. + +## Exploratory Testing Notes (2026-03-27) +- Plugin binary only implements `/api/v1/query` POST endpoint (other spec'd routes return 404) +- Slash command `/ask` is NOT registered by the deployed plugin binary +- The deployed plugin tarball (`server/data/plugins/org-memory.tar.gz`) was built on Mar 24 +- Plugin source code (Go files) lives in gitignored `server/plugins/org-memory/server/` directory +- pgvector extension must be manually installed in PostgreSQL for embeddings to work +- Password field in Mattermost login form uses `name="password-input"` not `name="password"` \ No newline at end of file diff --git a/agent-memories/org-memory-gemini-client.md b/agent-memories/org-memory-gemini-client.md new file mode 100644 index 000000000000..dadbcbf3a0c4 --- /dev/null +++ b/agent-memories/org-memory-gemini-client.md @@ -0,0 +1,29 @@ +# Gemini Client Implementation (Step 004) + +## SDK Version +- `google.golang.org/genai v0.7.0` — the Go SDK for Google Gemini +- Uses `genai.NewClient(ctx, &genai.ClientConfig{APIKey: key, Backend: genai.BackendGeminiAPI})` +- Methods: `client.Models.GenerateContent()`, `client.Models.EmbedContent()` +- Error type: `genai.APIError` (value type, not pointer) with `.Code` for HTTP status +- JSON output: `ResponseMIMEType: "application/json"` in `GenerateContentConfig` +- Helper: `genai.Ptr[T](val)` for optional pointer fields + +## Files Created +- `gemini_types.go` — GeminiClient interface, ExtractionItem, ThreadSummaryResult, ContextPassage, SynthesisResult types +- `gemini_client.go` — Implementation using genai SDK, l2Normalize function +- `gemini_errors.go` — Error classification (retryable/non-retryable/rate-limit), ErrCircuitOpen, ErrRateLimited sentinels +- `circuit_breaker.go` — 3-state circuit breaker (closed/open/half-open) with KV store persistence +- `rate_limiter.go` — Exponential backoff (1s→60s cap, ±20% jitter) with KV store persistence +- Test files for all components + +## Key Design Decisions +- Circuit breaker state in KV key `gemini_circuit_breaker`, rate limiter in `gemini_rate_limiter` +- Both use `pluginapi.KV.SetAtomicWithRetries` for cluster-wide consistency +- Non-retryable errors (400, 401, 403) bypass circuit breaker entirely +- Rate limit (429) triggers both backoff AND circuit breaker +- Timeouts: 30s embedding, 60s LLM operations +- `geminiClient` field on Plugin changed from `interface{}` to `GeminiClient` (typed interface) + +## Plugin Lifecycle Integration +- OnActivate: creates CircuitBreaker → RateLimiter → GeminiClient (steps 8-9) +- OnConfigurationChange: reinitializes GeminiClient if API key/model names change, preserves CB/RL instances diff --git a/agent-memories/org-memory-plugin-build-system.md b/agent-memories/org-memory-plugin-build-system.md new file mode 100644 index 000000000000..3473d6626fd0 --- /dev/null +++ b/agent-memories/org-memory-plugin-build-system.md @@ -0,0 +1,18 @@ +# Org-Memory Plugin Build System + +## Key Build Details +- Plugin source lives at `server/plugins/org-memory/` +- Go server source is in `server/plugins/org-memory/server/` with its own `go.mod` +- **CRITICAL**: Must set `GOWORK=off` when building the plugin because the parent `server/go.work` workspace interferes. The plugin has its own separate module. +- Go module path: `github.com/mattermost/mattermost-plugin-org-memory/server` +- The mattermost public SDK is imported as `github.com/mattermost/mattermost/server/public` +- Google AI SDK is imported as `google.golang.org/genai` +- Webapp uses webpack, React/ReactDOM are externals (provided by Mattermost window globals) +- `@mattermost/types` npm package has stale peer dep on TypeScript ^4.3 — we skip it and use inline types +- The `make verify` target builds everything and checks all artifacts exist +- Cross-compilation uses GOOS/GOARCH with 5 targets: darwin-amd64, darwin-arm64, linux-amd64, linux-arm64, windows-amd64 + +## Plugin Registration +- Webapp entry point calls `window.registerPlugin('org-memory', new OrgMemoryPlugin())` +- Plugin class needs `initialize(registry, store)` and `uninitialize()` methods +- Server plugin struct embeds `plugin.MattermostPlugin` and main calls `plugin.ClientMain(&Plugin{})` diff --git a/agent-memories/org-memory-plugin-testing-patterns.md b/agent-memories/org-memory-plugin-testing-patterns.md new file mode 100644 index 000000000000..60b80b90e1b4 --- /dev/null +++ b/agent-memories/org-memory-plugin-testing-patterns.md @@ -0,0 +1,39 @@ +# Org-Memory Plugin Testing Patterns + +## Mock API for Plugin Tests +- Use `plugintest.API` (from `server/public/plugin/plugintest`) for mocking the plugin API +- Use `plugintest.NewDriver(t)` for mocking the database driver +- Use `pluginapi.NewClient(api, driver)` (NOT `plugintest.NewClient`) to create a client with mocks + +## LoadPluginConfiguration Mock Pattern +The mock's `LoadPluginConfiguration` checks if the return value is `func(interface{}) error` and calls it with the dest argument: +```go +func mockLoadPluginConfiguration(cfg *configuration) func(dest interface{}) error { + return func(dest interface{}) error { + data, _ := json.Marshal(cfg) + return json.Unmarshal(data, dest) + } +} +api.On("LoadPluginConfiguration", mock.AnythingOfType("*main.configuration")). + Return(mockLoadPluginConfiguration(cfg)) +``` + +## GetMasterDB Mock Pattern +To mock `client.Store.GetMasterDB()`: +```go +driver.On("Conn", true).Return("conn-1", nil) +driver.On("ConnPing", "conn-1").Return(nil) +``` +To make it fail: `driver.On("Conn", true).Return("", fmt.Errorf("connection refused"))` + +## Mock Expectations Tips +- Use `.Maybe()` for expectations that may or may not be called (e.g., `GetConfig`, `GetServerVersion`, logging) +- `RegisterCommand` returns `error` (not `*model.AppError`) through the pluginapi layer +- `LogInfo`, `LogError`, `LogWarn` take variadic args — mock with `mock.Anything` for each arg +- `SetAPI(api)` and `SetDriver(driver)` are methods on `plugin.MattermostPlugin` + +## Key Plugin Lifecycle Facts +- `OnConfigurationChange` is called BEFORE `OnActivate` on startup +- During that first call, `p.client` is nil, so must use `p.API.LoadPluginConfiguration` directly +- `pluginapi.NewClient` must be created exactly once per plugin instance in OnActivate +- `OnDeactivate` must handle partial initialization (nil checks on all resources) diff --git a/agent-memories/org-memory-step006-hooks.md b/agent-memories/org-memory-step006-hooks.md new file mode 100644 index 000000000000..3b09deff8c0b --- /dev/null +++ b/agent-memories/org-memory-step006-hooks.md @@ -0,0 +1,30 @@ +# Step 006: Event Ingestion Hooks - Implementation Notes + +## Key Implementation Details + +### Files Created/Modified +- **Created:** `server/plugins/org-memory/server/hooks.go` — All 5 hook implementations + helpers +- **Modified:** `server/plugins/org-memory/server/plugin.go` — Added scheduler fields, init/shutdown, enabled teams caching +- **Modified:** `server/plugins/org-memory/server/store.go` — Added `MarkCompleteByPostID` to ProcessingLogStore +- **Modified:** `server/plugins/org-memory/server/sqlstore.go` — Implemented `MarkCompleteByPostID` +- **Modified:** Test files — Added KVList, KVDelete, LogDebug mock expectations + +### Important Patterns +1. **JobOnceScheduler is a process-wide singleton** — `cluster.GetJobOnceScheduler()` returns the same instance across all calls. `Start()` fails with "already been started" on second call. OnActivate handles this gracefully by checking for "already" in the error message. +2. **Enabled teams caching** — Team ID set is cached on the Plugin struct under `configLock` and updated in both `OnActivate` and `OnConfigurationChange`. +3. **File entries in processing log** — File source type entries use the file ID (not post ID) as the `PostID` field to satisfy the UNIQUE constraint on `(PostId, JobType, SourceType)`. +4. **Text extraction** — Plugin implements its own text extraction for supported text file extensions since `server/platform/services/docextractor/` is an internal server package not accessible to plugins. + +### Test Mock Requirements +Tests that call `OnActivate()` need these mock expectations: +- `api.On("KVGet", mock.Anything).Return(nil, nil).Maybe()` +- `api.On("KVSetWithOptions", mock.Anything, mock.Anything, mock.Anything).Return(true, nil).Maybe()` +- `api.On("KVList", mock.Anything, mock.Anything).Return(nil, nil).Maybe()` +- `api.On("KVDelete", mock.Anything).Return(nil).Maybe()` +- `api.On("LogDebug", mock.Anything, mock.Anything, mock.Anything).Maybe()` + +### Build Command +```bash +cd server/plugins/org-memory/server && GOWORK=off go build ./... +cd server/plugins/org-memory/server && GOWORK=off go test -count=1 -short ./... +``` \ No newline at end of file diff --git a/agent-memories/org-memory-step007-worker.md b/agent-memories/org-memory-step007-worker.md new file mode 100644 index 000000000000..268ddc686e69 --- /dev/null +++ b/agent-memories/org-memory-step007-worker.md @@ -0,0 +1,43 @@ +# Step 007: Processing Worker Implementation Notes + +## Key Files +- `server/worker.go` — Processing worker implementation (workerCallback, processItem, handleEmbedJob, handleExtractJob, recoverStaleItems) +- `server/worker_test.go` — Comprehensive tests for the worker + +## Architecture +- Worker is scheduled via `cluster.Schedule` with 1-second interval in `OnActivate` +- `Plugin.processingJob` (type `interface{ Close() error }`) holds the scheduled job +- `Plugin.workerMetrics` (type `WorkerMetrics`) tracks atomic counters for processing stats +- Worker uses `clusterMutex` (custom per-post distributed mutex using KV store) with 2-second timeout + +## Important Test Patterns + +### Variadic Logging Mocks +The worker uses `p.client.Log.Debug/Info/Warn/Error()` with structured key-value pairs, producing variadic calls with 1 to 13+ arguments through the plugintest.API mock. Standard 3-arg mock expectations fail. + +**Solution**: `setupVariadicLogMocks(api *plugintest.API)` registers mock expectations for argument counts 1 through 15 for all 4 log levels. Defined in `worker_test.go` but accessible across all test files in the package. + +### Background Worker Goroutine in Tests +`cluster.Schedule` starts a background goroutine that calls `workerCallback()`. In tests that call `OnActivate`, this goroutine may fire and call `ConnQuery` on the mock driver. + +**Solution**: +1. `mockDriverForMigrations(driver)` adds `ConnQuery` mock returning `("", error)` and `ConnClose` mock +2. Tests that bypass `mockDriverForMigrations` must add their own `ConnQuery` mock +3. `ConnQuery` returns `(string, error)` not `(plugin.ResultContainer, error)` — returning an error prevents the need to mock full Rows lifecycle + +### Test Cleanup +Tests with successful `OnActivate` should defer `p.processingJob.Close()` to stop the background goroutine. + +## Build Command +Always use `GOWORK=off` when building/testing the plugin: +``` +cd server/plugins/org-memory/server && GOWORK=off go test -count=1 -short ./... +``` + +## Constants +- `workerBatchSize = 10` +- `workerMaxRetries = 3` +- `workerStaleThresholdMs = 300000` (5 minutes) +- `workerMutexTimeout = 2 * time.Second` +- `workerMutexPrefix = "org_memory_post_lock_"` +- `workerThreadContextLimit = 10` diff --git a/agent-memories/org-memory-step008-debounce.md b/agent-memories/org-memory-step008-debounce.md new file mode 100644 index 000000000000..00f69c82903f --- /dev/null +++ b/agent-memories/org-memory-step008-debounce.md @@ -0,0 +1,21 @@ +# Step 008: Thread Debounce and Summarization - Implementation Notes + +## Files Created/Modified +- **Created:** `server/plugins/org-memory/server/debounce.go` — Full debounce handler: parseProps, filterThreadPosts, enrichPostsWithUsernames, buildThreadConversationText, summarizeLargeThread (map-reduce), storeThreadSummary, storeThreadExtractions, queueUnprocessedPostEmbeddings, createSummaryProcessingEntry, handleDebounceGeminiError, publishThreadSummaryEvent +- **Created:** `server/plugins/org-memory/server/debounce_test.go` — 30+ tests covering all paths +- **Modified:** `server/plugins/org-memory/server/hooks.go` — Removed placeholder `handleDebounceJob`; it's now in debounce.go +- **Modified:** `server/plugins/org-memory/server/worker.go` — Added "summary" source type to `handleEmbedJob` switch statement. Loads summary text from ThreadSummaryStore for embedding. + +## Key Design Decisions +1. **Separate file** — All debounce logic in `debounce.go` to keep hooks.go clean. The `handleDebounceJob` method is called from `dispatchScheduledJob` in hooks.go. +2. **Props parsing** — Props arrive as `map[string]string` when first scheduled but `map[string]interface{}` after KV deserialization. The parser handles both. +3. **Retry via rescheduling** — On Gemini failure, the job is rescheduled (not retried inline) using the same `JobOnceScheduler.ScheduleOnce`. Retry count is tracked in props. +4. **Map-reduce for large threads** — Threads >100 messages are split into groups of 50, each summarized separately, then group summaries synthesized into a final result. Extraction items from all groups are merged. +5. **Summary embedding** — A processing log entry with `SourceType="summary"` is created. The worker's `handleEmbedJob` loads the summary from `ThreadSummaryStore.GetByRootID()`. +6. **WebSocket event** — Event name `thread_summary_ready` is passed to `PublishWebSocketEvent`. Plugin API auto-prefixes it to `custom_org-memory_thread_summary_ready`. + +## Build & Test +```bash +cd server/plugins/org-memory/server && GOWORK=off go build ./... +cd server/plugins/org-memory/server && GOWORK=off go test -count=1 -short ./... +``` \ No newline at end of file diff --git a/agent-memories/org-memory-step009-rag-engine.md b/agent-memories/org-memory-step009-rag-engine.md new file mode 100644 index 000000000000..dc1a1494d9c8 --- /dev/null +++ b/agent-memories/org-memory-step009-rag-engine.md @@ -0,0 +1,25 @@ +# Step 009: RAG Query Engine + +## Key Files +- `server/plugins/org-memory/server/rag_engine.go` — RAG engine implementation +- `server/plugins/org-memory/server/rag_engine_test.go` — 34 unit tests + +## Architecture +- `RAGEngine` interface with `Query()` and `FollowUp()` operations — mockable for step 010 +- `ragEngine` struct holds `plugin.API` (raw API for GetPost/GetUser/etc), `pluginapi.Client` (for KV/Log), `Store`, and `GeminiClient` +- Constructor: `NewRAGEngine(pluginAPI plugin.API, client *pluginapi.Client, store Store, geminiClient GeminiClient)` + +## Key Design Decisions +- Uses `plugin.API` directly for Mattermost CRUD (GetPost, GetUser, GetTeamsForUser, etc.) — NOT `pluginapi.Client` which doesn't expose these +- Uses `pluginapi.Client` for KV store (`client.KV.Get/Set`) and logging (`client.Log`) +- KV Set with TTL: `client.KV.Set(key, value, pluginapi.SetExpiry(duration))` — returns `(bool, error)` +- Permission cache key: `perms:{userId}` with 60s TTL +- Session key: `session:{userId}:{sessionId}` with 30min TTL +- Citation formatting uses regex `\[(\d+)\]` to replace markers with `[N](permalink)` markdown links +- Uses `model.PermissionReadChannelContent` (NOT `PermissionReadChannel`) — per spec requirement for content-level permission check + +## Gotchas +- `pluginapi.KVService` has no `SetWithExpiry` method — use `Set()` with `pluginapi.SetExpiry()` option +- `pluginapi.Client` has no `API` field — must pass `plugin.API` separately +- Test helper `removeExpectedCallByMethod` is defined in `worker_test.go` and shared across test files — don't redefine it +- `model.PermissionReadChannelContent` is the correct permission constant (NOT `PermissionReadChannel`) — the spec explicitly requires content-level permission diff --git a/agent-memories/org-memory-step010-http-api.md b/agent-memories/org-memory-step010-http-api.md new file mode 100644 index 000000000000..9257822bfac8 --- /dev/null +++ b/agent-memories/org-memory-step010-http-api.md @@ -0,0 +1,30 @@ +# Step 010: HTTP API Router and Slash Command Handler + +## Files Created +- `server/plugins/org-memory/server/api.go` — Main HTTP API implementation +- `server/plugins/org-memory/server/api_test.go` — Comprehensive test suite + +## Files Modified +- `server/plugins/org-memory/server/plugin.go` — Updated /ask command registration + +## Key Patterns +- Plugin source lives in `server/plugins/org-memory/server/` (not the root) +- Go module: `github.com/mattermost/mattermost-plugin-org-memory/server` +- Must use `GOWORK=off` when running `go test` or `go vet` due to parent go.work +- Mock store, Gemini client, and variadic log mocks are defined in `worker_test.go` +- Tests use `plugintest.API`, `plugintest.NewDriver(t)`, and `pluginapi.NewClient()` +- Plugin HTTP handlers receive paths with `/plugins/{plugin_id}` already stripped + +## API Structure +- 13 endpoints under `/api/v1/` +- Authentication via `Mattermost-User-Id` header (set by server after session validation) +- Admin endpoints check `HasPermissionTo(userId, PermissionManageSystem)` +- Channel endpoints check `HasPermissionToChannel(userId, channelId, PermissionReadChannelContent)` +- KV store used for: webhook configs (prefix `webhook:`), backfill progress (prefix `backfill_progress:`) +- RAG engine created per-request via `NewRAGEngine(p.API, p.client, p.store, p.geminiClient)` + +## Test Patterns +- `setupAPITestPlugin()` creates a Plugin with all mocked dependencies +- `executeHTTPRequest()` builds and executes httptest requests through ServeHTTP +- `filterExpectedCalls()` helper to re-mock specific KVGet calls for different test scenarios +- All mock stores reused from `worker_test.go` (same package `main`) diff --git a/agent-memories/org-memory-step011-channel-summary-worker.md b/agent-memories/org-memory-step011-channel-summary-worker.md new file mode 100644 index 000000000000..2d36da5c1b5c --- /dev/null +++ b/agent-memories/org-memory-step011-channel-summary-worker.md @@ -0,0 +1,40 @@ +# Step 011: Channel Summary Worker + +## What was built +- **channel_summary_worker.go** — Daily and weekly channel summary generation workers +- Daily worker: custom `makeWaitForDailySummaryHour()` targeting configured hour (default 8 AM) +- Weekly worker: `MakeWaitForRoundedInterval(168 * time.Hour)` +- Both use `cluster.Schedule` for single-node execution + +## Key patterns +- `cluster.NextWaitInterval` signature: `func(now time.Time, metadata cluster.JobMetadata) time.Duration` +- `cluster.JobMetadata` has `LastFinished time.Time` — check `.IsZero()` for first run +- First run fires immediately by returning `time.Duration(0)` when `LastFinished.IsZero()` +- `channelSummaryJobs` struct holds both `*cluster.Job` handles with `Close()` method + +## Hierarchical summarization +- Daily: thread summaries + extractions → channel daily summary +- Weekly: daily summaries → channel weekly summary +- Source material is assembled as structured text with thread citations `[Thread: post_id]` + +## Store methods added +- `ThreadSummaryStore.ListActiveChannelIDsInWindow` — finds channels with activity +- `ExtractionStore.ListByChannelIDInWindow` — gets extractions for a channel+window +- `ChannelSummaryStore.ListByChannelAndTypeInWindow` — gets daily summaries for weekly rollup +- `ChannelSummaryStore.ListActiveChannelIDsByTypeInWindow` — finds channels with daily summaries + +## GeminiClient addition +- `SynthesizeChannelSummary(ctx, sourceText, channelName, summaryType string) (string, error)` +- Uses temperature 0.2, llmTimeout (60s), topic-organized system prompt + +## Gotchas +- `getChannelDisplayName` already existed in api.go — created `getChannelNameForSummary` wrapper to avoid duplicate method declaration +- Duplicate key errors (PostgreSQL 23505) handled gracefully — `isDuplicateKeyError()` helper +- Processing log entry uses generated summary ID as PostID so embedding worker can find it +- WebSocket events: `p.API.PublishWebSocketEvent` auto-prepends `custom_` prefix, so pass `org-memory_channel_summary_ready` +- Build requires `GOWORK=off` due to parent `server/go.work` + +## Build & test +- `cd server/plugins/org-memory && GOWORK=off go build ./...` +- `cd server/plugins/org-memory && GOWORK=off go test ./...` +- `make verify` — cross-platform build for all 5 targets \ No newline at end of file diff --git a/agent-memories/org-memory-step012-webhook-dispatcher.md b/agent-memories/org-memory-step012-webhook-dispatcher.md new file mode 100644 index 000000000000..32cd61975df1 --- /dev/null +++ b/agent-memories/org-memory-step012-webhook-dispatcher.md @@ -0,0 +1,44 @@ +# Step 012: Outbound Webhook Dispatcher + +## Files Created/Modified +- **server/webhook.go** (NEW) — Core webhook dispatcher with: + - `WebhookDispatcher` struct with pluginapi client, HTTP client, and failed delivery tracking (sync.Map) + - `WebhookPayload`, `WebhookExtractionItem`, `WebhookMetadata` types + - `ExtractionEvent` type (input from worker/debounce) + - CRUD: `CreateWebhook`, `ListWebhooks`, `GetWebhookByID`, `DeleteWebhook`, `FindWebhooksByChannel` + - `DispatchWebhooks` — groups by channel, finds matching webhooks, filters by type, dispatches concurrently + - `deliverToWebhook` — sends HTTP POST with retries (1s, 2s, 4s backoff), 30s timeout + - `computeHMACSignature` — HMAC-SHA256 hex-encoded + - `sendHTTPPost` — HTTP POST with Content-Type JSON, optional HMAC header, response size limit + - `recordFailure`, `GetFailedDeliveryCount`, `GetTotalFailedDeliveries` — in-memory atomic counters + - `redactURL` — removes query params for safe logging + +- **server/webhook_test.go** (NEW) — 40 tests covering HMAC, CRUD, dispatch, retries, concurrency, payloads, failure tracking + +- **server/api.go** (MODIFIED): + - Added `CreatedBy` field to `webhookConfig` struct + - Updated `handleWebhookCreate` to use dispatcher when available, sets CreatedBy + +- **server/plugin.go** (MODIFIED): + - Added `webhookDispatcher *WebhookDispatcher` field to Plugin struct + - Initialize in OnActivate step 15 + +- **server/worker.go** (MODIFIED): + - Added webhook dispatch after `handleExtractJob` stores extractions (fire-and-forget) + +- **server/debounce.go** (MODIFIED): + - Added webhook dispatch after `storeThreadExtractions` stores thread-level extractions + +## Key Design Decisions +- Dispatcher is a separate struct (not embedded in Plugin) for testability +- Uses sync.Map for thread-safe failed delivery counters +- HTTP client configured with 30s timeout and no redirect following +- KV store prefix `webhook:{channelId}:{webhookId}` for channel-scoped lookups +- Retries: 4 total attempts (1 original + 3 retries) with exponential backoff +- Dispatches concurrently to multiple webhooks using sync.WaitGroup +- Empty extraction list returns immediately (no KV lookups) +- Delivery failures don't affect processing log status + +## Build/Test +- `GOWORK=off go build .` — compiles clean +- `GOWORK=off go test -count=1 -short .` — all tests pass (including 40 new webhook tests) diff --git a/agent-memories/org-memory-step013-data-retention.md b/agent-memories/org-memory-step013-data-retention.md new file mode 100644 index 000000000000..ec8065facb3b --- /dev/null +++ b/agent-memories/org-memory-step013-data-retention.md @@ -0,0 +1,33 @@ +# Step 013: Data Retention and Lifecycle Cleanup + +## What Was Implemented +- `RunDataRetention(nowTime, batchSize int64) (int64, error)` hook on the Plugin struct +- 7 new `DataRetentionStore` interface methods (replaced the old `DeleteBefore` method) +- Batch budget distribution algorithm that proportionally allocates across 7 cleanup operations with rollover + +## DataRetentionStore Interface Methods +1. `DeleteOrphanedEmbeddings(limit int)` - LEFT JOIN Posts, WHERE p.Id IS NULL +2. `DeleteOrphanedExtractions(limit int)` - Same pattern +3. `DeleteOrphanedEntityMentions(limit int)` - Same pattern +4. `DeleteOrphanedThreadSummaries(limit int)` - Uses RootId instead of Id as PK +5. `DeleteAgedChannelSummaries(dailyCutoff, weeklyCutoff int64, limit int)` - Daily >90d, Weekly >365d by WindowEnd +6. `DeleteCompletedProcessingLogs(cutoff int64, limit int)` - Only "complete"/"error" status, >30d +7. `DeleteAgedQueryLogs(cutoff int64, limit int)` - All entries >90d + +## Key Design Decisions +- **Batch budget distribution**: `remaining / opsLeft` gives proportional allocation with rollover +- **Orphan detection via LEFT JOIN**: `LEFT JOIN Posts p ON e.PostId = p.Id WHERE p.Id IS NULL` +- **Subquery batch pattern**: `DELETE FROM table WHERE Id IN (SELECT Id FROM table WHERE cond LIMIT $1)` +- **Retention constants are compile-time**: Not user-configurable (90d daily, 365d weekly, 30d processing, 90d query) + +## Files Modified +- `store.go` - Replaced DataRetentionStore interface (7 methods instead of 1) +- `sqlstore.go` - Replaced implementation section with 7 new methods +- `hooks.go` - Added RunDataRetention hook + retention constants +- `worker_test.go` - Updated mockDataRetentionStore to match new interface +- `data_retention_test.go` - New test file with 25 tests + +## Build/Test +- `GOWORK=off go vet .` and `GOWORK=off go build .` from `/server/plugins/org-memory/server/` +- `GOWORK=off go test -count=1 .` runs all tests +- go.work is at `/server/go.work` which doesn't include the plugin, so GOWORK=off is required diff --git a/agent-memories/org-memory-step014-metrics.md b/agent-memories/org-memory-step014-metrics.md new file mode 100644 index 000000000000..ffa9dd8a1e76 --- /dev/null +++ b/agent-memories/org-memory-step014-metrics.md @@ -0,0 +1,47 @@ +# Step 014: Metrics and Observability + +## What was built +- `metrics.go` — Prometheus metrics collector with interface, no-op impl, and concrete impl +- `metrics_test.go` — Comprehensive test suite for all metrics methods + +## Key design decisions +- **Isolated registry**: Uses `prometheus.NewRegistry()` (not the global default) to avoid collisions +- **Namespace**: All metrics use `org_memory` namespace, producing names like `org_memory_processing_items_total` +- **Interface-based**: `MetricsCollector` interface enables no-op fallback and testing +- **GaugeFunc for live queries**: Queue depth, stale items, total embeddings use GaugeFunc callbacks that query the replica DB at scrape time +- **GaugeFunc safety**: All callbacks handle nil DB and errors gracefully (return 0) +- **Thread-safe circuit breaker state**: Uses `atomic.Int32` for concurrent access + +## Important: Column name in GaugeFunc +The `ai_processing_log` table does NOT have a `StartedAt` column. The stale items callback uses `CreateAt` to detect stale items, matching the `FindStale` store method. Table columns: Id, PostId, JobType, SourceType, Status, ScheduleAt, ErrorMessage, RetryCount, CreateAt, CompleteAt. + +## Circuit breaker state mapping +- CircuitClosed → 0 +- CircuitHalfOpen → 1 +- CircuitOpen → 2 +Helper: `circuitStateToInt()` converts `CircuitState` string to int. + +## Metrics inventory +- **Processing**: items_total (CounterVec), errors_total (CounterVec), duration_seconds (HistogramVec), queue_depth (GaugeFunc), stale_items (GaugeFunc), items_ingested_total (Counter) +- **Gemini**: requests_total (CounterVec: operation, status), request_duration_seconds (HistogramVec), rate_limit_hits_total (Counter), circuit_breaker_state (Gauge) +- **RAG**: queries_total (CounterVec: type), query_duration_seconds (Histogram), search_results (Histogram), permission_cache_total (CounterVec), session_total (CounterVec), no_results_total (Counter) +- **Embedding**: generated_total (Counter), chunks_per_post (Histogram), total_stored (GaugeFunc) +- **Summary**: generated_total (CounterVec: type), channels_skipped_total (Counter), batch_duration_seconds (Histogram) +- **Webhook**: deliveries_total (CounterVec: status), delivery_duration_seconds (Histogram) +- **Retention**: rows_deleted_total (CounterVec: table) + +## How to build +```bash +cd server/plugins/org-memory/server +GOWORK=off go build ./... +GOWORK=off go test ./... +``` + +## Plugin struct changes +- `metricsCollector` field: `MetricsCollector` interface (was `interface{}`) +- `metricsHandler http.Handler` field for ServeMetrics hook reuse +- OnActivate step 16 initializes metrics, falls back to no-op on failure +- `ServeMetrics` hook delegates to stored handler + +## prometheus dependency +Added `github.com/prometheus/client_golang` (v1.23.2) to go.mod. \ No newline at end of file diff --git a/agent-memories/org-memory-step017-backfill.md b/agent-memories/org-memory-step017-backfill.md new file mode 100644 index 000000000000..3ae8cd97d2bd --- /dev/null +++ b/agent-memories/org-memory-step017-backfill.md @@ -0,0 +1,34 @@ +# Step 017: Historical Backfill System + +## Files +- `server/backfill.go` — Core backfill handler, post fetching, thread tracking, progress estimation +- `server/api.go` — backfillRequest/backfillProgress types, start/pause/resume handlers, admin status enrichment +- `server/hooks.go` — dispatchScheduledJob routes `backfill:` prefix to handleBackfillJob +- `server/metrics.go` — 4 backfill metrics (batches_processed, posts_scanned, posts_queued, duration) + +## Architecture +- Backfill does NOT call Gemini — only creates processing log entries for the existing worker (step 007) +- Uses `JobOnceScheduler` with `backfill:{channelId}` key prefix for batch scheduling +- Compound cursor `(CreateAt, PostId)` for deterministic resumable pagination +- Initial batch uses `GetPostsSince`, subsequent batches use `GetPostsAfter` +- Thread roots tracked in separate KV key `backfill_threads:{channelId}` +- Progress persisted in KV under `backfill_progress:{channelId}` + +## Key Constants +- Default batch size: 100 +- Default delay: 500ms +- Max consecutive errors before stopping: 10 +- Retry delay: 5000ms + +## Gotchas +- `JobOnceScheduler` is a process-wide singleton (`sync.Once`) — test setup must handle this +- Pause race condition: batch handler re-checks KV status before saving to avoid overwriting admin's pause +- Nil scheduler check must happen before storing any state to avoid stuck "running" backfills +- `GOWORK=off` required for building due to workspace configuration + +## Build & Test +```bash +cd server/plugins/org-memory/server +GOWORK=off go build ./... +GOWORK=off go test ./... +``` \ No newline at end of file diff --git a/agent-memories/org-memory-step018-testing.md b/agent-memories/org-memory-step018-testing.md new file mode 100644 index 000000000000..19b0cef9ef74 --- /dev/null +++ b/agent-memories/org-memory-step018-testing.md @@ -0,0 +1,38 @@ +# Step 018 - Server-side Integration Tests + +## Plugin Module Location +- Module: `github.com/mattermost/mattermost-plugin-org-memory/server` +- go.mod: `/Users/pasivuorio/modernpath/mattermost/server/plugins/org-memory/server/go.mod` +- Must use `GOWORK=off` when running go commands because the server's go.work doesn't include this plugin module + +## Build & Test Commands +```bash +cd /Users/pasivuorio/modernpath/mattermost/server/plugins/org-memory/server +GOWORK=off go test -v -count=1 -timeout 120s . # Run all unit tests +GOWORK=off go test -v -count=1 -tags integration . # Run integration tests (needs DB) +GOWORK=off go vet . # Lint check +``` + +## Key Pattern: pluginapi KV.Set() calls api.KVSetWithOptions() +The `p.client.KV.Set(key, value)` method (from `pluginapi`) internally calls `api.KVSetWithOptions(key, valueBytes, model.PluginKVSetOptions{})`. So when asserting KV writes in tests, always use: +```go +api.AssertCalled(t, "KVSetWithOptions", key, value, mock.Anything) +``` +NOT `api.AssertCalled(t, "KVSet", ...)`. + +Similarly, `p.client.KV.Get(key, &value)` internally calls `api.KVGet(key)`. + +## Test Files Created (Step 018) +- `testhelper_test.go` (314 lines) - TestHelper struct, 9 factory methods, 4 vector helpers +- `hooks_test.go` (1200 lines) - 42 tests for all 5 event hooks +- `channel_summary_worker_test.go` (845 lines) - 41 tests for daily/weekly summary workers +- `backfill_test.go` (1262 lines) - 44 tests for historical backfill system +- `integration_test.go` (788 lines) - 11 integration tests gated by `//go:build integration` + +## Pre-existing Test Files (16 files, ~9,334 lines) +worker_test.go, api_test.go, plugin_test.go, configuration_test.go, debounce_test.go, +rag_engine_test.go, webhook_test.go, chunker_test.go, circuit_breaker_test.go, +rate_limiter_test.go, gemini_client_test.go, gemini_errors_test.go, +gemini_integration_test.go, metrics_test.go, store_test.go, data_retention_test.go + +## Total: 451 passing tests across 21 test files \ No newline at end of file diff --git a/docs/screenshots/csrf-fix-auth-works.png b/docs/screenshots/csrf-fix-auth-works.png new file mode 100644 index 000000000000..ea20507c57d0 Binary files /dev/null and b/docs/screenshots/csrf-fix-auth-works.png differ diff --git a/docs/screenshots/plugin-config-system-console.png b/docs/screenshots/plugin-config-system-console.png new file mode 100644 index 000000000000..ecd39a080808 Binary files /dev/null and b/docs/screenshots/plugin-config-system-console.png differ diff --git a/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/mock-org-memory-api.ts b/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/mock-org-memory-api.ts new file mode 100644 index 000000000000..6d5388849fe2 --- /dev/null +++ b/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/mock-org-memory-api.ts @@ -0,0 +1,271 @@ +// Shared route interception helpers for the org-memory plugin E2E tests. +// Provides reusable functions for mocking the plugin's API endpoints +// using Playwright's page.route() API, eliminating dependency on +// a real Gemini API key or server-side processing. + +import type {Page} from '@playwright/test'; + +// -- Route URL patterns -- + +const QUERY_ENDPOINT = '**/plugins/org-memory/api/v1/query'; +const FOLLOWUP_ENDPOINT = '**/plugins/org-memory/api/v1/query/followup'; +const THREAD_SUMMARY_GLOB = '**/plugins/org-memory/api/v1/threads/*/summary'; +const COMMANDS_ENDPOINT = '**/api/v4/commands/execute'; + +// -- Default response factories -- + +/** + * Default mock response for a query — includes answer, citations, and session. + */ +export function defaultQueryResponse(overrides: Record = {}) { + return { + answer: 'Based on the workspace conversations, the API was redesigned in Q3 to use REST endpoints with OpenAPI documentation [1]. The team agreed on versioned endpoints [2].', + citations: [ + { + ref_number: 1, + post_id: 'mock-post-1', + permalink: '/test-team/pl/mock-post-1', + channel_name: 'engineering', + author: 'alice', + snippet: 'We decided to redesign the API using REST.', + }, + { + ref_number: 2, + post_id: 'mock-post-2', + permalink: '/test-team/pl/mock-post-2', + channel_name: 'architecture', + author: 'bob', + snippet: 'Versioned endpoints are the way to go.', + }, + ], + session_id: 'mock-session-001', + ...overrides, + }; +} + +/** + * Default mock response for a follow-up query. + */ +export function defaultFollowUpResponse(overrides: Record = {}) { + return { + answer: 'The REST API redesign was completed in October. The main changes included switching from GraphQL to REST, adding OpenAPI specs, and implementing versioned endpoints [1].', + citations: [ + { + ref_number: 1, + post_id: 'mock-post-3', + permalink: '/test-team/pl/mock-post-3', + channel_name: 'engineering', + author: 'charlie', + snippet: 'Migration from GraphQL completed.', + }, + ], + session_id: 'mock-session-001', + ...overrides, + }; +} + +/** + * Default mock thread summary response. + */ +export function defaultThreadSummaryResponse(rootId: string, overrides: Record = {}) { + return { + root_id: rootId, + channel_id: 'mock-channel-1', + summary: 'The team discussed deployment strategy and agreed on blue-green deployments.', + key_takeaways: [ + 'Blue-green deployment was chosen', + 'Rollback plan documented', + ], + extractions: [ + {type: 'decision', content: 'Use blue-green deployment', confidence: 0.95}, + {type: 'action_item', content: 'Document rollback procedures', confidence: 0.88}, + ], + post_count: 12, + generated_at: new Date().toISOString(), + ...overrides, + }; +} + +// -- Internal helpers -- + +/** + * Generic route interceptor: filters by HTTP method, applies optional delay, + * and fulfills with a JSON response body. Extracted to eliminate duplication + * across the individual mock functions. + */ +async function interceptJsonRoute( + page: Page, + urlPattern: string, + method: string, + body: Record, + delay?: number, +): Promise { + await page.route(urlPattern, async (route) => { + if (route.request().method() !== method) { + await route.fallback(); + return; + } + + if (delay) { + await new Promise((resolve) => setTimeout(resolve, delay)); + } + + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(body), + }); + }); +} + +// -- Public mock functions -- + +/** + * Intercept the query endpoint and return a mock response. + * + * @param page - Playwright page instance + * @param response - The mock response to return (defaults to defaultQueryResponse) + * @param delay - Optional delay in ms before fulfilling (useful for testing loading states) + */ +export async function mockQueryResponse( + page: Page, + response?: Record, + delay?: number, +): Promise { + await interceptJsonRoute(page, QUERY_ENDPOINT, 'POST', response || defaultQueryResponse(), delay); +} + +/** + * Intercept the follow-up query endpoint and return a mock response. + * + * @param page - Playwright page instance + * @param response - The mock response to return (defaults to defaultFollowUpResponse) + * @param delay - Optional delay in ms before fulfilling + */ +export async function mockFollowupResponse( + page: Page, + response?: Record, + delay?: number, +): Promise { + await interceptJsonRoute(page, FOLLOWUP_ENDPOINT, 'POST', response || defaultFollowUpResponse(), delay); +} + +/** + * Intercept the thread summary endpoint for a specific root ID and return a mock response. + * + * @param page - Playwright page instance + * @param rootId - The root post ID to match in the URL + * @param summary - Optional custom summary response + */ +export async function mockThreadSummary( + page: Page, + rootId: string, + summary?: Record, +): Promise { + await interceptJsonRoute( + page, + `**/plugins/org-memory/api/v1/threads/${rootId}/summary`, + 'GET', + summary || defaultThreadSummaryResponse(rootId), + ); +} + +/** + * Intercept the query endpoint with a controlled promise that allows + * the test to observe loading state before resolving. + * + * @param page - Playwright page instance + * @param response - The mock response to return when resolved + * @returns An object with `resolve` and `reject` functions to control fulfillment + */ +export async function mockQueryResponseDeferred( + page: Page, + response?: Record, +): Promise<{resolve: () => void; reject: (error: string) => void}> { + const body = response || defaultQueryResponse(); + let resolvePromise: () => void; + let rejectPromise: (error: string) => void; + + const deferredPromise = new Promise((resolve, reject) => { + resolvePromise = resolve; + rejectPromise = (error: string) => reject(new Error(error)); + }); + + await page.route(QUERY_ENDPOINT, async (route) => { + if (route.request().method() !== 'POST') { + await route.fallback(); + return; + } + + try { + await deferredPromise; + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(body), + }); + } catch { + await route.fulfill({ + status: 500, + contentType: 'application/json', + body: JSON.stringify({error: 'Internal Server Error'}), + }); + } + }); + + return { + resolve: resolvePromise!, + reject: rejectPromise!, + }; +} + +/** + * Intercept slash command execution and return a mock ephemeral response. + * Only intercepts POST commands starting with '/ask'; all other commands fall through. + * + * @param page - Playwright page instance + * @param responseText - The text to include in the ephemeral response + */ +export async function mockSlashCommandResponse( + page: Page, + responseText?: string, +): Promise { + const text = responseText || 'Based on workspace conversations, the API decisions include REST endpoints and versioned APIs. [1](/test-team/pl/mock-post-1)'; + + await page.route(COMMANDS_ENDPOINT, async (route) => { + const request = route.request(); + if (request.method() !== 'POST') { + await route.fallback(); + return; + } + + try { + const postData = request.postDataJSON(); + if (postData?.command?.startsWith('/ask')) { + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ + response_type: 'ephemeral', + text, + }), + }); + return; + } + } catch { + // Not our command, fall through + } + + await route.fallback(); + }); +} + +/** + * Remove all org-memory API route interceptors. + */ +export async function unmockAllRoutes(page: Page): Promise { + await page.unroute(QUERY_ENDPOINT); + await page.unroute(FOLLOWUP_ENDPOINT); + await page.unroute(THREAD_SUMMARY_GLOB); + await page.unroute(COMMANDS_ENDPOINT); +} diff --git a/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/test-setup.ts b/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/test-setup.ts new file mode 100644 index 000000000000..b0155b19a7cd --- /dev/null +++ b/e2e-tests/playwright/specs/functional/plugins/org_memory/helpers/test-setup.ts @@ -0,0 +1,67 @@ +// Shared test setup and UI interaction helpers for org-memory plugin E2E tests. +// Eliminates duplicated initialization logic across spec files. + +import type {Page, Locator} from '@playwright/test'; +import {test, expect} from '@mattermost/playwright-lib'; + +export const PLUGIN_ID = 'org-memory'; + +/** + * Initialize test environment with plugin active and channels page visible. + * Configures server settings, ensures the org-memory plugin is enabled, + * navigates to the channels page, and dismisses onboarding overlays. + */ +export async function setupOrgMemoryTest(pw: any) { + const {adminClient, user, team, townSquareUrl} = await pw.initSetup(); + const {channelsPage, page} = await pw.testBrowser.login(user); + + await adminClient.patchConfig({ + PluginSettings: {Enable: true, EnableUploads: true}, + ServiceSettings: {EnableOnboardingFlow: false, EnableTutorial: false}, + }); + + let pluginActive = false; + try { + pluginActive = await pw.isPluginActive(adminClient, PLUGIN_ID); + } catch { + // Plugin not installed + } + if (!pluginActive) { + try { + await adminClient.enablePlugin(PLUGIN_ID); + } catch { + test.skip(true, 'org-memory plugin not available'); + } + } + + await channelsPage.goto(); + await channelsPage.toBeVisible(); + + await page.keyboard.press('Escape'); + await page.waitForTimeout(500); + + return {adminClient, user, team, townSquareUrl, channelsPage, page}; +} + +/** + * Open the org-memory RHS panel via the app bar icon. + * Waits for the panel to become visible before returning. + * + * @returns The RHS container locator for further assertions. + */ +export async function openOrgMemoryRhs(page: Page, channelsPage: any): Promise { + const appBarIcon = page.locator(`#app-bar-icon-${PLUGIN_ID}`); + await expect(appBarIcon).toBeVisible({timeout: 10000}); + await appBarIcon.click(); + await channelsPage.sidebarRight.toBeVisible(); + return page.locator('#sidebar-right'); +} + +/** + * Type a query into the RHS textarea and press Enter to submit. + */ +export async function submitQueryInRhs(rhsContainer: Locator, query: string): Promise { + const queryInput = rhsContainer.locator('textarea').first(); + await queryInput.fill(query); + await queryInput.press('Enter'); +} diff --git a/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_commands.spec.ts b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_commands.spec.ts new file mode 100644 index 000000000000..c2cb715455ea --- /dev/null +++ b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_commands.spec.ts @@ -0,0 +1,89 @@ +// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved. +// See LICENSE.txt for license information. + +// E2E tests for the org-memory plugin slash command and WebSocket events. +// Covers /ask slash command execution and thread summary WebSocket indicators. +// All tests use route interception — no real Gemini API key required. + +import {test, expect} from '@mattermost/playwright-lib'; + +import { + mockSlashCommandResponse, + mockQueryResponse, +} from './helpers/mock-org-memory-api'; +import {setupOrgMemoryTest, PLUGIN_ID} from './helpers/test-setup'; + +test.describe('Org Memory slash command and WebSocket @org-memory', () => { + test('slash command /ask produces an ephemeral post with response and citations', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + await mockSlashCommandResponse(page, + 'Based on workspace conversations, the API decisions include REST endpoints and versioned APIs. [1](/test-team/pl/mock-post-1)', + ); + + await channelsPage.postMessage('/ask what decisions were made about the API?'); + + const lastPost = await channelsPage.getLastPost(); + await lastPost.toBeVisible(); + await lastPost.toContainText('API decisions'); + + const postContainer = lastPost.container; + const citationLink = postContainer.locator('a[href*="/test-team/pl/mock-post-1"]'); + await expect(citationLink).toBeVisible({timeout: 5000}); + }); + + test('WebSocket thread summary event updates Redux state', async ({pw}) => { + const {adminClient, channelsPage, page, team} = await setupOrgMemoryTest(pw); + + await mockQueryResponse(page); + + // Create a thread (root post + reply) + const channel = await adminClient.getChannelByName(team.id, 'town-square'); + const rootPost = await adminClient.createPost({ + channel_id: channel.id, + message: 'Thread root post for summary testing', + }); + await adminClient.createPost({ + channel_id: channel.id, + message: 'Reply to thread', + root_id: rootPost.id, + }); + + // Reload to see the new posts + await channelsPage.goto(); + await channelsPage.toBeVisible(); + await page.keyboard.press('Escape'); + await page.waitForTimeout(500); + + // Open the RHS so the plugin's WebSocket handler is active + const appBarIcon = page.locator(`#app-bar-icon-${PLUGIN_ID}`); + await expect(appBarIcon).toBeVisible({timeout: 10000}); + await appBarIcon.click(); + await page.locator('#sidebar-right').waitFor({state: 'visible'}); + + // Simulate a WebSocket event by dispatching the Redux action directly. + // This mimics what the WebSocket handler does when the server sends + // custom_org-memory_thread_summary_ready. + await page.evaluate((rootId: string) => { + const store = (window as any).store || (window as any).__store; + if (store) { + store.dispatch({ + type: 'org-memory/THREAD_SUMMARY_AVAILABLE', + rootId, + }); + } + }, rootPost.id); + + // Verify the thread summary state was updated in Redux + const summaryAvailable = await page.evaluate((rootId: string) => { + const store = (window as any).store || (window as any).__store; + if (!store) { + return false; + } + const state = store.getState(); + return state['plugins-org-memory']?.threadSummariesAvailable?.[rootId] === true; + }, rootPost.id); + + expect(summaryAvailable).toBe(true); + }); +}); diff --git a/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_installation.spec.ts b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_installation.spec.ts new file mode 100644 index 000000000000..a34a2b1598af --- /dev/null +++ b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_installation.spec.ts @@ -0,0 +1,74 @@ +// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved. +// See LICENSE.txt for license information. + +// E2E test: Plugin installation and activation. +// Verifies the org-memory plugin can be installed and activated, +// and the app bar icon appears in the UI. +// Tagged @smoke @org-memory for selective test execution. + +import {test, expect} from '@mattermost/playwright-lib'; + +const PLUGIN_ID = 'org-memory'; + +test.describe('Org Memory plugin installation @smoke @org-memory', () => { + test('should install and activate plugin, and show app bar icon', async ({pw}) => { + // 1. Initialize test environment with a fresh user and team + const {adminClient, user} = await pw.initSetup(); + const {channelsPage, page} = await pw.testBrowser.login(user); + + // 2. Configure server settings + await adminClient.patchConfig({ + PluginSettings: { + Enable: true, + EnableUploads: true, + }, + ServiceSettings: { + EnableOnboardingFlow: false, + EnableTutorial: false, + }, + }); + + // 3. Navigate to channels page + await channelsPage.goto(); + await channelsPage.toBeVisible(); + + // 4. Install and enable the org-memory plugin. + // In a real environment, this would use a plugin tarball URL. + // For testing purposes, we check if it's already installed or skip. + // The plugin must be pre-built and available at the configured URL. + let pluginInstalled = false; + try { + pluginInstalled = await pw.isPluginActive(adminClient, PLUGIN_ID); + } catch { + // Plugin not installed yet + } + + if (!pluginInstalled) { + // Try to enable if already uploaded but not active + try { + await adminClient.enablePlugin(PLUGIN_ID); + pluginInstalled = true; + } catch { + // Plugin not uploaded — skip with informative message. + // In CI, the plugin tarball would be built and installed + // as part of the test setup pipeline. + test.skip(true, 'org-memory plugin not pre-installed; skipping installation test'); + } + } + + // 5. Verify plugin is active via API + await expect + .poll(async () => { + return await pw.isPluginActive(adminClient, PLUGIN_ID); + }) + .toBe(true); + + // 6. Verify the app bar icon appears in the UI + // Dismiss any overlays that may have appeared + await page.keyboard.press('Escape'); + await page.waitForTimeout(500); + + const appBarIcon = page.locator(`#app-bar-icon-${PLUGIN_ID}`); + await expect(appBarIcon).toBeVisible({timeout: 10000}); + }); +}); diff --git a/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_rhs.spec.ts b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_rhs.spec.ts new file mode 100644 index 000000000000..19f933a49cb0 --- /dev/null +++ b/e2e-tests/playwright/specs/functional/plugins/org_memory/org_memory_rhs.spec.ts @@ -0,0 +1,223 @@ +// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved. +// See LICENSE.txt for license information. + +// E2E tests for the org-memory plugin RHS panel. +// Covers opening/closing, query submission, loading states, +// response rendering, citation navigation, follow-up queries, +// empty query handling, and new conversation reset. +// All tests use route interception — no real Gemini API key required. + +import {test, expect} from '@mattermost/playwright-lib'; + +import { + mockQueryResponse, + mockQueryResponseDeferred, + mockFollowupResponse, + defaultQueryResponse, +} from './helpers/mock-org-memory-api'; +import {setupOrgMemoryTest, openOrgMemoryRhs, submitQueryInRhs, PLUGIN_ID} from './helpers/test-setup'; + +test.describe('Org Memory RHS panel @org-memory', () => { + test('clicking the app bar icon opens the RHS panel', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + + await expect(rhsContainer).toContainText(/Ask your workspace|Organizational Memory/); + }); + + test('clicking the app bar icon again closes the RHS panel', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + const appBarIcon = page.locator(`#app-bar-icon-${PLUGIN_ID}`); + + // Open + await appBarIcon.click(); + await channelsPage.sidebarRight.toBeVisible(); + + // Close by clicking again + await appBarIcon.click(); + await expect(page.locator('#sidebar-right')).not.toBeVisible(); + }); + + test('close button closes the RHS panel', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + await openOrgMemoryRhs(page, channelsPage); + + await channelsPage.sidebarRight.close(); + await expect(page.locator('#sidebar-right')).not.toBeVisible(); + }); + + test('typing a query and pressing Enter shows loading then response', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + const deferred = await mockQueryResponseDeferred(page); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'What decisions were made about the API?'); + + // Verify loading indicator appears + await expect(rhsContainer.locator('.org-memory-spinner')).toBeVisible({timeout: 5000}); + await expect(rhsContainer).toContainText('Searching your workspace...'); + + // Resolve the deferred response + deferred.resolve(); + + // Verify loading goes away and response appears + await expect(rhsContainer.locator('.org-memory-spinner')).not.toBeVisible({timeout: 10000}); + await expect(rhsContainer).toContainText('API was redesigned', {timeout: 5000}); + }); + + test('response renders with markdown formatting', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + await mockQueryResponse(page); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'What about the API?'); + + await expect(rhsContainer).toContainText('API was redesigned', {timeout: 10000}); + await expect(rhsContainer).toContainText('Sources'); + }); + + test('citation links are present with correct permalink URLs', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + const citationPermalink = '/test-team/pl/mock-post-1'; + await mockQueryResponse(page, defaultQueryResponse({ + citations: [ + { + ref_number: 1, + post_id: 'mock-post-1', + permalink: citationPermalink, + channel_name: 'engineering', + author: 'alice', + snippet: 'We decided to redesign the API.', + }, + ], + })); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'API decisions'); + + await expect(rhsContainer).toContainText('Sources', {timeout: 10000}); + + const citationLink = rhsContainer.locator(`a[href="${citationPermalink}"]`); + await expect(citationLink).toBeVisible(); + await expect(citationLink).toContainText('[1]'); + }); + + test('citation link navigates to the original post permalink', async ({pw}) => { + const {adminClient, channelsPage, page, team} = await setupOrgMemoryTest(pw); + + // Create a real post that we can navigate to + const channel = await adminClient.getChannelByName(team.id, 'town-square'); + const realPost = await adminClient.createPost({ + channel_id: channel.id, + message: 'This is a test post for citation navigation', + }); + + const permalink = `/${team.name}/pl/${realPost.id}`; + + await mockQueryResponse(page, defaultQueryResponse({ + citations: [ + { + ref_number: 1, + post_id: realPost.id, + permalink, + channel_name: 'town-square', + author: 'sysadmin', + snippet: 'Test post for citation', + }, + ], + })); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'Show me the decision'); + + await expect(rhsContainer).toContainText('Sources', {timeout: 10000}); + + const citationLink = rhsContainer.locator(`a[href="${permalink}"]`); + await expect(citationLink).toBeVisible(); + await citationLink.click(); + + await expect(page).toHaveURL(new RegExp(realPost.id), {timeout: 10000}); + }); + + test('follow-up query uses follow-up endpoint and both exchanges are visible', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + await mockQueryResponse(page); + await mockFollowupResponse(page); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'What about the API redesign?'); + + // Wait for initial response + await expect(rhsContainer).toContainText('API was redesigned', {timeout: 10000}); + + // Submit a follow-up query + const followUpInput = rhsContainer.locator('textarea[placeholder="Ask a follow-up question..."]'); + await expect(followUpInput).toBeVisible({timeout: 5000}); + await followUpInput.fill('When was it completed?'); + await followUpInput.press('Enter'); + + // Wait for follow-up response + await expect(rhsContainer).toContainText('completed in October', {timeout: 10000}); + + // Verify both exchanges are visible in the conversation history + await expect(rhsContainer).toContainText('What about the API redesign?'); + await expect(rhsContainer).toContainText('API was redesigned'); + await expect(rhsContainer).toContainText('When was it completed?'); + await expect(rhsContainer).toContainText('completed in October'); + }); + + test('pressing Enter with empty input does not trigger API call', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + // Set up a mock that tracks whether it was called + let apiCalled = false; + await page.route('**/plugins/org-memory/api/v1/query', async (route) => { + apiCalled = true; + await route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(defaultQueryResponse()), + }); + }); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + + const queryInput = rhsContainer.locator('textarea').first(); + await queryInput.press('Enter'); + + // Wait a moment to ensure no API call is made + await page.waitForTimeout(1000); + + expect(apiCalled).toBe(false); + await expect(rhsContainer.locator('.org-memory-spinner')).not.toBeVisible(); + }); + + test('New conversation resets the panel to initial state', async ({pw}) => { + const {channelsPage, page} = await setupOrgMemoryTest(pw); + + await mockQueryResponse(page); + + const rhsContainer = await openOrgMemoryRhs(page, channelsPage); + await submitQueryInRhs(rhsContainer, 'What about the API?'); + + await expect(rhsContainer).toContainText('API was redesigned', {timeout: 10000}); + + // Click "New conversation" to reset + const newConvButton = rhsContainer.locator('button', {hasText: '+ New conversation'}); + await expect(newConvButton).toBeVisible(); + await newConvButton.click(); + + // Verify the panel resets to initial state + const initialInput = rhsContainer.locator('textarea[placeholder="Ask a question about your workspace..."]'); + await expect(initialInput).toBeVisible({timeout: 5000}); + await expect(rhsContainer).not.toContainText('API was redesigned'); + await expect(rhsContainer).toContainText('Ask your workspace'); + }); +}); diff --git a/mp-sw-factory.yaml b/mp-sw-factory.yaml new file mode 100644 index 000000000000..5aa990b05737 --- /dev/null +++ b/mp-sw-factory.yaml @@ -0,0 +1,146 @@ +# mp-sw-factory.yaml — Mattermost monorepo + org-memory plugin +# This is a large Go + React/TypeScript monorepo (Mattermost server) with an +# in-tree plugin at server/plugins/org-memory/ that is the primary development focus. + +# ─── AI Instruction Fields ────────────────────────────────────────────────────── + +testing: | + This repo has multiple test surfaces. The primary development focus is the + **org-memory plugin** at `server/plugins/org-memory/`. + + ## Plugin Go tests (primary — run these first) + ```bash + cd server/plugins/org-memory/server && GOWORK=off go test -v -count=1 ./... + ``` + - CRITICAL: `GOWORK=off` is required because the parent `server/go.work` workspace + interferes with the plugin's own `go.mod`. + - 21 test files covering: plugin lifecycle, configuration, hooks, worker, debounce, + circuit breaker, rate limiter, chunker, RAG engine, API, backfill, metrics, + channel summary worker, data retention, Gemini client, Gemini errors. + - Tests use `plugintest.API` mocks, `plugintest.NewDriver(t)` for DB driver mocks, + and `pluginapi.NewClient(api, driver)` for the client. + - Look for "PASS" or "FAIL" in output. Exit code 0 = all passed. + + ## Plugin webapp tests + ```bash + cd server/plugins/org-memory/webapp && npm test + ``` + - Jest 29 with jsdom environment, ts-jest for TypeScript. + - Coverage thresholds: branches 70%, functions 70%, lines 75%, statements 75%. + - Test files in `src/__tests__/`. + + ## Mattermost server tests (only if modifying server/ code outside the plugin) + ```bash + cd server && make test-server + ``` + - Requires Docker services (PostgreSQL, etc.) via `make start-docker`. + - Very slow — avoid unless modifying core server code. + + ## Mattermost webapp tests (only if modifying webapp/ code) + ```bash + cd webapp && npm run test + ``` + - Jest across multiple workspaces (channels, platform/*). + + ## E2E tests + - **Cypress** (v15.11): `cd e2e-tests/cypress && npm run test:ci` + - **Playwright** (v1.58): `cd e2e-tests/playwright && npm run test:ci` + - Both require a running Mattermost server. Use `cd e2e-tests && make` for full setup. + - These are slow and should only be run selectively for relevant changes. + +linting: | + ## Plugin Go linting (primary) + There is no golangci-lint config specific to the plugin. Use `go vet`: + ```bash + cd server/plugins/org-memory/server && GOWORK=off go vet ./... + ``` + + ## Plugin webapp type checking + ```bash + cd server/plugins/org-memory/webapp && npm run lint + ``` + This runs `tsc --noEmit` — a type checker, not a traditional linter. It reports + TypeScript type errors. No autofix available for type errors. + + ## Mattermost server linting (only if modifying server/ code outside the plugin) + ```bash + cd server && make check-style + ``` + - Uses golangci-lint v2.6.0 with config at `server/.golangci.yml`. + - Linters: bidichk, errcheck, govet, ineffassign, misspell, modernize, revive, + staticcheck, unconvert, unused, whitespace. + + ## Mattermost webapp linting (only if modifying webapp/ code) + ```bash + cd webapp && npm run check + ``` + - ESLint 8.57 across all workspaces. + - Autofix: `cd webapp && npm run fix` + + ## Mattermost webapp type checking + ```bash + cd webapp && npm run check-types + ``` + +install: | + ## Plugin dependencies (primary) + ```bash + # Go dependencies (plugin server) + cd server/plugins/org-memory/server && GOWORK=off go mod download + + # Webapp dependencies (plugin frontend) + cd server/plugins/org-memory/webapp && npm install + ``` + + ## Full Mattermost dependencies (if needed) + ```bash + # Server Go workspace setup + cd server && make setup-go-work && go mod download + + # Webapp npm dependencies + cd webapp && npm install + + # Docker services for local dev (PostgreSQL, Redis, Minio, etc.) + cd server && make start-docker + ``` + + ## Requirements + - Go 1.24+ (check `server/.go-version` for exact version) + - Node.js 24+ / npm 11+ + - Docker & docker-compose (for server integration tests) + - PostgreSQL with pgvector extension (for plugin) + +logs: | + ## Plugin logs + - Plugin logs go through Mattermost's plugin logging API (`p.API.LogInfo`, + `p.API.LogError`, `p.API.LogWarn`). + - In development: logs appear in the Mattermost server stdout/stderr. + - Test output: `go test -v` shows log output inline. + + ## Server logs + - Dev server: `cd server && make run-server` — logs to stdout. + - Docker services: `docker-compose logs -f` from `server/`. + + ## Webapp logs + - Browser console for frontend debugging. + - Webpack dev server output: `cd webapp && make dev` + +coderabbit: auto + +# ─── Pipeline Step Configuration ──────────────────────────────────────────────── + +steps: + run_lint: + # Lint plugin Go code + typecheck plugin webapp + check: "(cd server/plugins/org-memory/server && GOWORK=off go vet ./...) && (cd server/plugins/org-memory/webapp && npx tsc --noEmit)" + context: "(cd server/plugins/org-memory/server && GOWORK=off go vet ./... 2>&1) | tail -30; (cd server/plugins/org-memory/webapp && npx tsc --noEmit 2>&1) | tail -30" + + run_tests: + # Run plugin Go tests + plugin webapp tests + check: "(cd server/plugins/org-memory/server && GOWORK=off go test -count=1 ./...) && (cd server/plugins/org-memory/webapp && npm test -- --watchAll=false)" + context: "(cd server/plugins/org-memory/server && GOWORK=off go test -v -count=1 ./... 2>&1) | tail -80; (cd server/plugins/org-memory/webapp && npm test -- --watchAll=false 2>&1) | tail -40" + + check_build: + # Build plugin server binary (native platform only) + webapp bundle + check: "(cd server/plugins/org-memory/server && GOWORK=off go build -o /dev/null .) && (cd server/plugins/org-memory/webapp && npx webpack --mode=production)" + context: "(cd server/plugins/org-memory/server && GOWORK=off go build -o /dev/null . 2>&1) | tail -30; (cd server/plugins/org-memory/webapp && npx webpack --mode=production 2>&1) | tail -30" diff --git a/tasks/0001-org-memory-ai-knowledge-base/001.md b/tasks/0001-org-memory-ai-knowledge-base/001.md new file mode 100644 index 000000000000..929633441911 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/001.md @@ -0,0 +1,97 @@ +# 001: Plugin scaffold, manifest, and build system + +## Context + +This is the foundational step that creates the plugin's directory structure, Go module, plugin manifest, build system, and bare entry point. Everything in subsequent steps depends on this scaffold existing. The existing `server/plugins/org-memory/` directory contains stale compiled artifacts from a deleted codebase with OpenAI-oriented settings — it must be replaced entirely with a fresh scaffold configured for Gemini. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/plugins/org-memory/plugin.json` — The existing manifest to be replaced. Provides the plugin ID (`org-memory`), name, description, version, and min_server_version. The settings schema must be rewritten for Gemini (currently references OpenAI base URLs, gpt-4o-mini, text-embedding-3-small, 1536 dimensions). +- `server/plugins/github/plugin.json` — Reference manifest showing best practices: multi-platform executable paths for 5 architectures (darwin-amd64, darwin-arm64, linux-amd64, linux-arm64, windows-amd64), webapp bundle path, icon_path for assets, rich settings_schema with text, bool, number, dropdown, and secret field types, header/footer help text. +- `server/public/plugin/client.go` — Defines `MattermostPlugin` base struct. The plugin's main struct must embed this. Entry point calls `plugin.ClientMain(&Plugin{})`. +- `server/public/pluginapi/client.go` — The `pluginapi.Client` that wraps the raw plugin API. The plugin should import this package for later use. +- `server/public/model/utils.go` — Provides `model.NewId()` for ID generation. The plugin's go.mod must include the Mattermost public model package. + +## Reuse Opportunities + +- The existing `server/plugins/org-memory/plugin.json` structure (ID, name, description, executable paths, webapp bundle path) can be reused as a starting template, but the settings_schema must be completely replaced with Gemini-oriented settings. +- The GitHub plugin manifest demonstrates the full range of settings_schema field types (text, bool, number, generated, dropdown, secret) — follow its patterns for the Gemini settings. +- The Mattermost plugin SDK conventions for Go module naming, import paths, and build targets are standardized — the scaffold should follow the same patterns used by community plugins (separate go.mod for the plugin, importing from `github.com/mattermost/mattermost/server/public/...`). + +## Deliverables + +### Plugin manifest (plugin.json) + +Replace the existing `server/plugins/org-memory/plugin.json` with a new manifest that: + +- Retains the plugin ID `org-memory`, updates the name to "Organizational Memory", and keeps min_server_version at `9.5.0` +- Defines server executables for all 5 platform targets: darwin-amd64, darwin-arm64, linux-amd64, linux-arm64, windows-amd64 +- Defines the webapp bundle path at `webapp/dist/main.js` +- Includes an icon_path for the plugin icon asset +- Replaces the entire settings_schema with Gemini-oriented settings: + - **GeminiAPIKey** — text, secret, no default. Help text explaining this is the Google AI API key for Gemini. + - **GeminiModel** — text, default `gemini-2.5-flash`. The model used for summarization, extraction, and RAG synthesis. + - **EmbeddingModel** — text, default `gemini-embedding-001`. The model used for generating vector embeddings. + - **EmbeddingDimensions** — number, default `768`. The dimensionality of generated embeddings. + - **WorkerPollMs** — number, default `1000`. How frequently the background worker polls for pending processing items (milliseconds). + - **ThreadDebounceSeconds** — number, default `120`. How long a thread must be inactive before triggering summarization. + - **MaxChunksPerPost** — number, default `32`. Maximum number of chunks a single post can produce for embedding. + - **SkipBotPosts** — bool, default `true`. Whether to skip processing messages from bot accounts. + - **EnableForTeams** — text, default empty string. Comma-separated list of team IDs to enable processing for; empty means all teams. + - **DailySummaryHour** — number, default `8`. The hour of day (0-23) at which daily channel summaries are generated. +- Includes a descriptive header explaining the plugin's purpose and a note that pgvector must be enabled + +### Plugin server source directory + +Create a new source directory structure under `server/plugins/org-memory/server/` with: + +- A Go module file (go.mod) declaring the module path and Go version, with dependencies on the Mattermost server public packages (plugin, pluginapi, model) and the Google AI Go SDK (`google.golang.org/genai`) +- A main plugin file containing the Plugin struct that embeds `plugin.MattermostPlugin` and calls `plugin.ClientMain` as the entry point +- The plugin struct should be a bare skeleton at this stage — no hook implementations, no business logic. Just the minimum needed to compile and be loaded by Mattermost. + +### Plugin webapp source directory + +Create a new source directory structure under `server/plugins/org-memory/webapp/` with: + +- A package.json declaring the webapp as a Node.js project with dependencies on React, TypeScript, and the Mattermost webapp types +- A webpack or equivalent bundler configuration that produces a single `dist/main.js` bundle +- A minimal entry point that registers the plugin with Mattermost via `window.registerPlugin('org-memory', ...)` with empty `initialize` and `uninitialize` functions +- A TypeScript configuration file + +### Build system + +Create a Makefile (or equivalent build script) at the plugin root (`server/plugins/org-memory/`) that supports: + +- Building the Go server binary for all 5 platform targets (cross-compilation via GOOS/GOARCH) +- Building the webapp bundle via the Node.js toolchain +- A `dist` or `bundle` target that assembles the final plugin package: manifest + server binaries + webapp bundle + assets into the correct directory structure +- A `clean` target to remove build artifacts + +### Plugin icon asset + +Create a simple SVG icon in `server/plugins/org-memory/assets/` for the plugin's branding in the System Console and app bar. + +### Remove stale artifacts + +Delete the existing compiled artifacts: +- `server/plugins/org-memory/server/dist/` (old server binaries) +- `server/plugins/org-memory/webapp/dist/` (old webapp bundle) + +These will be regenerated by the new build system. + +## Acceptance Criteria + +- [x] The plugin manifest at `server/plugins/org-memory/plugin.json` is valid JSON with plugin ID `org-memory`, min_server_version `9.5.0`, and all 10 Gemini-oriented settings in the settings_schema (GeminiAPIKey, GeminiModel, EmbeddingModel, EmbeddingDimensions, WorkerPollMs, ThreadDebounceSeconds, MaxChunksPerPost, SkipBotPosts, EnableForTeams, DailySummaryHour) +- [x] The manifest references server executables for all 5 platforms (darwin-amd64, darwin-arm64, linux-amd64, linux-arm64, windows-amd64) and the webapp bundle at `webapp/dist/main.js` +- [x] No references to OpenAI, GPT, or text-embedding-3 remain anywhere in the plugin directory +- [x] The Go module compiles successfully with `go build` and produces a server binary that Mattermost can load as a plugin process +- [x] The Go source imports the Mattermost plugin SDK and the Google AI Go SDK as dependencies +- [x] The Plugin struct embeds `plugin.MattermostPlugin` and the main function calls `plugin.ClientMain` +- [x] The webapp builds successfully with `npm run build` (or equivalent) and produces a `webapp/dist/main.js` bundle +- [x] The webapp entry point registers the plugin with Mattermost via `window.registerPlugin('org-memory', ...)` and Mattermost recognizes it on load +- [x] The Makefile can cross-compile the server binary for all 5 platform targets +- [x] The Makefile can build the webapp bundle +- [x] Running the full build produces a complete plugin directory structure matching what plugin.json declares (server binaries at the declared paths, webapp bundle at the declared path, icon asset at the declared path) +- [x] The old compiled artifacts (server/dist/, webapp/dist/) from the previous implementation have been removed and replaced by the new build output diff --git a/tasks/0001-org-memory-ai-knowledge-base/002.md b/tasks/0001-org-memory-ai-knowledge-base/002.md new file mode 100644 index 000000000000..95a2c70e6be1 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/002.md @@ -0,0 +1,107 @@ +# 002: Configuration management and plugin lifecycle + +## Context + +With the plugin scaffold from step 001 in place, this step wires up the full plugin lifecycle: loading configuration from the manifest's settings_schema into a typed struct, initializing the pluginapi.Client for organized API access, validating required settings (especially the Gemini API key), registering the `/ask` slash command, and implementing graceful shutdown. This is the last "infrastructure" step before database and business logic work begins — all subsequent steps depend on a functioning plugin that activates cleanly, reloads configuration on changes, and shuts down without leaking resources. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/hooks.go` (lines 82-107) — Defines `OnActivate`, `OnDeactivate`, and `OnConfigurationChange` hooks. Key lifecycle detail: `OnConfigurationChange` is called once *before* `OnActivate`, so the config struct is populated before activation begins. `OnActivate` errors terminate the plugin; `OnConfigurationChange` errors are logged but non-fatal. +- `server/public/pluginapi/client.go` (lines 36-69) — `pluginapi.NewClient(api, driver)` creates the helper client with organized services (Bot, Channel, Cluster, Configuration, KV, Log, Post, Store, User, etc.). Must be created exactly once per plugin instance during `OnActivate`. +- `server/public/pluginapi/configuration.go` — `ConfigurationService.LoadPluginConfiguration(dest any)` unmarshals the plugin's JSON config into a Go struct. Field names match settings_schema keys (case-insensitive matching). +- `server/channels/app/plugin_api_tests/manual.test_load_configuration_plugin/main.go` — Real test plugin showing the configuration struct pattern: exported fields mapping to plugin.json settings keys, `OnConfigurationChange` calling `LoadPluginConfiguration`, and embedding a `BasicConfig` struct for composition. +- `server/channels/app/plugin_commands_test.go` (lines 64-77) — Shows `RegisterCommand` in `OnActivate` with `model.Command` fields: TeamId, Trigger, DisplayName, AutoComplete, AutoCompleteDesc. Also shows re-registration during `OnConfigurationChange`. +- `server/public/pluginapi/example_client_test.go` — Canonical pattern: Plugin struct with `client *pluginapi.Client` field, initialized in `OnActivate` via `pluginapi.NewClient(p.API, p.Driver)`. +- `server/public/pluginapi/store.go` — `StoreService` with `GetMasterDB()` returning `*sql.DB`. Called during activation to obtain the database handle for later use by the knowledge store. +- `server/public/plugin/environment.go` (lines 283-305) — Shows execution order: plugin state set to Running, then `OnActivate()` called. If OnActivate returns an error, the supervisor is shut down and the plugin fails to start. + +## Reuse Opportunities + +- The `pluginapi.Client` provides organized access to all Mattermost services. Use `client.Configuration.LoadPluginConfiguration` for config loading, `client.SlashCommand.Register` for command registration, `client.Store.GetMasterDB` for database access, `client.Log` for structured logging, and `client.KV` for key-value operations. This avoids calling the raw `p.API.*` methods directly throughout the codebase. +- The configuration struct pattern is well-established: exported Go struct fields with names matching the plugin.json settings_schema keys. The implementer should follow this exact pattern for the 10 Gemini-oriented settings defined in step 001. +- `model.Command` struct fields for slash command registration are standardized. The `/ask` command should follow the same registration pattern seen in the test plugins. + +## Deliverables + +### Configuration struct + +Define a Go struct with exported fields mapping to each of the 10 settings from the plugin manifest: + +- GeminiAPIKey (string, secret) +- GeminiModel (string) +- EmbeddingModel (string) +- EmbeddingDimensions (integer) +- WorkerPollMs (integer) +- ThreadDebounceSeconds (integer) +- MaxChunksPerPost (integer) +- SkipBotPosts (boolean) +- EnableForTeams (string — comma-separated team IDs) +- DailySummaryHour (integer) + +The struct should also provide a validation method that checks for required settings (at minimum, GeminiAPIKey must be non-empty) and returns a descriptive error if validation fails. It should provide a convenience method that parses EnableForTeams into a set/map of team IDs for efficient lookup during event ingestion. + +### OnConfigurationChange implementation + +Implement the `OnConfigurationChange` hook to: + +1. Load the plugin configuration into the configuration struct using the plugin API +2. Validate the loaded configuration +3. If validation fails, log an error with a descriptive message and return the error +4. Store the validated configuration on the Plugin struct for access by other components + +Since `OnConfigurationChange` is called before `OnActivate` on startup (and also on live config changes), it must work correctly whether or not the plugin is fully initialized yet. It should not attempt to reconfigure components that haven't been created. + +### OnActivate implementation + +Implement the `OnActivate` hook to perform the following initialization sequence: + +1. Create the `pluginapi.Client` and store it on the Plugin struct +2. Verify that the configuration has been loaded (it should have been populated by the prior `OnConfigurationChange` call) +3. Validate that the GeminiAPIKey is present — if not, return an error that terminates activation with a clear message +4. Obtain a database handle via `client.Store.GetMasterDB()` and store it on the Plugin struct for use by later steps (schema migration, store operations) +5. Register the `/ask` slash command with appropriate display name, description, auto-complete hint, and auto-complete description. The command should be workspace-wide (empty TeamId means all teams). +6. Log a success message indicating the plugin has activated + +If any step fails, return an error to prevent the plugin from starting in a broken state. Resources acquired before the failure point should be cleaned up. + +Note: Background scheduler initialization, schema migration, and Gemini client creation are **not** part of this step — they will be added in later steps as those components are implemented. The Plugin struct should have placeholder fields for these components (e.g., fields for the scheduler, store, and Gemini client) that will be populated in subsequent steps. + +### OnDeactivate implementation + +Implement the `OnDeactivate` hook to: + +1. Shut down any background schedulers that have been started (placeholder for now — later steps will register schedulers) +2. Close the database connection via the store service +3. Log a message indicating clean shutdown + +The deactivation must be safe to call even if activation never fully completed (partial initialization). Each resource cleanup should check whether the resource was actually initialized before attempting to close it. + +### Plugin struct fields + +Extend the Plugin struct (created in step 001) with fields for: + +- The `pluginapi.Client` instance +- The configuration struct +- The database handle (`*sql.DB`) +- Placeholder fields for components added in later steps: a Gemini client interface, a store interface, scheduler references, and a metrics collector. These fields can be nil/zero-valued at this stage but should be defined so that later steps can populate them without modifying the struct definition. + +## Acceptance Criteria + +- [x] The configuration struct has exported fields for all 10 plugin settings (GeminiAPIKey, GeminiModel, EmbeddingModel, EmbeddingDimensions, WorkerPollMs, ThreadDebounceSeconds, MaxChunksPerPost, SkipBotPosts, EnableForTeams, DailySummaryHour) +- [x] Configuration validation rejects an empty GeminiAPIKey with a descriptive error message +- [x] Configuration validation accepts a valid configuration with all required fields populated +- [x] The EnableForTeams string is parseable into a set of team IDs, and an empty string is interpreted as "all teams enabled" +- [x] OnConfigurationChange successfully loads the plugin configuration into the struct and stores it on the Plugin +- [x] OnConfigurationChange works correctly when called before OnActivate (during the initial startup sequence) +- [x] OnConfigurationChange works correctly when called after OnActivate (during a live config update) and propagates updated values +- [x] OnActivate creates the pluginapi.Client and stores it on the Plugin struct +- [x] OnActivate obtains a database handle via the store service and stores it on the Plugin struct +- [x] OnActivate fails with a descriptive error if the GeminiAPIKey is not configured +- [x] OnActivate registers the `/ask` slash command with trigger "ask", a display name, auto-complete enabled, and a help description +- [x] OnActivate logs a success message on successful initialization +- [x] OnDeactivate completes without errors when all resources were initialized +- [x] OnDeactivate completes without errors when called after a partial initialization (some resources are nil) +- [x] OnDeactivate closes the database connection +- [x] The Plugin struct has placeholder fields defined for the Gemini client, store interface, schedulers, and metrics collector so that subsequent steps can populate them without struct changes diff --git a/tasks/0001-org-memory-ai-knowledge-base/003.md b/tasks/0001-org-memory-ai-knowledge-base/003.md new file mode 100644 index 000000000000..aba09c9b345b --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/003.md @@ -0,0 +1,240 @@ +# 003: Database schema and knowledge store + +## Context + +With the plugin lifecycle operational from step 002, this step creates the seven database tables that form the knowledge base's persistence layer, plus the store interface that all other components use for data access. The plugin must create the pgvector extension and all tables idempotently during OnActivate (there is no plugin migration framework — plugins run raw SQL against the database). This step also implements the full store interface with CRUD operations, vector similarity search, and permission-filtered queries that subsequent steps (workers, RAG engine, API) depend on. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/channels/db/migrations/postgres/000001_create_teams.up.sql` — Core migration showing Mattermost SQL conventions: `VARCHAR(26)` for IDs, `bigint` for timestamps, `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. Soft delete via `deleteat bigint` column (0 = active, >0 = deletion timestamp). +- `server/channels/db/migrations/postgres/000053_create_retention_policies.up.sql` — Shows how foreign keys are added idempotently using `DO $$ BEGIN IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '...') THEN ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY ... ON DELETE CASCADE; END IF; END; $$;` blocks. +- `server/channels/db/migrations/postgres/000149_create_recaps.up.sql` — Shows inline foreign key declaration with `FOREIGN KEY (RecapId) REFERENCES Recaps(Id) ON DELETE CASCADE` inside CREATE TABLE. +- `server/channels/store/sqlstore/bot_store.go` — Reference store implementation: constructor pre-builds a Squirrel SelectBuilder, GET methods use `GetReplica().Get()` with `sql.ErrNoRows` → `store.NewErrNotFound()` error mapping, writes use `GetMaster().Exec()`, Squirrel queries use `sq.Dollar` placeholder format for PostgreSQL. +- `server/channels/store/sqlstore/plugin_store.go` — Plugin key-value store showing Squirrel with `INSERT ... ON CONFLICT DO UPDATE` (upsert) pattern, `ToSql()` → `Exec(queryString, args...)` flow, and `errors.Wrap` for error context. +- `server/channels/store/store.go` — Core store interface pattern: a facade interface with getter methods returning sub-store interfaces (`Team() TeamStore`, `Post() PostStore`, etc.). Each sub-store defines its own CRUD contract. +- `server/public/pluginapi/store.go` — `StoreService.GetMasterDB()` returns a singleton `*sql.DB`. `GetReplicaDB()` falls back to master if no replica configured. `DriverName()` returns the driver name. `Close()` cleans up connections. The store test in `store_test.go` confirms singleton behavior. +- `server/public/model/utils.go` — `model.NewId()` generates 26-character base32-encoded UUID v4 strings for row IDs. +- `server/channels/store/sqlstore/store.go` (lines 915-926) — Query builder setup: `sq.StatementBuilder.PlaceholderFormat(sq.Dollar)` for PostgreSQL dollar-sign placeholders. + +## Reuse Opportunities + +- **Squirrel query builder** (`github.com/mattermost/squirrel`) is already a Mattermost dependency. The plugin store should use it with `sq.Dollar` placeholder format for all parameterized queries, following the same `Select().From().Where().ToSql()` → `db.Query(sql, args...)` pattern used in core stores. +- **model.NewId()** from `github.com/mattermost/mattermost/server/public/model` generates IDs consistent with all Mattermost tables. Use for all new row IDs. +- **pluginapi.StoreService** provides `GetMasterDB()` and `GetReplicaDB()` — the plugin should use master for writes and replica for reads (read queries in the RAG engine and extraction listing). +- **Error wrapping** via `github.com/pkg/errors` with `errors.Wrap(err, "context")` and `errors.Wrapf(err, "context %s", id)` is the standard pattern across all Mattermost stores. +- **Store interface pattern** from core (facade with sub-store getters) can be applied in miniature: a plugin-level store interface with methods grouped by table, making it easy to mock for unit tests. + +## Deliverables + +### Schema migration runner + +Add schema migration logic to the OnActivate lifecycle (from step 002). During plugin activation, after obtaining the database handle, the plugin must execute idempotent SQL statements to: + +1. Create the pgvector extension (`CREATE EXTENSION IF NOT EXISTS vector`). If this fails (insufficient privileges), activation must fail with an error message instructing the admin to install the extension manually. +2. Create all 7 tables with `CREATE TABLE IF NOT EXISTS`. +3. Create all indexes with `CREATE INDEX IF NOT EXISTS`. +4. Add foreign key constraints idempotently (check pg_constraint before adding). + +All migration SQL must be safe to run concurrently from multiple plugin instances in a HA cluster — the IF NOT EXISTS and constraint-existence checks ensure this. + +### Table: ai_embeddings + +Vector storage for post/file/summary content. Each row represents one chunk of embedded content. + +Columns: +- **Id** — VARCHAR(26), primary key, generated via model.NewId() +- **PostId** — VARCHAR(26), not null, foreign key to Posts(Id) with ON DELETE CASCADE +- **ChannelId** — VARCHAR(26), not null, foreign key to Channels(Id) with ON DELETE CASCADE +- **SourceType** — VARCHAR(20), not null, one of: 'post', 'file', 'summary' +- **ChunkIndex** — integer, not null, default 0, sequential index within the parent post +- **ContentSnapshot** — text, not null, the original text content of this chunk (for citation display) +- **Embedding** — vector(768), not null, L2-normalized embedding from Gemini +- **CreateAt** — bigint, not null, Unix milliseconds +- **DeleteAt** — bigint, default 0, soft delete timestamp + +Indexes: +- HNSW index on Embedding column using `vector_cosine_ops` operator class for approximate nearest-neighbor cosine similarity search +- B-tree index on ChannelId for permission-filtered queries +- B-tree composite index on (PostId, ChunkIndex) for chunk ordering and deduplication lookups + +### Table: ai_extractions + +Extracted knowledge items (decisions, action items, topics). + +Columns: +- **Id** — VARCHAR(26), primary key +- **PostId** — VARCHAR(26), not null, foreign key to Posts(Id) with ON DELETE CASCADE +- **ChannelId** — VARCHAR(26), not null, foreign key to Channels(Id) with ON DELETE CASCADE +- **ExtractionType** — VARCHAR(32), not null, one of: 'decision', 'action_item', 'topic' +- **Content** — text, not null +- **Metadata** — JSONB, nullable, structured metadata (assignee, status, confidence) +- **IsEdited** — boolean, default false +- **EditorUserId** — VARCHAR(26), nullable +- **CreateAt** — bigint, not null +- **UpdateAt** — bigint, not null +- **DeleteAt** — bigint, default 0 + +Indexes: +- Composite index on (ChannelId, ExtractionType) with a WHERE DeleteAt = 0 partial index condition for filtered listing queries + +### Table: ai_thread_summaries + +One summary per thread, keyed by root post ID. + +Columns: +- **RootId** — VARCHAR(26), primary key, foreign key to Posts(Id) with ON DELETE CASCADE +- **ChannelId** — VARCHAR(26), not null, foreign key to Channels(Id) with ON DELETE CASCADE +- **Summary** — text, not null +- **KeyTakeaways** — JSONB, nullable, array of strings +- **ModelVersion** — VARCHAR(64), not null +- **LastActivityAt** — bigint, not null +- **UpdateAt** — bigint, not null + +### Table: ai_channel_summaries + +Rolling daily and weekly channel digests. + +Columns: +- **Id** — VARCHAR(26), primary key +- **ChannelId** — VARCHAR(26), not null, foreign key to Channels(Id) with ON DELETE CASCADE +- **SummaryType** — VARCHAR(16), not null, one of: 'daily', 'weekly' +- **Summary** — text, not null +- **WindowStart** — bigint, not null +- **WindowEnd** — bigint, not null +- **CreateAt** — bigint, not null + +Constraints: +- UNIQUE constraint on (ChannelId, SummaryType, WindowStart) to prevent duplicate summaries for the same window + +### Table: ai_entity_mentions + +Cross-channel entity linking for timeline queries. + +Columns: +- **Id** — VARCHAR(26), primary key +- **EntityName** — VARCHAR(255), not null, normalized to lowercase +- **EntityType** — VARCHAR(32), nullable, one of: 'project', 'technology', 'person' +- **PostId** — VARCHAR(26), not null, foreign key to Posts(Id) with ON DELETE CASCADE +- **ChannelId** — VARCHAR(26), not null, foreign key to Channels(Id) with ON DELETE CASCADE +- **ExtractionId** — VARCHAR(26), nullable, foreign key to ai_extractions(Id), nullable because the link is optional +- **CreateAt** — bigint, not null + +Indexes: +- Composite index on (EntityName, CreateAt) for timeline queries ordered by time + +### Table: ai_processing_log + +Work queue and idempotency tracker for async processing. + +Columns: +- **Id** — VARCHAR(26), primary key +- **PostId** — VARCHAR(26), not null +- **JobType** — VARCHAR(32), not null, one of: 'embed', 'extract', 'summarize' +- **SourceType** — VARCHAR(20), not null, default 'post', one of: 'post', 'file', 'summary'. Distinguishes whether the content to process comes from a post message, an attached file, or a generated summary. +- **Status** — VARCHAR(16), not null, one of: 'pending', 'processing', 'complete', 'error' +- **ScheduleAt** — bigint, nullable, for debounce timing +- **ErrorMessage** — text, nullable +- **RetryCount** — integer, default 0 +- **CreateAt** — bigint, not null +- **CompleteAt** — bigint, nullable + +Constraints: +- UNIQUE constraint on (PostId, JobType, SourceType) to prevent duplicate work items for the same post, job type, and content source. This three-column constraint allows a single post to have separate embed entries for the post message, each attached file, and a generated summary without conflict. + +Indexes: +- Composite index on (Status, ScheduleAt) for worker polling queries that fetch pending items ordered by schedule time + +### Table: ai_query_log + +Audit trail for user queries. Write-only (no updates or deletes). + +Columns: +- **Id** — VARCHAR(26), primary key +- **UserId** — VARCHAR(26), not null, foreign key to Users(Id) with ON DELETE CASCADE +- **QueryText** — text, not null +- **ChannelIdsSearched** — JSONB, nullable, array of channel ID strings +- **ResultCount** — integer, not null +- **LatencyMs** — integer, not null +- **CreateAt** — bigint, not null + +### Store interface + +Define a store interface (or set of interfaces) that abstracts all database operations. The interface should be designed for easy mocking in unit tests. Group methods logically: + +**Embedding operations:** +- Save one or more embeddings for a post (batch insert for multiple chunks) +- Delete embeddings by post ID (for reprocessing or cascade cleanup) +- Soft-delete embeddings by post ID (set DeleteAt) +- Search embeddings by vector similarity, filtered by a list of permitted channel IDs, excluding soft-deleted entries, returning top N results ordered by cosine similarity + +**Extraction operations:** +- Save one or more extractions for a post +- Get extraction by ID +- List extractions filtered by channel ID and/or extraction type, excluding soft-deleted entries, with pagination +- Update an extraction's content (for manual editing — sets IsEdited, EditorUserId, UpdateAt) +- Soft-delete extractions by post ID +- Delete extractions by post ID (hard delete for cascade) + +**Thread summary operations:** +- Upsert a thread summary (insert or update by RootId) +- Get thread summary by root ID +- List thread summaries by channel ID within a time window (for channel summary generation) + +**Channel summary operations:** +- Save a channel summary +- Get the most recent channel summary by channel ID and type (daily/weekly) +- List channel summaries by channel ID with optional type filter + +**Entity mention operations:** +- Save one or more entity mentions for a post (batch insert) +- Get entity timeline by name — list mentions across channels, filtered by permitted channel IDs, ordered by creation time, with pagination +- Delete entity mentions by post ID + +**Processing log operations:** +- Create a processing log entry (with idempotency — ignore if duplicate PostId+JobType+SourceType exists). Used for initial ingestion where skipping duplicates is correct. +- Upsert a processing log entry — insert if no matching (PostId, JobType, SourceType) row exists, or reset an existing row's status to "pending" (clearing error message and resetting retry count) if one does exist. Used for post edits where reprocessing an already-completed entry is required. +- Update status (pending → processing, processing → complete, processing → error with message) +- List pending items ordered by schedule time (for worker polling) +- Find stale items stuck in "processing" status for longer than a threshold duration +- Reset stale items back to "pending" +- Get processing stats (count by status) for the admin status endpoint + +**Query log operations:** +- Insert a query log entry (write-only) +- Get query stats for the admin status endpoint (count, average latency) + +**Data retention operations:** +- Delete embeddings, extractions, entity mentions, and summaries whose source posts were created before a given timestamp, in batches up to a given size, returning the count of deleted rows + +### OnActivate integration + +Wire the schema migration and store initialization into the OnActivate lifecycle: +1. After obtaining the database handle (from step 002), run the schema migration +2. Create the store instance with the database handle +3. Store the store instance on the Plugin struct (in the placeholder field from step 002) + +## Acceptance Criteria + +- [x] Plugin activation on a fresh database with pgvector available creates all 7 tables, the pgvector extension, all indexes, and all foreign key constraints without errors +- [x] Plugin activation on a database where the tables already exist succeeds without errors (all CREATE statements are idempotent) +- [x] Plugin activation fails with a descriptive error if the pgvector extension cannot be created (insufficient privileges or extension not installed) +- [x] All table IDs use VARCHAR(26) and are generated via model.NewId() +- [x] All timestamps are stored as bigint (Unix milliseconds) +- [x] ai_embeddings and ai_extractions support soft delete via a DeleteAt column (0 = active, non-zero = deleted) +- [x] ai_embeddings has an HNSW index on the Embedding column with vector_cosine_ops +- [x] ai_processing_log enforces uniqueness on (PostId, JobType, SourceType) to prevent duplicate work items while allowing separate entries for the same post's message, attached files, and generated summaries +- [x] ai_channel_summaries enforces uniqueness on (ChannelId, SummaryType, WindowStart) to prevent duplicate summaries +- [x] Foreign keys from plugin tables to Posts, Channels, and Users use ON DELETE CASCADE +- [x] Inserting a 768-dimensional vector into ai_embeddings and retrieving it via cosine similarity search returns the expected row +- [x] Vector similarity search correctly filters by a list of channel IDs and excludes soft-deleted rows +- [x] The processing log create operation is idempotent — inserting a duplicate (PostId, JobType, SourceType) does not produce an error or a second row +- [x] The processing log upsert operation resets an existing entry's status to "pending" (clearing error, resetting retry count) when the entry already exists — used for reprocessing after post edits +- [x] The processing log polling query returns pending items ordered by ScheduleAt +- [x] Extraction listing filters correctly by channel ID and extraction type, excluding soft-deleted rows +- [x] Thread summary upsert creates a new row on first call and updates the existing row on subsequent calls for the same RootId +- [x] Entity timeline query returns mentions ordered by creation time, filtered by permitted channel IDs +- [x] The data retention operation deletes rows in batches and returns the count of deleted rows +- [x] The store interface is defined as a Go interface (or set of interfaces) that can be mocked for unit testing +- [x] The store instance is created during OnActivate and stored on the Plugin struct diff --git a/tasks/0001-org-memory-ai-knowledge-base/004.md b/tasks/0001-org-memory-ai-knowledge-base/004.md new file mode 100644 index 000000000000..30436bbddc79 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/004.md @@ -0,0 +1,123 @@ +# 004: Gemini client with resilience patterns + +## Context + +The Gemini client is the bridge between the plugin's processing logic and Google's AI services. Every core feature — embedding generation, knowledge extraction, thread summarization, and RAG synthesis — depends on this component. This step builds the client with four distinct operations behind a mockable interface, plus production-grade resilience (exponential backoff on rate limits, circuit breaker for sustained failures) with cluster-wide state sharing via the KV store. The client must be robust enough that transient Gemini API issues don't cascade into data loss or corrupt processing state. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/platform/services/remotecluster/service.go` (lines 103-122) — Reference HTTP client configuration: Transport with connection timeouts (30s dial, 30s keepalive), HTTP/2 support, idle connection management, TLS handshake timeout. This is the closest architectural match in the codebase for outbound HTTPS calls to external services. +- `server/platform/services/remotecluster/sendmsg.go` (lines 143-173) — Shows context-with-timeout pattern for outbound requests: `context.WithTimeout()`, `http.NewRequest()`, `req.WithContext()`, status code error checking, `defer resp.Body.Close()`. +- `server/channels/utils/backoff.go` — Existing progressive retry utility with predefined backoff timeout slices. Provides `ProgressiveRetry(operation)` and `CustomProgressiveRetry(operation, timeouts)`. Could be used as a reference for the exponential backoff implementation, though the Gemini client needs more control (e.g., response-code-specific behavior, jitter). +- `server/public/pluginapi/kv.go` — KV store wrapper with `Set(key, value, ...options)`, `Get(key, &dest)`, `SetExpiry(ttl)`, and `SetAtomicWithRetries(key, valueFunc)`. The atomic retry function (5 attempts, 10ms between retries) is useful for safely updating shared circuit breaker state across cluster nodes. +- `server/public/pluginapi/utils.go` — Additional backoff utilities available in the pluginapi package, similar to the core backoff patterns. +- `server/enterprise/elasticsearch/opensearch/common.go` — Shows TLS configuration for external services and retry-on-status patterns (retry on 429, 502, 503, 504) as configuration for the Elasticsearch client. + +## Reuse Opportunities + +- **KV store atomic operations**: The `pluginapi.KV.SetAtomicWithRetries` method provides built-in compare-and-swap with retry logic, ideal for updating circuit breaker state without race conditions across cluster nodes. +- **KV store TTL**: `pluginapi.KV.Set` with `SetExpiry(ttl)` can be used for rate limit backoff state that should auto-expire after the cooldown period. +- **Backoff utility**: The `server/channels/utils/backoff.go` progressive retry pattern provides a starting point, but the Gemini client needs response-code-aware backoff (e.g., only retry on 429 and 5xx, not on 400 or 401) and doubling intervals rather than fixed schedules. +- **Context timeout pattern**: All outbound HTTP calls in the codebase use `context.WithTimeout` — the Gemini client should follow this convention to ensure individual API calls have bounded latency. + +## Deliverables + +### Gemini client interface + +Define a Go interface for the Gemini client with four operations: + +**Generate embeddings** — Accepts a text string and returns a 768-dimensional float32 vector. The returned vector must be L2-normalized (unit magnitude). The client calls the gemini-embedding-001 model via the Google AI Go SDK, requesting the configured number of dimensions (768 by default). If the SDK returns a vector that is not already normalized, the client normalizes it before returning. + +**Extract knowledge** — Accepts conversation text (one or more messages assembled into a prompt) and returns a structured result containing zero or more extraction items. Each item has a type (decision, action_item, or topic), content text, and optional metadata (assignee for action items, confidence level). The client calls Gemini 2.5 Flash with a system prompt instructing the model to analyze the conversation and extract structured knowledge. The prompt design should yield parseable structured output (e.g., by requesting JSON output mode or using a structured extraction prompt format). This operation is used for individual post-level extraction during backfill (step 017) and for the processing worker's "extract" job type (step 007). + +**Summarize and extract (thread)** — Accepts conversation text (a full thread assembled as a multi-message prompt) and returns a combined result containing: a concise summary of the thread discussion (string), a list of key takeaways (important conclusions or outcomes, as an array of strings), and zero or more typed extraction items (decisions, action items, topics) with content and optional source post ID references. This is a single Gemini call that produces summary, takeaways, and extractions together, avoiding redundant calls. The client calls Gemini 2.5 Flash with a system prompt instructing the model to produce all three outputs in a single structured response. This operation is used by the thread debounce handler (step 008) and differs from "extract knowledge" in that it additionally produces a summary narrative and key takeaways alongside the structured extractions. + +**Synthesize answer (RAG)** — Accepts a user's natural language query, a list of context passages (each with a source identifier and content text), and optionally prior conversation history (for multi-turn). Returns a synthesized answer string with inline citation references (e.g., "[1]", "[2]") mapping to the provided source identifiers. The client calls Gemini 2.5 Flash with a system prompt that instructs the model to: answer only from the provided context, cite sources by reference number, and explicitly state when it cannot find relevant information rather than fabricating an answer. + +The interface must be mockable for unit testing — all four operations should be methods on the interface, not standalone functions. + +### Gemini client implementation + +Implement the interface using the `google.golang.org/genai` Go SDK: + +- Initialize the SDK client with the configured Gemini API key during plugin activation +- Use the configured model names (GeminiModel for LLM tasks, EmbeddingModel for embeddings) from the plugin configuration +- Each operation creates the appropriate request type for the SDK (embedding request vs. content generation request) +- All SDK calls must use a Go context with a timeout (30 seconds for embedding, 60 seconds for extraction and synthesis) to prevent unbounded blocking + +The client must be reconfigurable when the plugin configuration changes (new API key, new model names) without requiring a full plugin restart. The OnConfigurationChange lifecycle (from step 002) should propagate updated values to the client. + +### L2 normalization + +The embedding operation must L2-normalize all returned vectors before they are used by callers. L2 normalization divides each component of the vector by the vector's magnitude (square root of the sum of squared components). This ensures cosine similarity calculations in pgvector produce correct results. The normalization should handle edge cases: zero vectors (all components zero) should be returned as-is without division by zero. + +### Exponential backoff on rate limits + +When the Gemini API returns an HTTP 429 (Too Many Requests) response, the client must: + +1. Not immediately retry the request +2. Apply exponential backoff: start at 1 second, double on each consecutive 429 response, cap at 60 seconds +3. Add jitter to the backoff duration (±20%) to prevent thundering herd effects across cluster nodes +4. Store the current backoff state (next allowed request time) in the KV store with a TTL matching the backoff duration, so all cluster nodes respect the same cooldown +5. Before making any Gemini API call, check the KV store for an active backoff period and fail fast with a "rate limited" error if within the backoff window +6. Reset the backoff state after a successful API call + +### Circuit breaker + +Implement a circuit breaker pattern with three states: + +**Closed (normal operation)** — All requests pass through to the Gemini API. Track consecutive failure count. + +**Open (failing fast)** — After 5 consecutive failures (any error, not just 429), the breaker opens. While open, all Gemini API calls fail immediately with a "circuit open" error without making an actual API request. The breaker stays open for 30 seconds. + +**Half-open (testing)** — After the 30-second open period, the breaker transitions to half-open. The next request is allowed through as a test. If it succeeds, the breaker closes and resets the failure counter. If it fails, the breaker re-opens for another 30 seconds. + +Circuit breaker state (current state, failure count, last state transition time) must be stored in the KV store using atomic compare-and-swap operations so that all cluster nodes share a consistent view. When one node opens the breaker, all nodes should see it as open. + +### Error classification + +The client must classify Gemini API errors into categories that inform retry and circuit breaker behavior: + +- **Retryable errors**: HTTP 429 (rate limit), HTTP 500/502/503/504 (server errors), network timeouts, connection refused. These increment the circuit breaker failure counter and trigger backoff. +- **Non-retryable errors**: HTTP 400 (bad request — likely a prompt issue), HTTP 401/403 (authentication — API key problem). These should be returned immediately to the caller without triggering the circuit breaker, since retrying won't help. +- **Success**: Resets the backoff state and closes the circuit breaker if it was half-open. + +### OnActivate integration + +Wire the Gemini client into the plugin lifecycle: +1. During OnActivate (after configuration is loaded), create the Gemini client with the configured API key and model names +2. Store the client instance on the Plugin struct (in the placeholder field from step 002) +3. If the API key is missing, activation already fails (from step 002); the Gemini client creation should also validate that the SDK initializes without errors + +### OnConfigurationChange integration + +When configuration changes: +1. If the API key or model names have changed, reinitialize the Gemini SDK client with the new values +2. The circuit breaker and backoff state should be preserved across reconfiguration (they are in the KV store, not in-memory) + +## Acceptance Criteria + +- [x] The Gemini client interface defines four operations: generate embeddings, extract knowledge, summarize and extract (thread), and synthesize answer +- [x] The interface is a Go interface that can be mocked for unit testing +- [x] Embedding generation calls the gemini-embedding-001 model and returns a 768-dimensional float32 vector +- [x] Embedding vectors are L2-normalized — the magnitude of any returned vector equals 1.0 within floating-point tolerance +- [x] L2 normalization handles zero vectors without panicking (division by zero) +- [x] Knowledge extraction calls Gemini 2.5 Flash and returns typed extraction items (decision, action_item, topic) with content and optional metadata +- [x] Thread summarize-and-extract calls Gemini 2.5 Flash and returns a combined result: summary text, key takeaways array, and typed extraction items in a single response +- [x] RAG synthesis calls Gemini 2.5 Flash with a user query and context passages, and returns an answer with citation references +- [x] RAG synthesis instructs the model to state when it cannot find relevant information rather than fabricating answers +- [x] All Gemini API calls use a Go context with a timeout (30s for embeddings, 60s for LLM tasks) +- [x] When the Gemini API returns HTTP 429, the client applies exponential backoff starting at 1 second, doubling up to 60 seconds, with ±20% jitter +- [x] Backoff state is stored in the KV store so all cluster nodes respect the same cooldown +- [x] Subsequent API calls during an active backoff period fail fast without making an actual API request +- [x] After a successful API call, the backoff state is reset +- [x] After 5 consecutive failures, the circuit breaker opens and all calls fail immediately without contacting the API +- [x] The circuit breaker transitions to half-open after 30 seconds +- [x] A successful request in half-open state closes the circuit breaker and resets the failure counter +- [x] A failed request in half-open state re-opens the circuit breaker for another 30 seconds +- [x] Circuit breaker state is stored in the KV store using atomic operations for cluster-wide consistency +- [x] Non-retryable errors (HTTP 400, 401, 403) do not trigger the circuit breaker or backoff +- [x] The Gemini client is created during OnActivate and stored on the Plugin struct +- [x] Configuration changes (new API key or model names) reinitialize the SDK client without losing circuit breaker state diff --git a/tasks/0001-org-memory-ai-knowledge-base/005.md b/tasks/0001-org-memory-ai-knowledge-base/005.md new file mode 100644 index 000000000000..8deb4e1270b3 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/005.md @@ -0,0 +1,121 @@ +# 005: Chunking engine + +## Context + +Every post that enters the processing pipeline must be split into embeddable chunks before the Gemini client can generate vector embeddings. Raw posts range from a single sentence to the maximum 4000 runes (or 65535 bytes in v2), and file attachments can be far larger. The chunking engine sits between event ingestion (step 006) and the embedding/extraction worker (step 007): the worker fetches a post's content, passes it through the chunker to get a list of sized, overlapping text segments, and then sends each segment to the Gemini client (step 004) for embedding. The design prioritizes semantic coherence — chunks should break at paragraph and sentence boundaries rather than at arbitrary character counts — so that each embedded vector represents a meaningful unit of text that the RAG engine (step 009) can return as a useful citation. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/shared/markdown/paragraph.go` — Mattermost's markdown paragraph parser. Produces `Paragraph` blocks containing `Text []Range` (byte position ranges within the original markdown string). Part of the full markdown AST that decomposes content into blocks: paragraphs, lists, block quotes, fenced code blocks, etc. +- `server/public/shared/markdown/inspect.go` — Provides `Inspect(markdown string, f func(any) bool)` for depth-first traversal of the full markdown AST. Also provides `InspectBlock` and `InspectInline` for targeted traversal. Has a built-in max scanning length of 128KB (`maxLen = 1024 * 64 * 2`). +- `server/public/shared/markdown/inlines.go` — Defines inline elements within paragraphs: `Text`, `CodeSpan`, `SoftLineBreak`, `HardLineBreak`, `InlineLink`, `InlineImage`. Provides `MergeInlineText(inlines []Inline) []Inline` for consolidating adjacent text nodes and `ParseInlines` for converting ranges to typed inline elements. +- `server/public/shared/markdown/lines.go` — `ParseLines(markdown string) []Line` splits raw markdown into lines on `\n` and `\r` boundaries. Each `Line` has a `Range` with start `Position` and `End` byte offsets. +- `server/channels/utils/markdown.go` — `StripMarkdown(markdown string) (string, error)` converts Mattermost-flavored markdown to plain text using the goldmark parser. Also provides `StripMarkdownAndDecode` for HTML entity decoding. +- `server/public/model/utils.go` — `LimitRunes(s string, maxRunes int) (string, bool)` truncates text to a maximum rune count and returns whether truncation occurred. `LimitBytes(s string, maxBytes int) (string, bool)` does the same for byte count. Both handle UTF-8 correctly. +- `server/public/model/post.go` — Post message size limits: `PostMessageMaxRunesV1 = 4000`, `PostMessageMaxBytesV2 = 65535`. The `Post` struct carries `Message` (rendered content) and `MessageSource` (unmodified content for editing). `FileIds StringArray` links to attached files. +- `server/platform/services/docextractor/` — Document content extraction: `Extractor` interface with `Match(filename)`, `Extract(filename, reader, maxSize)`. Extractors exist for plain text (with Unicode validation), PDF, Office documents (DOC, DOCX, PPTX, ODT), HTML, and archives. The plain extractor validates that content contains only graphic Unicode characters and whitespace. +- `server/public/shared/markdown/markdown.go` — Low-level character classification: `isWhitespace(c rune)`, `isWord(c rune)`, `nextNonWhitespace(markdown, position)`, `nextLine(markdown, position)`. Used internally by the markdown parser for boundary detection. + +## Reuse Opportunities + +- **StripMarkdown** from `server/channels/utils/markdown.go` can convert Mattermost-formatted messages into plain text before chunking. This removes formatting noise (bold markers, link syntax, heading markers) that would otherwise dilute embedding quality. The chunker should strip markdown from post content as a preprocessing step. +- **Mattermost markdown parser** can be used to identify block-level boundaries (paragraphs, list items, code blocks, block quotes) within post content before falling back to character-level splitting. The `Inspect` traversal plus `Paragraph` block detection provides structural awareness of the content. +- **LimitRunes** provides safe UTF-8-aware truncation when a final chunk exceeds the target size after merging. This avoids splitting multi-byte characters at byte boundaries. +- **Document extractors** in `server/platform/services/docextractor/` handle file content extraction for attached documents. The chunking engine should accept pre-extracted text from these extractors as input alongside the post message text. +- **Rune counting** via `utf8.RuneCountInString` is used consistently throughout the codebase for text length measurement. The chunker should use rune counting (as a proxy for token counting) rather than byte counting, since rune count more closely approximates token count for most text. + +## Deliverables + +### Chunk result type + +Define a result type that represents a single chunk of text ready for embedding. Each chunk must carry: + +- The text content of the chunk (plain text, markdown stripped) +- The sequential chunk index (0-based) indicating the chunk's position within the original content +- The byte offset range within the original content that this chunk was derived from, so that citation display can map a matched chunk back to its approximate location in the source + +The chunking function returns a slice of these chunk results. + +### Chunking function + +Implement the main chunking function that accepts a text string and configuration parameters (target chunk size in approximate tokens, overlap percentage, maximum chunks) and returns an ordered slice of chunk results. The function implements the following algorithm: + +**Step 1: Preprocessing** — Strip markdown formatting from the input to produce clean plain text. Collapse multiple consecutive whitespace characters (including newlines beyond paragraph breaks) into single spaces within paragraphs. Trim leading and trailing whitespace from the full text. If the input is empty or whitespace-only after preprocessing, return an empty slice. + +**Step 2: Paragraph splitting** — Split the preprocessed text at paragraph boundaries (sequences of two or more consecutive newlines). Each resulting segment is a candidate paragraph. Preserve the relative ordering of paragraphs. + +**Step 3: Sentence splitting for oversized paragraphs** — For any paragraph that exceeds the target chunk size, further split it at sentence boundaries. A sentence boundary is defined as a period, exclamation mark, or question mark followed by whitespace and an uppercase letter, or followed by end-of-string. This heuristic handles common prose while avoiding false splits on abbreviations (e.g., "e.g.", "Dr.", "U.S.") by requiring the subsequent uppercase letter. If a single sentence still exceeds the target chunk size, it remains as-is (it will become an oversized chunk rather than being split mid-word). + +**Step 4: Merging small segments** — Iterate through the segments produced by steps 2 and 3. Merge consecutive small segments into a single chunk as long as the combined text does not exceed the target chunk size. When adding the next segment would exceed the target, finalize the current chunk and start a new one with the next segment. This ensures that short paragraphs and sentences are grouped together into reasonably-sized chunks rather than producing many tiny embeddings. + +**Step 5: Overlap insertion** — For each pair of adjacent chunks (chunk N and chunk N+1), prepend the last portion of chunk N to chunk N+1. The overlap size is calculated as a percentage (default 10%) of the target chunk size in approximate tokens. The overlap text is taken from the end of the preceding chunk, split at a word boundary to avoid cutting words. This preserves context across chunk boundaries so that information spanning a boundary can still be found by vector search on either adjacent chunk. + +**Step 6: Truncation** — If the total number of chunks exceeds the configured maximum (MaxChunksPerPost from the plugin configuration, default 32), truncate the list to the maximum count. Log a warning message including the original text length and the number of chunks that were discarded so administrators can tune the configuration if needed. + +**Step 7: Index assignment** — Assign sequential 0-based chunk indices to each chunk in the final list. These indices are stored alongside the embedding in the database (ChunkIndex column in ai_embeddings) and used for ordering when reconstructing content for citation display. + +### Token estimation + +The chunking engine needs to estimate token counts for text segments without calling the Gemini tokenizer API (which would add latency and API cost). Implement a token estimation function that approximates the token count of a text string. The estimation should use the heuristic that one token equals approximately 4 characters (runes) for English text, which is a widely-accepted approximation for modern language models. The estimation function is used for: + +- Determining whether a paragraph exceeds the target chunk size +- Calculating the merge threshold during the segment merging step +- Calculating the overlap size (10% of target chunk size) + +The target chunk size in the configuration is specified in approximate tokens (default ~500 tokens, which corresponds to roughly 2000 characters). The estimation need not be exact — it serves as a sizing heuristic, and slight over- or under-shooting is acceptable since the embedding model handles variable-length inputs. + +### File content integration + +The chunking engine must handle two input sources: + +**Post messages** — The primary input. The chunker receives the post's `Message` field content, strips markdown, and produces chunks as described above. + +**File attachment text** — When a post has file attachments with extracted text content (from Mattermost's built-in document extraction or the plugin's own extraction), the chunker receives the extracted text as a separate input. File content is chunked independently from the post message, with its own sequence of chunk indices. The caller (the processing worker in step 007) is responsible for setting the `SourceType` field on the resulting embeddings to distinguish post-derived chunks from file-derived chunks. + +The chunking function itself is source-agnostic — it accepts text and returns chunks. The caller determines the source type and passes it through when storing embeddings. + +### Edge case handling + +The chunking engine must handle the following edge cases gracefully: + +- **Empty input**: Return an empty slice, no error +- **Whitespace-only input**: Return an empty slice after preprocessing strips all content +- **Input shorter than target chunk size**: Return a single chunk containing the entire text (no splitting needed) +- **Single very long sentence with no sentence boundaries**: Return as a single oversized chunk (do not split mid-word). If it exceeds the target size, it is still a valid chunk — the embedding model can handle variable-length inputs. +- **Content that is entirely a code block**: Treat code blocks as opaque text segments — do not attempt sentence splitting within code. Code blocks from the markdown parser should be kept as atomic units during paragraph-level splitting. +- **Unicode content**: All text processing must be Unicode-aware, using rune iteration rather than byte indexing. The token estimation heuristic (4 chars per token) applies to rune counts, not byte counts. +- **Content with markdown tables**: Tables should be preserved as atomic segments during paragraph splitting, since splitting a table row from its header would produce meaningless chunks. +- **Extremely long content (e.g., large file extractions)**: Apply the maximum chunk limit to prevent runaway memory usage. After truncation, the chunks still represent the beginning of the content, which is often the most relevant portion. + +### Configuration integration + +The chunking engine reads three configuration values from the plugin configuration (step 002): + +- **MaxChunksPerPost** — The hard cap on chunks produced per input text (default 32). Used in the truncation step. +- **Target chunk size** — Not a separate configuration field; derived from the embedding model's practical limits. Use a constant of approximately 500 tokens (~2000 characters) as the target. This balances embedding quality (too-small chunks lose context) against retrieval precision (too-large chunks dilute the embedding signal). +- **Overlap percentage** — Not a separate configuration field; use a constant of 10%. This provides cross-boundary context without excessive duplication. + +The target chunk size and overlap percentage are compile-time constants rather than configuration fields because they are tuned to the embedding model's characteristics and changing them requires understanding embedding quality trade-offs. MaxChunksPerPost is user-configurable because it affects resource consumption (API calls, storage) rather than embedding quality. + +## Acceptance Criteria + +- [x] The chunking function accepts a text string and returns an ordered slice of chunk results, each containing the chunk text, a sequential 0-based index, and byte offset information +- [x] Markdown formatting is stripped from input text before chunking, producing clean plain text for embedding +- [x] Text is split at paragraph boundaries (double newlines) as the primary segmentation strategy +- [x] Paragraphs exceeding the target chunk size (~500 tokens) are further split at sentence boundaries (period/exclamation/question mark followed by whitespace and uppercase letter) +- [x] Consecutive small segments are merged together until the combined text approaches the target chunk size +- [x] Adjacent chunks have approximately 10% overlap — the end of chunk N is prepended to chunk N+1 +- [x] Overlap text is split at word boundaries to avoid cutting words +- [x] The total number of chunks is capped at MaxChunksPerPost (default 32), with excess chunks truncated and a warning logged +- [x] Token count estimation uses a rune-based heuristic (approximately 4 characters per token) without calling the Gemini tokenizer API +- [x] Empty input returns an empty slice without error +- [x] Whitespace-only input returns an empty slice after preprocessing +- [x] Input shorter than the target chunk size returns a single chunk containing the entire text +- [x] A single long sentence without sentence boundaries is returned as one oversized chunk (no mid-word splitting) +- [x] Code blocks are treated as atomic segments during paragraph splitting and are not split at sentence boundaries +- [x] All text processing is Unicode-aware, using rune counts rather than byte counts for length estimation +- [x] The chunking function is deterministic — identical input always produces identical output +- [x] Chunk indices are sequential starting from 0 and correspond to the chunk's position in the original content +- [x] The chunking function is a pure function (no side effects, no database access, no API calls) that can be unit tested in isolation diff --git a/tasks/0001-org-memory-ai-knowledge-base/006.md b/tasks/0001-org-memory-ai-knowledge-base/006.md new file mode 100644 index 000000000000..227e55abb612 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/006.md @@ -0,0 +1,137 @@ +# 006: Event ingestion hooks + +## Context + +The event ingestion layer is the real-time entry point for all content flowing into the knowledge base. Every post created, edited, or deleted in Mattermost passes through a corresponding plugin hook, and this step implements the handlers that decide what to do with each event. The ingestion hooks do not perform any heavy processing themselves — they make a quick eligibility check (skip bots, system messages, disabled teams), write a lightweight processing log entry to the database (step 003), and schedule or cancel debounce timers (step 008). The actual embedding, extraction, and summarization work is deferred to the background workers (steps 007 and 008), keeping the hook handlers fast and non-blocking. This separation is critical because hooks run synchronously in the Mattermost server's request path — a slow hook degrades the entire server's responsiveness. + +This step also handles file upload interception and user deactivation cleanup, completing the full set of event-driven entry points that feed the processing pipeline. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/hooks.go` — Defines all hook signatures. The relevant hooks for this step are: `MessageHasBeenPosted(c *Context, post *model.Post)`, `MessageHasBeenUpdated(c *Context, newPost, oldPost *model.Post)`, `MessageHasBeenDeleted(c *Context, post *model.Post)`, `FileWillBeUploaded(c *Context, info *model.FileInfo, file io.Reader, output io.Writer) (*model.FileInfo, string)`, and `UserHasBeenDeactivated(c *Context, user *model.User)`. +- `server/public/pluginapi/post.go` (lines 270-337) — `ShouldProcessMessage(post, ...options)` is a built-in helper that checks whether a post should be processed by a plugin. By default it skips system messages, webhook posts, and bot users. It also supports options for channel ID filtering and bot DM filtering. This is the primary eligibility check the hooks should use. +- `server/public/model/post.go` — Post struct with key fields: `RootId` (non-empty for threaded replies, identifies the thread root), `UserId`, `ChannelId`, `Message`, `FileIds` (string array of attached file IDs), `Type` (system message prefix check via `IsSystemMessage()`), and properties `PostPropsFromBot` and `PostPropsFromWebhook`. +- `server/public/model/file_info.go` — `FileInfo` struct with `Id`, `PostId`, `ChannelId`, `Name`, `Extension`, `MimeType`, `Size`, and `Content` (extracted text content, not sent to client). The `Content` field may already be populated by Mattermost's built-in document extraction. +- `server/public/pluginapi/cluster/job_once_scheduler.go` — `GetJobOnceScheduler(pluginAPI)` returns the singleton scheduler. `SetCallback(func(key string, props any))` registers the handler that fires when a scheduled job executes. `ScheduleOnce(key, runAt, props)` schedules a one-time job (fails if the key already exists — must Cancel first to reschedule). `Cancel(key)` removes a scheduled job. Props are serialized as JSON with a max size of 10KB. +- `server/channels/app/post.go` (lines 404-414) — Shows how the Mattermost server invokes `MessageHasBeenPosted` asynchronously via `a.Srv().Go(func() { ... })`. This confirms hooks run in a separate goroutine from the request handler, but they still block the hook dispatch pipeline for other plugins. +- `server/public/plugin/api.go` — Plugin API methods needed by the hooks: `GetChannel(channelId)` to resolve team membership for team-level enablement checks, `GetFileInfo(fileId)` to retrieve file metadata, `GetFile(fileId)` to retrieve raw file bytes when built-in extraction is unavailable, `GetPost(postId)` for fetching the full post when needed. +- `server/platform/services/docextractor/` — Document extraction implementations: plain text extractor (with Unicode validation), PDF extractor, Office document extractor (DOC, DOCX, PPTX, ODT, HTML, RTF). The `Extractor` interface defines `Match(filename) bool` and `Extract(filename, reader, maxSize) (string, error)`. These can be used to extract text from attached files when `FileInfo.Content` is empty. + +## Reuse Opportunities + +- **ShouldProcessMessage** from `pluginapi/post.go` handles the most common eligibility checks (skip bots, system messages, webhooks) out of the box. The hooks should call this first and return early if it says to skip. This avoids reimplementing bot detection logic and stays consistent with other Mattermost plugins. +- **JobOnceScheduler** is the cluster-safe mechanism for thread debounce. The cancel-then-reschedule pattern (cancel the existing debounce timer for a thread, then schedule a new one with a fresh future time) is exactly the pattern this step needs for the thread debounce trigger. The scheduler's singleton nature means only one callback is registered — this step sets up the callback, and step 008 implements the callback's dispatch logic. +- **Document extractors** from `server/platform/services/docextractor/` can be used to extract text from file attachments when Mattermost's built-in extraction hasn't populated the `FileInfo.Content` field. The plugin can instantiate the appropriate extractor based on the file extension and extract text content for embedding. +- **Channel lookup via GetChannel** provides the `TeamId` field needed to check whether a post belongs to an enabled team. The channel is fetched once per hook invocation and the team ID is compared against the plugin's configured enabled team list. +- **Post.RootId** provides thread identification without additional API calls. If `RootId` is non-empty, the post is a reply in a thread and the `RootId` is the thread root's post ID. If `RootId` is empty, the post is itself a root post, and its `Id` serves as the thread key. + +## Deliverables + +### MessageHasBeenPosted hook + +Implement the handler for new messages. When a post is created: + +1. **Eligibility check** — Call `ShouldProcessMessage` to skip bot posts, system messages, and webhook posts. Return immediately if the post should not be processed. + +2. **Team enablement check** — Resolve the post's channel to get the team ID. Check whether the team is in the plugin's enabled teams list (from configuration, step 002). If the team is not enabled, return immediately. If the enabled teams list is empty (meaning all teams are enabled), skip this check. + +3. **Create processing log entry** — Insert a new row into the `ai_processing_log` table (step 003) with status "pending", job type "embed", source type "post", the post's ID, channel ID, and current timestamp as the scheduled time. This entry tells the embedding worker (step 007) that this post needs embedding. + + **Design note — embed only, no extract:** Real-time ingestion creates only "embed" entries, not "extract" entries. Knowledge extraction (decisions, action items, topics) happens at the thread level via the debounce summarization handler (step 008), not per individual post. Individual post-level extraction entries are only created during historical backfill (step 017) for posts that are not part of multi-message threads. This avoids redundant extraction — a single thread-level extraction with full conversation context produces higher-quality results than per-post extraction in isolation. + +4. **Trigger thread debounce** — Determine the thread root ID: use `Post.RootId` if non-empty, otherwise use `Post.Id` (the post is itself a thread root). Cancel any existing debounce job for this thread root (using the key pattern `debounce:{rootId}`), then schedule a new one-time job via the `JobOnceScheduler` set to fire after the configured debounce interval (from the plugin configuration's debounce seconds setting). The job's props should include the thread root ID and channel ID so the debounce handler (step 008) knows which thread to summarize. + +5. **File content extraction** — If the post has attached files (`FileIds` is non-empty), iterate over each file ID. For each file, retrieve the `FileInfo` to check whether extracted text content is already available in the `Content` field. If content is available, create an additional processing log entry with job type "embed" and source type "file" for the file's content. If content is not available, retrieve the raw file bytes via the plugin API, attempt text extraction using the document extractors (matching by file extension), and if successful, create the processing log entry for the extracted text. If extraction fails or the file type is unsupported, skip that file silently (log at debug level). + +### MessageHasBeenUpdated hook + +Implement the handler for edited messages. When a post is updated: + +1. **Eligibility check** — Same as the posted hook: call `ShouldProcessMessage` on the new post to check if it should be processed. Return immediately if not. + +2. **Team enablement check** — Same as the posted hook. + +3. **Mark for reprocessing** — Check whether the post's message content actually changed by comparing `newPost.Message` to `oldPost.Message`. If the content is identical (e.g., only metadata changed), return without action. If the content changed, use the store's **upsert** operation (not the idempotent create) to insert a processing log entry with status "pending", job type "embed", and source type "post". The upsert resets an existing entry's status to "pending" even if it was previously "complete", ensuring the edited post gets reprocessed. This is distinct from the idempotent create used for initial ingestion, which skips entries that already exist. + +4. **Re-trigger thread debounce** — An edit to a message in a thread may change the thread's summary. Follow the same debounce cancel-and-reschedule pattern as the posted hook. + +### MessageHasBeenDeleted hook + +Implement the handler for deleted messages. When a post is deleted: + +1. **Cancel pending processing** — Cancel any pending debounce job for the deleted post's thread. If the deleted post is a thread root (`RootId` is empty), cancel the debounce job keyed to the post's own ID. If it's a reply, cancel-and-reschedule the debounce for the parent thread (the thread still exists, just lost one message, so its summary should be regenerated). + +2. **Clean up derived data** — Call the store's delete methods to remove all embeddings, extractions, and entity mentions associated with the deleted post's ID. The database schema (step 003) uses ON DELETE CASCADE from the Posts table, so if the post row is deleted by Mattermost core, the cascade handles cleanup automatically. However, the hook should also explicitly delete derived data as a defense-in-depth measure, since the cascade depends on foreign key constraints which may not exist if the plugin's tables use a different ID scheme than direct FK references. + +3. **Update processing log** — Mark any pending or in-progress processing log entries for the deleted post as "complete" (nothing to do), preventing the worker from attempting to process a post that no longer exists. + +### FileWillBeUploaded hook + +Implement the handler for file uploads. This hook fires before the file is committed to storage, giving the plugin access to the file content stream: + +1. **Content extraction opportunity** — Read the file content from the provided `io.Reader` and attempt text extraction using the document extractors. If extraction succeeds, store the extracted text in a plugin-scoped KV store entry keyed by the file ID, so that when the corresponding `MessageHasBeenPosted` hook fires for the post containing this file, the extracted text is available without a second file read. + +2. **Pass-through behavior** — This hook must not reject or modify the file upload. Return `nil` for the modified FileInfo and an empty string for the rejection reason, allowing the upload to proceed normally. The plugin is only observing, not gatekeeping. + +3. **Size and type guard** — Only attempt extraction for files under a reasonable size threshold (e.g., 10MB) and for supported file types (text, PDF, Office documents). For unsupported types or oversized files, skip extraction silently and let the `MessageHasBeenPosted` hook handle what it can. + +### UserHasBeenDeactivated hook + +Implement the handler for user deactivation: + +1. **Clean up conversation sessions** — Delete all multi-turn conversation session entries from the KV store for the deactivated user. Sessions are keyed by user ID plus session identifier (from step 009), so the cleanup should use `KVList` with a prefix matching the user's ID pattern and delete each matching entry. + +2. **Preserve knowledge content** — Do not delete any embeddings, extractions, summaries, or other knowledge base content created from the user's posts. The knowledge belongs to the conversation context, not the individual user. Other users' RAG queries may depend on this content. + +### JobOnceScheduler initialization + +During plugin activation (OnActivate, extending step 002), set up the one-time job scheduler: + +1. **Get the singleton scheduler** — Call `GetJobOnceScheduler` with the plugin API to obtain the shared scheduler instance. + +2. **Register the callback** — Call `SetCallback` with a dispatcher function that inspects the job key prefix to determine which handler to invoke. Jobs with the `debounce:` prefix are dispatched to the thread debounce handler (implemented in step 008). Jobs with the `backfill:` prefix are dispatched to the backfill handler (implemented in step 017). Unrecognized prefixes are logged as warnings and ignored. + +3. **Start the scheduler** — Call `Start()` to begin polling for scheduled jobs. This must happen after `SetCallback` and before any hooks fire, so it should be one of the last initialization steps in OnActivate. + +4. **Shutdown** — During OnDeactivate (extending step 002), cancel all pending scheduled jobs and stop the scheduler to prevent jobs from firing after the plugin has shut down. List all scheduled jobs via `ListScheduledJobs()` and cancel each one. + +### Enabled teams resolution + +The team enablement check used by the message hooks needs efficient access to the configured team list: + +1. **Parse on configuration change** — When the plugin configuration changes (OnConfigurationChange from step 002), parse the comma-separated enabled team IDs string into a set (map) for O(1) lookup. Store this set on the Plugin struct. + +2. **Empty means all** — If the enabled teams configuration is empty or not set, the plugin processes posts from all teams. The hooks should check for this case and skip the team lookup entirely. + +3. **Channel-to-team resolution** — For each post, the hook needs to know which team owns the channel. Call `GetChannel(channelId)` to retrieve the channel's `TeamId`. For direct message channels (which have an empty TeamId), always process them regardless of team configuration, since DMs are cross-team. + +## Acceptance Criteria + +- [x] MessageHasBeenPosted hook skips bot posts, system messages, and webhook posts using `ShouldProcessMessage` +- [x] MessageHasBeenPosted hook skips posts from teams not in the enabled teams list (when the list is non-empty) +- [x] MessageHasBeenPosted hook creates a processing log entry with "pending" status and "embed" job type for each new post +- [x] MessageHasBeenPosted hook determines the thread root ID (Post.RootId if non-empty, otherwise Post.Id) for debounce scheduling +- [x] MessageHasBeenPosted hook cancels any existing debounce job for the thread root and schedules a new one at the configured debounce interval in the future +- [x] MessageHasBeenPosted hook creates processing log entries for file attachments when extractable text content is available +- [x] MessageHasBeenPosted hook attempts document extraction for file attachments when the FileInfo.Content field is empty, using supported extractors +- [x] MessageHasBeenPosted hook creates processing log entries with source type "post" for the post message and source type "file" for each file attachment, using the idempotent create operation (skip if exists) +- [x] Real-time ingestion creates only "embed" job type entries, not "extract" — knowledge extraction is handled at the thread level by the debounce handler (step 008) +- [x] MessageHasBeenUpdated hook only creates reprocessing entries when the post message content actually changed (compares newPost.Message to oldPost.Message) +- [x] MessageHasBeenUpdated hook uses the store's upsert operation (not idempotent create) to reset existing "complete" entries back to "pending" for reprocessing +- [x] MessageHasBeenUpdated hook re-triggers the thread debounce when content changes +- [x] MessageHasBeenDeleted hook cancels the debounce job for a deleted thread root post +- [x] MessageHasBeenDeleted hook reschedules the debounce for the parent thread when a reply is deleted (the thread still exists but its content changed) +- [x] MessageHasBeenDeleted hook removes all derived data (embeddings, extractions, entity mentions) for the deleted post +- [x] MessageHasBeenDeleted hook marks pending processing log entries for the deleted post as "complete" +- [x] FileWillBeUploaded hook extracts text content from supported file types and stores it in the KV store keyed by file ID +- [x] FileWillBeUploaded hook never rejects or modifies the file upload — always returns nil FileInfo and empty rejection string +- [x] FileWillBeUploaded hook skips extraction for unsupported file types and files exceeding the size threshold +- [x] UserHasBeenDeactivated hook deletes all multi-turn conversation sessions for the deactivated user from the KV store +- [x] UserHasBeenDeactivated hook does not delete any knowledge base content (embeddings, extractions, summaries) belonging to the user's posts +- [x] The JobOnceScheduler is initialized during OnActivate with a callback that dispatches jobs by key prefix +- [x] The JobOnceScheduler is shut down during OnDeactivate (all pending jobs cancelled, scheduler stopped) +- [x] Direct message channels (empty TeamId) are always processed regardless of the enabled teams configuration +- [x] All hook handlers return quickly — no Gemini API calls, no heavy computation, only database writes and scheduler operations +- [x] File extraction failures are logged at debug level and do not prevent the post from being processed diff --git a/tasks/0001-org-memory-ai-knowledge-base/007.md b/tasks/0001-org-memory-ai-knowledge-base/007.md new file mode 100644 index 000000000000..acf15e82c7d6 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/007.md @@ -0,0 +1,167 @@ +# 007: Processing worker for embedding and extraction + +## Context + +The event ingestion hooks (step 006) write lightweight "pending" entries to the `ai_processing_log` table but perform no AI processing. This step builds the background worker that drains that queue — a recurring job that polls for pending items, acquires a distributed lock per post to prevent duplicate work across cluster nodes, calls the Gemini client (step 004) via the chunking engine (step 005) to generate embeddings and extract knowledge, stores results in the database (step 003), and updates the processing log status. The worker must be resilient to crashes, API failures, and cluster topology changes. If a worker node dies mid-processing, another node must detect the stale item and recover it. If the Gemini API is unavailable (circuit breaker open), the worker must back off gracefully without consuming resources. + +This is the engine that converts raw posts into searchable knowledge. Every embedding, every extraction, and every entity mention flows through this worker. Its throughput and reliability directly determine how quickly new content becomes queryable and how much data is lost during failures. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/pluginapi/cluster/job.go` — `Schedule(pluginAPI, key, nextWaitInterval, callback)` creates a recurring job backed by a distributed mutex. `MakeWaitForInterval(duration)` creates a wait function that fires at fixed intervals relative to last completion. The job loop acquires the mutex, reads metadata from KV store, checks if it's time to run, executes the callback, and updates the `LastFinished` timestamp. The callback runs holding the cluster mutex, so only one node executes per interval. +- `server/public/pluginapi/cluster/mutex.go` — `NewMutex(pluginAPI, key)` creates a distributed mutex. `Lock()` blocks with exponential backoff (1s-5min, ±500ms jitter) until the mutex is acquired. `Unlock()` releases via KV store deletion. TTL is 15 seconds, refreshed every 7.5 seconds while held. If a node crashes while holding the lock, the TTL expires and another node can acquire it. +- `server/public/pluginapi/cluster/wait.go` — Backoff and jitter utilities used internally by the cluster package: `nextWaitInterval` doubles on error and resets on success, `addJitter()` adds ±500ms randomization. These patterns should inform the worker's own polling interval behavior. +- `server/channels/jobs/jobs_watcher.go` — Mattermost's core job watcher pattern: a goroutine that polls the database on a fixed interval, queries for pending jobs, and distributes them to workers. Uses `time.After` with jitter for the poll interval. Default polling interval is 15 seconds. The worker should follow this general pattern but at a faster cadence (1 second) since embedding latency is the primary bottleneck. +- `server/channels/jobs/simple_worker.go` — The core `SimpleWorker` implementation: receive job from channel, claim it with optimistic locking (pending → in-progress status transition), execute the work, set success or error status. This claim-execute-complete pattern is exactly what the processing worker needs. +- `server/channels/jobs/jobs.go` (lines 211-247) — `UpdateInProgressJobData` uses `LastActivityAt` timestamps as heartbeats. `HandleJobPanic` recovers from panics, captures goroutine stack trace, sets job status to error, then re-panics. The processing worker should implement both patterns — heartbeat for stale detection and panic recovery for crash safety. +- `server/public/pluginapi/cluster/job_once.go` (lines 14-37) — Retry logic with failure limits: `maxNumFails = 3`, increment counter on each failure, give up after exceeding the limit. This maps directly to the processing log's retry counter behavior. +- `server/public/pluginapi/log.go` — Plugin logging: `LogError(msg, keyValuePairs...)`, `LogWarn`, `LogInfo`, `LogDebug`. Key-value pairs are alternating string keys and typed values. Used throughout the cluster package for structured logging. + +## Reuse Opportunities + +- **cluster.Schedule with MakeWaitForInterval** is the primary mechanism for the recurring worker. Schedule a job with a 1-second interval that fires the worker's poll-and-process cycle. The cluster scheduling already handles distributed coordination — only one node runs the worker at a time, and the mutex automatically failover if a node crashes. +- **cluster.Mutex** provides per-post locking. Within each poll cycle, the worker queries for pending items, then attempts to acquire a mutex keyed to each post's ID before processing it. If another node's worker is already processing the same post (e.g., after a race condition during failover), the mutex prevents duplicate work. The 15-second TTL provides automatic recovery if the lock holder crashes. +- **SimpleWorker claim pattern** from the core jobs system (pending → in-progress → complete/error) maps directly to the processing log's status field. The worker updates status to "processing" after acquiring the mutex (claim), performs the work, then updates to "complete" or increments the retry counter and returns to "pending" (on failure) or marks as "error" (after max retries). +- **HandleJobPanic pattern** should be adapted for the worker's per-item processing loop. Wrap each item's processing in a deferred panic recovery that logs the stack trace and marks the item as errored, preventing a single bad input from crashing the entire worker loop. +- **Structured logging** from the cluster package (alternating key-value pairs) should be used consistently for all worker log messages, including post ID, job type, processing duration, and error details for each item. + +## Deliverables + +### Worker initialization and lifecycle + +Create a recurring background worker using `cluster.Schedule` that starts during plugin activation (OnActivate, extending steps 002 and 006) and stops during deactivation: + +1. **Schedule the worker** — Call `cluster.Schedule` with a 1-second interval (`MakeWaitForInterval(1 * time.Second)`) and the worker's main callback function. Store the returned `*Job` handle on the Plugin struct for shutdown. + +2. **Shutdown** — During OnDeactivate, call `Close()` on the job handle to stop the recurring schedule. The worker should finish processing the current item (if any) before stopping. Do not cancel mid-processing — let the current item complete to avoid leaving it in a "processing" state. + +3. **Plugin struct integration** — Store the worker's job handle alongside the other scheduler handles (from step 006) so that all background jobs can be shut down together during deactivation. + +### Poll-and-process cycle + +Each time the recurring worker fires (every ~1 second, coordinated by the cluster scheduler), it executes one poll-and-process cycle: + +1. **Check circuit breaker** — Before querying for pending items, check whether the Gemini circuit breaker is open (step 004). If the circuit breaker is open, skip this cycle entirely and return. This prevents the worker from claiming items it can't process, which would just increment their retry counters needlessly. + +2. **Query pending items** — Call the store's processing log query method (step 003) to fetch a batch of pending items ordered by scheduled time (oldest first). Limit the batch size to a reasonable number (e.g., 10) to prevent a single poll cycle from holding the cluster mutex for too long. Only select items whose scheduled time is in the past (i.e., they are ready to process). + +3. **Process each item** — For each pending item in the batch, process it individually with its own error handling (described below). If any item's processing encounters a circuit breaker open error from the Gemini client, stop processing the remaining batch and return — the API is unavailable, and continuing would waste time. + +4. **Return** — After processing the batch (or early-returning due to circuit breaker), the cycle is complete. The cluster scheduler will fire the next cycle approximately 1 second later. + +### Per-item processing + +For each individual pending item from the processing log: + +1. **Acquire per-post mutex** — Create a distributed mutex keyed to the item's post ID (with a distinguishing prefix to avoid collision with other mutex users, e.g., `worker:{postId}`). Attempt to lock it with a short timeout — if another node is already processing this post, skip the item and move to the next one. Do not block indefinitely waiting for the lock. + +2. **Update status to processing** — Once the mutex is acquired, update the processing log entry's status from "pending" to "processing" and record the current timestamp. This serves as the claim marker — other workers will not pick up this item because it is no longer "pending". + +3. **Fetch post content** — Retrieve the full post via the plugin API (`GetPost`). If the post no longer exists (it was deleted between queuing and processing), mark the processing log entry as "complete" and release the mutex. This is a normal condition, not an error. + +4. **Route by job type** — The processing log entry specifies a job type. Route to the appropriate handler: + + - **embed** — Pass the post's message content through the chunking engine (step 005) to produce text chunks. For each chunk, call the Gemini client's embedding generation operation (step 004) to produce a 768-dimensional vector. Store each chunk's embedding in the `ai_embeddings` table (step 003) with the chunk's text content, index, channel ID, and source type. If the item has a source type of "file", use the file's extracted text content instead of the post message. + + - **extract** — Pass the post's message content (and thread context if available) to the Gemini client's knowledge extraction operation (step 004). Parse the structured response into individual extraction items (decisions, action items, topics). Store each extraction in the `ai_extractions` table (step 003). For each extraction that identifies an entity (project name, technology, person), create corresponding entries in the `ai_entity_mentions` table. + +5. **Mark complete** — After successful processing, update the processing log entry's status to "complete" and record the completion timestamp. + +6. **Handle failure** — If any step fails (Gemini API error, database error, unexpected error): + - Increment the item's retry count in the processing log + - Record the error message in the processing log's error field + - If the retry count has reached the maximum (3 retries), set the status to "error" permanently — the item will not be retried again + - If the retry count is below the maximum, return the status to "pending" so the item will be picked up again on a future poll cycle + - Log the error with structured fields (post ID, job type, retry count, error message) + +7. **Release mutex** — Unlock the per-post mutex after processing completes (success or failure). Use a deferred unlock to ensure the mutex is released even if a panic occurs during processing. + +8. **Panic recovery** — Wrap the per-item processing in a deferred panic recovery handler. If a panic occurs, log the error with a stack trace, mark the processing log entry as "error", and release the mutex. Do not let a single item's panic terminate the entire poll cycle — continue processing the remaining items in the batch. + +### Stale item detection + +Items can become stuck in "processing" status if the worker node crashes after claiming them but before completing them. The mutex TTL (15 seconds) ensures the lock is released, but the processing log status remains "processing" indefinitely. Implement stale detection: + +1. **Detection query** — As part of each poll cycle (after processing the pending batch), query for items with "processing" status whose status update timestamp is older than a staleness threshold (5 minutes). This threshold is much longer than the expected processing time for any single item (typically under 30 seconds), so items exceeding it are almost certainly stuck. + +2. **Recovery action** — For each stale item, reset its status to "pending" and increment its retry count. Log a warning including the post ID, how long the item has been stuck, and the original error (if any). The item will be picked up by the next poll cycle and reprocessed. + +3. **Max retry still applies** — If a stale item has already reached the maximum retry count, set its status to "error" instead of returning it to "pending". This prevents infinite retry loops for items that consistently crash the worker. + +### Batch size and throughput management + +The worker processes items in small batches rather than one at a time to amortize the overhead of polling and mutex acquisition: + +1. **Configurable batch size** — Use a reasonable default batch size of 10 items per poll cycle. This is not exposed in the plugin configuration (it's a tuning parameter, not a user-facing setting) but is defined as a named constant for easy adjustment. + +2. **Sequential processing within batch** — Items within a batch are processed sequentially, not concurrently. This simplifies error handling, prevents Gemini API rate limit bursts, and makes the worker's behavior predictable. The 1-second poll interval combined with batches of 10 provides a maximum theoretical throughput of 10 items per second, which is sufficient for real-time ingestion since users don't post faster than the worker can process. + +3. **Empty batch fast return** — If the pending query returns no items, return immediately from the poll cycle. Do not busy-wait or add artificial delay — the cluster scheduler's 1-second interval already provides the pacing. + +### Embedding job handler + +When the job type is "embed", the worker performs the full embedding pipeline for a single post: + +1. **Determine content source** — Check the processing log entry's source type. If "post", use the post's message content. If "file", retrieve the extracted file text from the KV store (where it was stored by the FileWillBeUploaded hook in step 006) or from the FileInfo's Content field via the plugin API. + +2. **Chunk the content** — Pass the text content through the chunking engine (step 005). If the chunking engine returns an empty slice (empty or whitespace-only content), mark the processing log entry as "complete" (nothing to embed) and return. + +3. **Generate embeddings** — For each chunk returned by the chunking engine, call the Gemini client's embedding generation operation. Each call returns a 768-dimensional L2-normalized vector. + +4. **Store embeddings** — For each chunk, insert a row into the `ai_embeddings` table with: a new ID (via `model.NewId()`), the post ID, the channel ID, the embedding vector, the chunk's text content (as the content snapshot for citation display), the chunk index, the source type, and a creation timestamp. Use the store interface from step 003. + +5. **Delete previous embeddings** — Before storing new embeddings (for reprocessing due to post edits), delete any existing embeddings for this post ID and source type. This ensures edited posts get fresh embeddings rather than accumulating stale ones alongside new ones. The delete-then-insert sequence is safe because it occurs within the per-post mutex. + +### Extraction job handler + +When the job type is "extract", the worker performs knowledge extraction for a single post: + +1. **Prepare context** — Assemble the post's message content. If the post is part of a thread (RootId is non-empty), optionally include recent preceding messages in the thread to provide context for the extraction. Limit the context to a reasonable amount (e.g., the 5 most recent messages before this post) to avoid excessive Gemini token usage. + +2. **Call extraction** — Pass the assembled text to the Gemini client's knowledge extraction operation. The response contains zero or more typed extraction items (decisions, action items, topics) with content text and optional metadata. + +3. **Store extractions** — For each extraction item, insert a row into the `ai_extractions` table with: a new ID, the post ID, the channel ID, the extraction type, the content text, structured metadata (as JSON), and creation timestamp. + +4. **Create entity mentions** — For each extraction that identifies named entities (project names, technologies, people), insert rows into the `ai_entity_mentions` table with: the normalized entity name (lowercased), the entity type, a reference to the extraction's ID, the post ID, the channel ID, and creation timestamp. + +5. **Delete previous extractions** — Before storing new results, delete existing extractions and entity mentions for this post ID. Same rationale as for embeddings — edits should produce fresh extractions. + +### Metrics integration points + +The worker should track key operational metrics (for step 014 to expose): + +1. **Items processed counter** — Increment on each successful completion, tagged by job type (embed/extract). +2. **Items failed counter** — Increment on each failure, tagged by job type and whether it was a permanent error (max retries reached) or a transient retry. +3. **Processing latency** — Record the wall-clock time from claim (status → processing) to completion for each item, tagged by job type. +4. **Queue depth** — Record the number of pending items returned by each poll query. +5. **Stale items recovered counter** — Increment each time a stale item is reset to pending. + +These metrics are collected as in-memory values on the Plugin struct and exposed by the ServeMetrics hook (step 014). This step only needs to update the counters at the appropriate points — the metrics infrastructure is built later. + +## Acceptance Criteria + +- [x] The processing worker is scheduled via `cluster.Schedule` with a 1-second interval and starts during OnActivate +- [x] The worker stops gracefully during OnDeactivate, finishing the current item before shutting down +- [x] Each poll cycle queries the processing log for pending items ordered by scheduled time (oldest first), limited to a batch size of 10 +- [x] The worker skips the poll cycle entirely when the Gemini circuit breaker is open +- [x] Each item is processed within a per-post distributed mutex to prevent duplicate work across cluster nodes +- [x] The processing log status transitions from "pending" to "processing" when an item is claimed +- [x] The processing log status transitions from "processing" to "complete" on successful processing +- [x] On failure, the retry count is incremented and the item returns to "pending" status for retry +- [x] After 3 consecutive failures on the same item, the status is set to "error" permanently +- [x] The error message is recorded in the processing log for failed items +- [x] If a post no longer exists when processing begins, the item is marked "complete" without error +- [x] The embedding job handler chunks content via the chunking engine and generates one embedding per chunk via the Gemini client +- [x] Embeddings are stored with the chunk's text content, sequential index, post ID, channel ID, and source type +- [x] Previous embeddings for a post are deleted before storing new ones (to handle reprocessing after edits) +- [x] The extraction job handler calls Gemini's knowledge extraction and stores typed results (decisions, action items, topics) with metadata +- [x] Entity mentions are created for named entities identified in extractions +- [x] Previous extractions and entity mentions for a post are deleted before storing new ones +- [x] Stale items stuck in "processing" status for more than 5 minutes are detected and reset to "pending" (or "error" if max retries reached) +- [x] Panic recovery is implemented per item — a panic during one item's processing does not crash the worker loop +- [x] The per-post mutex is always released after processing (success, failure, or panic), using deferred unlock +- [x] The worker stops processing the current batch and returns early if a Gemini circuit breaker open error is encountered mid-batch +- [x] Empty batches (no pending items) return immediately without artificial delay +- [x] The worker logs structured messages at appropriate levels: debug for routine operations, warn for stale recovery, error for permanent failures diff --git a/tasks/0001-org-memory-ai-knowledge-base/008.md b/tasks/0001-org-memory-ai-knowledge-base/008.md new file mode 100644 index 000000000000..29370ca8c21a --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/008.md @@ -0,0 +1,119 @@ +# 008: Thread debounce and summarization + +## Context + +Individual posts are embedded by the processing worker (step 007), but threads have a higher-order structure that individual post embeddings miss — the progression of a discussion, the decisions that emerge from back-and-forth dialogue, and the consensus that forms across multiple messages. Thread summarization captures this structure by waiting until a thread goes quiet, then feeding the full conversation to Gemini for summarization and knowledge extraction at the thread level. The "debounce" is critical: without it, every single reply would trigger a summarization, wasting Gemini API quota on partial thread states. Instead, the plugin schedules a one-time job each time a reply arrives in a thread, cancelling the previous timer, so the summarization only fires after the configured quiet period (e.g., 30 seconds of inactivity). + +Step 006 already handles the schedule/cancel/reschedule pattern in the message hooks and initializes the `JobOnceScheduler`. This step implements the callback that fires when a debounce timer expires — the handler that fetches the thread, assembles the conversation text, calls Gemini for summarization, stores the result, and ensures individual post embeddings are queued for any posts that haven't been processed yet. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/channels/app/summarization.go` — Mattermost's existing AI summarization implementation. `buildConversationTextWithIDs(posts)` assembles a conversation string from a slice of posts, formatting each as `[HH:MM] username (Post ID: xxx): message\n` and collecting post IDs for citation. `SummarizePosts` sends the assembled text to an LLM with a system prompt that requests highlights and action items in JSON format with source permalinks. This is the closest architectural reference for the plugin's own thread summarization. +- `server/channels/app/recap.go` — `fetchPostsForRecap` demonstrates thread/channel post retrieval: fetches via `GetPostsSince`, converts `PostList` (map + order array) to an ordered slice, enriches each post with the author's username via `GetUser`, and limits to a configurable count. This fetch-enrich-slice pattern should be followed when assembling thread content. +- `server/channels/jobs/recap/worker.go` — The recap background worker shows the full lifecycle: `processRecapJob` iterates channels, calls `ProcessRecapChannel` per channel, tracks progress, handles partial failures (some channels fail, others succeed), publishes WebSocket updates on status changes. The progress tracking and partial failure handling patterns are relevant. +- `server/public/pluginapi/cluster/job_once_scheduler.go` — The singleton `JobOnceScheduler`. `SetCallback(func(key string, props any))` registers the dispatch function. When a scheduled job fires, the callback receives the key (without the `once_` storage prefix) and the deserialized props. Step 006 registers the callback; this step implements the handler that the callback dispatches to for `debounce:` prefixed keys. +- `server/public/plugin/api.go` — `GetPostThread(postId string) (*model.PostList, *model.AppError)` retrieves all posts in a thread given the root post ID. Returns a `PostList` with `Order` (post IDs in creation order) and `Posts` (map of post ID to Post). Also: `GetUser(userId string) (*model.User, *model.AppError)` for enriching posts with usernames. +- `server/public/model/post_list.go` — `PostList` struct with `Order []string` and `Posts map[string]*Post`. `ToSlice()` returns ordered `[]*Post`. `SortByCreateAt()` sorts posts by creation timestamp. These methods simplify converting the thread retrieval result into an ordered conversation. +- `server/public/model/recap.go` — `AIRecapSummaryResponse` with `Highlights []string` and `ActionItems []string` — the response shape Mattermost uses for its recap feature. The plugin's thread summary storage (step 003: `ai_thread_summaries` table) uses a similar structure with a summary text field and key takeaways JSON array. + +## Reuse Opportunities + +- **Conversation text assembly** from `buildConversationTextWithIDs` in `summarization.go` provides a proven pattern for formatting thread posts into LLM-ready text. The plugin should follow the same format — timestamp, username, post ID, message per line — so that the Gemini extraction prompt can reference specific posts by ID for citation. +- **Post enrichment with usernames** from `fetchPostsForRecap` in `recap.go` demonstrates the pattern of calling `GetUser` per post to replace user IDs with human-readable usernames in the conversation text. The plugin should do the same, caching user lookups within the thread processing to avoid repeated API calls for users who posted multiple messages. +- **PostList to ordered slice conversion** using `PostList.Order` iteration plus `PostList.Posts` map lookup is the standard pattern for working with `GetPostThread` results. The plugin should iterate `Order` to get chronological post sequence. +- **Structured JSON output for summaries** as demonstrated in `summarizePostsJSONSchema` — the plugin's Gemini extraction prompt should request structured JSON with highlights, decisions, action items, and key takeaways, similar to the existing recap schema but extended with the plugin's extraction types (decisions, action items, topics). +- **WebSocket status publishing** from `publishRecapUpdate` in the recap worker can inform the plugin's own pattern for notifying the webapp when a thread summary becomes available (step 016). + +## Deliverables + +### Debounce callback handler + +Implement the handler function that the `JobOnceScheduler` callback dispatches to when a debounce job fires (key prefix `debounce:`). Step 006 registered the callback and routes by prefix; this step provides the actual handler implementation: + +1. **Parse the job key and props** — Extract the thread root post ID and channel ID from the job's key or props. The key format established in step 006 is `debounce:{rootId}`, and the props include the channel ID. + +2. **Validate the thread still exists** — Call `GetPost` with the thread root post ID. If the root post has been deleted since the debounce was scheduled, log at debug level and return without processing. This is a normal condition when a thread root is deleted after the last reply. + +3. **Fetch the full thread** — Call `GetPostThread` with the root post ID to retrieve all posts in the thread. The result is a `PostList` containing the root post and all replies. + +4. **Convert to ordered post slice** — Iterate the `PostList.Order` array, looking up each post in `PostList.Posts`, to build a chronologically ordered slice. Filter out system messages and bot posts from the slice (the same eligibility criteria as the message hooks in step 006). + +5. **Check thread size** — If the thread contains fewer than 2 eligible posts (only the root, no meaningful replies), skip summarization. A single message doesn't constitute a conversation worth summarizing. Log at debug level and return. + +6. **Enrich posts with usernames** — For each post in the ordered slice, call `GetUser` to retrieve the author's username. Cache user lookups within this processing call to avoid redundant API calls for users who posted multiple times in the thread. + +7. **Assemble conversation text** — Format the enriched post slice into a conversation text string following the pattern from `buildConversationTextWithIDs`: each post as a line with timestamp, username, post ID, and message content. Collect the post IDs into a separate list for citation reference. + +8. **Handle large threads** — If the thread exceeds a message count threshold (100 messages), apply a map-reduce approach: split the ordered posts into groups (e.g., chunks of 50 messages), summarize each group separately via Gemini, then synthesize the group summaries into a final thread summary with a second Gemini call. This bounds token usage and latency for very long threads. For threads under the threshold, proceed with a single summarization call. + +9. **Call Gemini for summarization** — Pass the conversation text to the Gemini client's "summarize and extract" operation (step 004). This is a dedicated operation (separate from "extract knowledge") that returns a combined result containing: a concise summary of the thread discussion, a list of key takeaways, and typed extraction items (decisions, action items, topics) with content and source post ID references — all from a single Gemini call. + +10. **Store the thread summary** — Insert or update a row in the `ai_thread_summaries` table (step 003) keyed by the thread root post ID. The row includes: the summary text, key takeaways as a JSON array, the model version used for generation, and the current timestamp as the last activity time. If a summary already exists for this thread (from a previous debounce cycle), update it with the new content — threads evolve as new replies arrive, so summaries must be regenerated. + +11. **Store extractions** — For each extraction item (decision, action item, topic) produced by Gemini, store it in the `ai_extractions` table with the thread root post ID as the source. Create corresponding entity mention entries in `ai_entity_mentions` for any named entities. Before storing, delete previous thread-level extractions for this root post ID to avoid accumulating stale extractions from earlier summarization runs. + +12. **Queue unprocessed individual embeddings** — After summarization, check whether each individual post in the thread has been processed for embedding (query the processing log for each post). For any post that does not yet have a "complete" embedding entry in the processing log, create a new "pending" entry so the processing worker (step 007) will embed it on its next poll cycle. This ensures that even if a post's initial embedding was missed (e.g., the worker was down when the post arrived), the debounce handler catches it. + +### Thread summary embedding + +After storing the thread summary, also embed the summary text itself so it becomes searchable via the RAG engine: + +1. **Create a processing log entry** — Insert a "pending" entry in the processing log with job type "embed" and a source type of "summary" (distinct from "post" and "file") for the thread root post ID. + +2. **The processing worker handles it** — The embedding worker (step 007) will pick up this entry on its next poll cycle, chunk the summary text, and generate embeddings. The source type "summary" tells the worker that the content comes from the thread summary text rather than the raw post message. + +### Error handling + +The debounce handler operates outside the processing worker's retry framework (it's triggered by the `JobOnceScheduler`, not polled from the processing log), so it needs its own error handling: + +1. **Gemini API failure** — If the Gemini client returns an error (circuit breaker open, rate limited, API error), the handler should reschedule the debounce job for a short future time (e.g., 60 seconds) to retry. This reuses the same schedule/cancel pattern from step 006 — schedule a new debounce job for the same thread root with a future time. The retry should be attempted up to 3 times (tracked in the job's props). After 3 failed attempts, log an error and abandon the summarization — the thread's individual posts are still embedded by the worker, so no data is permanently lost. + +2. **Database failure** — If storing the summary or extractions fails, log the error and do not reschedule. The next time activity occurs in the thread, a new debounce cycle will trigger and attempt summarization again. + +3. **Panic recovery** — Wrap the entire handler in a deferred panic recovery. Log the stack trace and move on — a panic in one thread's summarization must not crash the scheduler or affect other debounce jobs. + +### WebSocket notification + +After successfully storing a thread summary, publish a custom WebSocket event to notify connected clients that a summary is available for this thread: + +1. **Event name** — Pass `thread_summary_ready` as the event name to `PublishWebSocketEvent`. The plugin API automatically prepends `custom_org-memory_` (the `custom_{pluginid}_` prefix), so the full wire-format event name received by clients is `custom_org-memory_thread_summary_ready`. The webapp (step 016) registers its WebSocket handler using this fully-prefixed name. + +2. **Event payload** — Include the thread root post ID and channel ID so the webapp (step 016) knows which thread to update. + +3. **Broadcast scope** — Broadcast the event to the channel where the thread lives. Only users who are members of the channel and currently connected will receive it. Use the plugin API's `PublishWebSocketEvent` method. + +### Configuration dependencies + +The debounce handler uses these configuration values from step 002: + +- **Debounce interval** — The quiet period before summarization fires. Used by step 006 when scheduling; the handler itself doesn't read this value directly (it fires when the timer expires). +- **Large thread threshold** — The message count above which the map-reduce approach is used (default 100). This is a compile-time constant, not a user-facing configuration field. +- **Model version tracking** — The handler stores the Gemini model name (from configuration) alongside the summary to track which model version produced it. If the model is upgraded later, existing summaries can be identified for regeneration. + +## Acceptance Criteria + +- [x] The debounce callback handler is dispatched when the `JobOnceScheduler` fires a job with the `debounce:` key prefix +- [x] The handler fetches the full thread via `GetPostThread` and converts it to a chronologically ordered post slice +- [x] Bot posts and system messages are filtered out of the thread before summarization +- [x] Threads with fewer than 2 eligible posts are skipped without calling Gemini +- [x] Each post is enriched with the author's username (via `GetUser`) for the conversation text, with user lookups cached within the handler +- [x] Conversation text follows the format: timestamp, username, post ID, and message per line +- [x] For threads exceeding 100 messages, a map-reduce approach splits the thread into groups, summarizes each group, then synthesizes a final summary +- [x] The Gemini client's "summarize and extract" operation is called with the conversation text and returns a combined result: summary, key takeaways, and typed extractions +- [x] The thread summary is stored in `ai_thread_summaries` with summary text, key takeaways, model version, and activity timestamp +- [x] If a summary already exists for the thread root, it is replaced with the updated summary +- [x] Extraction items (decisions, action items, topics) are stored in `ai_extractions` with the thread root post ID as source +- [x] Previous thread-level extractions for the root post are deleted before storing new ones +- [x] Entity mentions are created for named entities identified in thread-level extractions +- [x] Individual posts in the thread that lack "complete" embedding entries get new "pending" processing log entries +- [x] A processing log entry with source type "summary" is created for the thread summary text so it gets embedded by the worker +- [x] If the thread root post has been deleted, the handler returns without error +- [x] If the Gemini API fails, the debounce job is rescheduled for 60 seconds in the future, up to 3 retry attempts +- [x] After 3 failed retry attempts, the summarization is abandoned with an error log +- [x] Database write failures are logged but do not trigger retries (the next thread activity will trigger a new debounce cycle) +- [x] Panic recovery is implemented — a panic in one thread's summarization does not affect other debounce jobs +- [x] A custom WebSocket event (`custom_org-memory_thread_summary_ready`) is published after successful summary storage +- [x] The WebSocket event includes the thread root post ID and channel ID and is broadcast to the channel +- [x] The model version used for summarization is recorded in the thread summary row diff --git a/tasks/0001-org-memory-ai-knowledge-base/009.md b/tasks/0001-org-memory-ai-knowledge-base/009.md new file mode 100644 index 000000000000..396b31cf0330 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/009.md @@ -0,0 +1,190 @@ +# 009: RAG query engine with permission filtering + +## Context + +This is the core intelligence layer that turns stored knowledge into answers. When a user asks "What did we decide about the database migration?" the RAG query engine must: resolve which channels the user can read, embed the query into a vector, search for the most similar content chunks across only those permitted channels, fetch surrounding context for the top results, send everything to Gemini for answer synthesis, and format the response with clickable citation links back to the original posts. Every step must respect Mattermost's permission model — a user must never see knowledge derived from channels they cannot access. + +The engine also supports multi-turn conversation: after an initial answer, users can ask follow-up questions that build on the previous context. This requires maintaining short-lived session state (conversation history) in the KV store, with a TTL to automatically clean up inactive sessions. + +This step has no UI surface of its own — it is a backend component called by the HTTP API (step 010) and the slash command handler (step 010). Its interface must be clean and mockable so both callers and tests can drive it independently. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/api.go` (line 380) — `GetTeamsForUser(userID string) ([]*model.Team, *model.AppError)` returns all teams the user belongs to in a single call (not paginated). This is the starting point for multi-team permission resolution. +- `server/public/plugin/api.go` (line 598) — `GetChannelMembersForUser(teamID, userID string, page, perPage int) ([]*model.ChannelMember, *model.AppError)` returns paginated channel memberships for a user within a specific team. Requires iterating over all teams and all pages to build the complete channel list. +- `server/public/plugin/api.go` (line 1002) — `HasPermissionToChannel(userID, channelId string, permission *model.Permission) bool` checks whether a user has a specific permission on a channel. Used as a fallback verification for individual result items. +- `server/public/model/permission.go` — `PermissionReadChannelContent` is the permission constant for reading channel messages. This is the permission to check for RAG result filtering — users who can read a channel's content can see knowledge derived from it. +- `server/public/plugin/api.go` (line 52) — `GetConfig() *model.Config` returns the server configuration. `Config.ServiceSettings.SiteURL` provides the base URL for constructing post permalinks in the format `{SiteURL}/pl/{postId}`. +- `server/public/plugin/api.go` (line 721) — `GetPost(postId string) (*model.Post, *model.AppError)` retrieves a full post for enriching search results with original content and thread context. +- `server/public/plugin/api.go` (line 976) — `PublishWebSocketEvent(event string, payload map[string]any, broadcast *model.WebsocketBroadcast)` for sending real-time events. The event name is automatically prefixed with `custom_{pluginid}_`. +- `server/public/pluginapi/kv.go` — KV store with TTL support: `Set(key, value, SetExpiry(ttl))` stores a value that auto-expires after the specified duration. `Get(key, &dest)` retrieves and unmarshals. `Delete(key)` removes explicitly. Values max ~256KB per key. +- `server/public/model/channel_member.go` — `ChannelMember` struct with `ChannelId` and `UserId`. Returned by `GetChannelMembersForUser`. The `ChannelId` values form the permission-allowed set for vector search filtering. +- `server/public/model/channel.go` — `Channel` struct with `TeamId` and `Type`. Channel types include `"O"` (open/public), `"P"` (private), `"G"` (group message), `"D"` (direct message). Direct messages have an empty `TeamId`. +- `server/channels/app/summarization.go` — `buildConversationTextWithIDs` provides the pattern for assembling context passages with post IDs for citation reference. The RAG engine's context assembly for Gemini follows a similar approach — each passage includes source identifiers that map back to post permalinks. + +## Reuse Opportunities + +- **GetTeamsForUser + GetChannelMembersForUser pagination** is the standard multi-team channel enumeration pattern. The engine should build a set of allowed channel IDs by iterating teams, then paginating through channel memberships per team. This set is reused for both the vector search filter and result validation. +- **KV store with TTL** for permission caching. Fetching channel memberships involves multiple paginated API calls across multiple teams, which is expensive for users in many channels. Cache the resolved channel ID set in the KV store with a 60-second TTL keyed by user ID. Subsequent queries within the TTL window hit the cache instead of re-resolving permissions. +- **KV store with TTL** for multi-turn session state. Conversation history (previous query/answer pairs) is stored in the KV store keyed by user ID + session ID with a 30-minute TTL. This provides automatic cleanup of inactive sessions without a background sweeper. +- **HasPermissionToChannel** as a secondary verification. The cached channel membership set provides the primary permission filter for vector search, but individual result items can be double-checked against `HasPermissionToChannel(userID, channelId, PermissionReadChannelContent)` for defense-in-depth — a user's permissions may have changed since the cache was populated. +- **SiteURL from GetConfig** for permalink construction. The engine formats citation links as `{SiteURL}/pl/{postId}`, which is the standard Mattermost post permalink format. + +## Deliverables + +### RAG query engine interface + +Define a Go interface for the RAG query engine with two operations: + +**Query** — Accepts a user ID and query text. Returns a structured response containing: the synthesized answer text with inline citation markers, a list of citation objects (each with a reference number, post ID, permalink URL, and content snippet), and a session ID for follow-up queries. This is the entry point for new queries from the HTTP API and slash command. + +**FollowUp** — Accepts a user ID, query text, and session ID. Returns the same response structure as Query. Loads the previous conversation history from the session, appends the new query, and sends the full context to Gemini for synthesis. If the session ID is invalid or expired, falls back to a fresh query (no error — sessions are best-effort). + +The interface must be mockable for testing the HTTP API and slash command handlers independently of the RAG pipeline. + +### Permission resolution + +Implement the channel permission resolver that determines which channels a user can read: + +1. **Check cache** — Look up the user's cached channel ID set in the KV store with key `perms:{userId}`. If found and not expired (TTL manages expiry automatically), use the cached set. + +2. **Resolve from API** — If no cache hit, build the permitted channel set: + - Call `GetTeamsForUser(userId)` to get all teams + - For each team, paginate through `GetChannelMembersForUser(teamId, userId, page, perPage)` with a page size of 200, collecting all `ChannelId` values into a set + - Continue paginating until a page returns fewer results than the page size + - The resulting set contains all channels the user is a member of across all teams + +3. **Cache the result** — Store the channel ID set in the KV store with a 60-second TTL. The set is stored as a serialized list of channel ID strings. + +4. **Handle direct messages** — Direct message channels (type `"D"` and `"G"`) have an empty `TeamId` and are not returned by team-scoped channel membership queries. However, `GetChannelMembersForUser` with an empty string as teamID may not work. If direct message channels are not returned by the team iteration, the engine should accept them by checking `HasPermissionToChannel` on individual result items that reference channels not in the cached set. This ensures DM-derived knowledge is accessible to participants. + +### Query vector generation + +When a query is received: + +1. **Embed the query** — Pass the user's query text through the Gemini client's embedding generation operation (step 004) to produce a 768-dimensional query vector. The same embedding model (gemini-embedding-001) and L2 normalization used for content embeddings ensures the query vector is in the same embedding space. + +2. **Handle embedding failure** — If the Gemini client returns an error (circuit breaker open, rate limited), return a user-friendly error to the caller indicating the knowledge base is temporarily unavailable. Do not retry — the HTTP handler or slash command can decide whether to communicate this to the user. + +### Vector similarity search + +Execute a permission-filtered vector search against the `ai_embeddings` table: + +1. **Build the query** — Construct a SQL query that selects from `ai_embeddings` where: + - The `ChannelId` is in the user's permitted channel set (from permission resolution) + - The `DeleteAt` is 0 (not soft-deleted) + - Results are ordered by cosine distance (ascending — smaller distance = more similar) using the pgvector `<=>` operator between the stored embedding vector and the query vector + - Limited to the top N results (default 10) + +2. **Use the replica database** — Read queries should use `GetReplicaDB()` to avoid loading the primary database with search traffic. + +3. **Return result rows** — Each result row includes: the embedding ID, post ID, channel ID, content snapshot (the chunk text stored alongside the embedding), chunk index, source type, and the cosine distance score. + +### Result enrichment + +For each search result, fetch additional context to provide Gemini with enough information for accurate synthesis: + +1. **Fetch the source post** — Call `GetPost(postId)` to retrieve the full post. If the post no longer exists (deleted between embedding and query), skip this result. + +2. **Permission double-check** — Verify that the user still has `PermissionReadChannelContent` on the result's channel. If the cache is stale (user was removed from a channel after caching), skip this result. This is a defense-in-depth measure. + +3. **Fetch thread context** — If the post is part of a thread (RootId is non-empty), fetch the thread summary from `ai_thread_summaries` if available. The thread summary provides broader conversation context that helps Gemini understand how the matched chunk fits into a larger discussion. + +4. **Build passage object** — For each valid result, create a context passage containing: a reference number (sequential, starting from 1), the content snapshot from the embedding, the post author's username (via `GetUser`), the post's creation timestamp, the channel name (via `GetChannel`), and the thread summary snippet (if available). These passages are sent to Gemini for synthesis. + +5. **Generate permalink** — Construct the post permalink as `{SiteURL}/pl/{postId}` using the server's `SiteURL` from configuration. Store it on the passage for citation link formatting. + +### Answer synthesis + +Send the enriched context to Gemini for answer generation: + +1. **Assemble the synthesis prompt** — Build a prompt for the Gemini client's RAG synthesis operation (step 004) that includes: + - The user's query + - All context passages, each labeled with a reference number + - Instructions to answer only from the provided context + - Instructions to cite sources by reference number (e.g., "[1]", "[2]") + - Instructions to state explicitly when the provided context does not contain enough information to answer the query, rather than fabricating an answer + - For multi-turn follow-ups: the previous conversation history (query/answer pairs from the session) + +2. **Call Gemini** — Pass the assembled prompt to the Gemini client's synthesis operation. The response is the raw answer text with inline citation markers. + +3. **Handle no results** — If the vector search returned zero results (no matching content in the user's accessible channels), skip the Gemini call and return a canned response: "I couldn't find any relevant information in your accessible channels. Try rephrasing your question or asking about a different topic." + +4. **Handle synthesis failure** — If the Gemini client returns an error, return a user-friendly error to the caller. Do not return partial results — either the full synthesized answer or an error. + +### Citation formatting + +Parse the Gemini-generated answer and replace citation markers with actual post links: + +1. **Parse citation references** — Scan the answer text for citation markers in the format `[N]` where N is a reference number. Map each reference number back to the corresponding passage object. + +2. **Format as Mattermost markdown** — Replace each citation marker with a Mattermost-compatible markdown link. The format should be `[N]({permalink})` where the permalink navigates to the original post. When clicked in the Mattermost client, this link navigates directly to the source post. + +3. **Build citation list** — In addition to inline citations, build a list of all cited sources with: reference number, post permalink, channel name, author username, and a short content snippet. This list is returned alongside the answer for the UI to display as a "Sources" section. + +4. **Handle missing citations** — If Gemini references a number that doesn't correspond to a provided passage (hallucinated citation), strip the marker from the answer rather than linking to nothing. Log a warning when this occurs. + +### Multi-turn session management + +Support follow-up queries that build on previous context: + +1. **Session creation** — When a new query is processed (no session ID provided), generate a new session ID (using `model.NewId()`) and include it in the response. The caller (HTTP API or slash command) passes this ID back for follow-up queries. + +2. **Session storage** — Store the conversation history in the KV store with key `session:{userId}:{sessionId}`. The value is a list of query/answer pairs (the user's query text and Gemini's synthesized answer). Set a TTL of 30 minutes on each write — every interaction refreshes the TTL, so sessions stay alive as long as the user is actively querying. + +3. **Follow-up context loading** — When a follow-up query arrives with a session ID, load the session from the KV store. If the session exists, append the conversation history to the synthesis prompt so Gemini can reference prior answers. If the session has expired or doesn't exist, treat it as a fresh query. + +4. **History truncation** — If the conversation history grows large enough to approach the KV store's value size limit (~256KB), truncate the oldest query/answer pairs, keeping only the most recent exchanges. This ensures writes succeed while preserving recent context. A practical limit is approximately 10 exchanges before considering truncation. + +5. **Session update** — After each successful query or follow-up, update the session in the KV store with the new exchange appended and a refreshed TTL. + +### Query audit logging + +After each query (initial or follow-up), record an entry in the `ai_query_log` table (step 003): + +1. **Log fields** — Record: query ID (via `model.NewId()`), user ID, query text, the list of channel IDs that were searched, the number of results returned by vector search, the number of results after enrichment filtering, the total processing latency (wall-clock time from query receipt to response), and the timestamp. + +2. **Write-only** — The query log is append-only. No updates or deletes are performed on query log entries. + +3. **Async if possible** — The audit log write should not delay the query response to the user. If the store interface supports fire-and-forget inserts, use that. Otherwise, a synchronous write is acceptable since it's a single insert. + +### Metrics integration points + +The engine should track key operational metrics (for step 014 to expose): + +1. **Query count** — Increment on each query (new and follow-up), tagged by type. +2. **Query latency** — Record the end-to-end latency from query receipt to response delivery. +3. **Search result count** — Record the number of vector search results before and after permission filtering and enrichment. +4. **Permission cache hit/miss** — Track how often the cached channel set is used versus freshly resolved. +5. **Session hit/miss** — Track how often follow-up queries find a valid session versus falling back to fresh queries. + +## Acceptance Criteria + +- [x] The RAG engine interface defines Query and FollowUp operations that are mockable for testing +- [x] Query accepts a user ID and query text and returns a synthesized answer with citations and a session ID +- [x] FollowUp accepts a user ID, query text, and session ID and returns the same response structure +- [x] Permission resolution fetches the user's channel memberships across all teams via GetTeamsForUser and paginated GetChannelMembersForUser +- [x] The resolved channel ID set is cached in the KV store with a 60-second TTL keyed by user ID +- [x] Subsequent queries within the cache TTL window use the cached channel set without re-resolving +- [x] The query text is embedded via the Gemini client's embedding operation to produce a 768-dimensional query vector +- [x] Vector similarity search queries ai_embeddings filtered by the user's permitted channel IDs and excluding soft-deleted entries +- [x] Search results are ordered by cosine distance (ascending) and limited to the top 10 +- [x] Vector search uses the replica database connection (GetReplicaDB) +- [x] Each search result is enriched with the original post content via GetPost +- [x] Results whose posts no longer exist are skipped gracefully +- [x] A permission double-check via HasPermissionToChannel filters out results from channels the user has since lost access to +- [x] Thread summaries from ai_thread_summaries are included as additional context for results that are part of threads +- [x] Post permalinks are generated in the format {SiteURL}/pl/{postId} +- [x] The synthesis prompt includes all context passages with reference numbers, the user's query, and instructions to cite sources and not fabricate answers +- [x] When no vector search results are found, a canned "no results" message is returned without calling Gemini +- [x] Citation markers in the answer ([N]) are replaced with Mattermost markdown links to post permalinks +- [x] Hallucinated citation references (numbers not matching any provided passage) are stripped from the answer +- [x] A citation list is returned alongside the answer with reference number, permalink, channel name, author, and content snippet +- [x] Multi-turn sessions are stored in the KV store with key session:{userId}:{sessionId} and 30-minute TTL +- [x] Each query interaction refreshes the session TTL +- [x] Expired or missing sessions on follow-up queries fall back to fresh queries without error +- [x] Conversation history is truncated when approaching the KV store size limit, keeping the most recent exchanges +- [x] Query audit log entries are written to ai_query_log with user ID, query text, channels searched, result count, and latency +- [x] Gemini API failures return a user-friendly error message to the caller diff --git a/tasks/0001-org-memory-ai-knowledge-base/010.md b/tasks/0001-org-memory-ai-knowledge-base/010.md new file mode 100644 index 000000000000..d53450050af3 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/010.md @@ -0,0 +1,248 @@ +# 010: HTTP API router and slash command handler + +## Context + +Up to this point, the knowledge base is fully operational but invisible to users. Posts are ingested, chunked, embedded, extracted, and summarized — but no user-facing interface exists to query it or inspect its contents. This step builds the HTTP API layer and slash command handler that expose the pipeline to consumers: the webapp (steps 015-016), external integrations, and command-line users. + +The HTTP API provides 13 endpoints under `/plugins/org-memory/api/v1/` covering the full feature surface: RAG queries, thread and channel summaries, entity timelines, extraction management, admin operations, and webhook configuration. Every endpoint requires authentication, and admin endpoints require system administrator privileges. The slash command provides a lightweight alternative — users can type `/ask ` in any channel to get an ephemeral response without opening the RHS panel. + +Both the API and slash command are thin dispatch layers. They parse inputs, check permissions, delegate to the RAG engine (step 009) or store (step 003), format outputs, and return responses. No business logic should live in the handlers themselves — that belongs in the engine and store. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/hooks.go` (lines 109-116) — `ServeHTTP(c *Context, w http.ResponseWriter, r *http.Request)` hook signature. The Mattermost server strips the `/plugins/{plugin_id}` prefix from the request path before passing it to the plugin, so the plugin receives paths starting with `/api/v1/...`. The `Mattermost-User-Id` header is set by the server after session validation — if present, the request is authenticated. +- `server/public/plugin/hooks.go` (lines 118-122) — `ExecuteCommand(c *Context, args *model.CommandArgs) (*model.CommandResponse, *model.AppError)` hook signature. Receives the parsed command arguments including the user's ID, channel ID, team ID, and the full command text. +- `server/public/plugin/plugintest/example_hello_user_test.go` (line 25-30) — Reference ServeHTTP implementation: extracts `Mattermost-User-Id` header, calls `GetUser`, writes response. Shows the minimal pattern for an authenticated plugin HTTP handler. +- `server/public/model/command.go` (lines 21-45) — `Command` struct for registration: `Trigger` (the slash command word), `AutoComplete` (enable autocomplete), `AutoCompleteDesc` (description in autocomplete), `AutoCompleteHint` (hint text showing expected arguments), `DisplayName`, `Description`. +- `server/public/model/command_args.go` — `CommandArgs` struct: `UserId`, `ChannelId`, `TeamId`, `RootId`, `Command` (the full command string including the trigger word), `SiteURL`. +- `server/public/model/command_response.go` — `CommandResponse` struct: `ResponseType` (`CommandResponseTypeEphemeral` for private responses), `Text` (the response body as markdown), `Props`, `Attachments`. Returned from `ExecuteCommand`. +- `server/public/plugin/api.go` (line 29) — `RegisterCommand(command *model.Command) error` registers a slash command during plugin activation. Previously wired in step 002. +- `server/public/plugin/api.go` (line 685) — `SendEphemeralPost(userID string, post *model.Post) *model.Post` creates a post visible only to one user. Useful for longer slash command responses that need rich formatting. +- `server/public/plugin/api.go` (lines 984-1002) — `HasPermissionTo(userID string, permission *model.Permission) bool` for system-scope permission checks. `HasPermissionToChannel(userID, channelId string, permission *model.Permission) bool` for channel-scope checks. +- `server/public/model/permission.go` — `PermissionManageSystem` is the system administrator permission. `PermissionReadChannelContent` for verifying read access to channel data. +- `server/channels/app/plugin_requests.go` (lines 156-274) — Server-side authentication flow: extracts token from Authorization header (Bearer/Token prefix), Cookie, or query parameter. Validates session. Sets `Mattermost-User-Id` header after CSRF checks. The plugin can trust this header because the server clears it before validation and only re-sets it for valid sessions. +- `server/channels/app/integration_action.go` (line 255) — Request body size limiting via `io.LimitReader` with a 1MB cap to prevent memory exhaustion attacks. The plugin should apply the same pattern for POST/PUT endpoints. + +## Reuse Opportunities + +- **Mattermost-User-Id header authentication** is provided by the server's plugin request pipeline. The plugin does not need to implement session validation — if the header is present, the user is authenticated. If absent, the request is unauthenticated and should be rejected with HTTP 401. This is the standard plugin authentication pattern. +- **CommandResponse with ephemeral type** for slash command responses. The `CommandResponseTypeEphemeral` constant ensures the response is visible only to the invoker. For longer or richer responses, `SendEphemeralPost` can be used instead, which supports full Mattermost markdown. +- **HasPermissionTo with PermissionManageSystem** provides the admin guard for admin-only endpoints (backfill, status, webhook management). This is the standard system admin check used across Mattermost plugins. +- **HasPermissionToChannel with PermissionReadChannelContent** provides per-channel access control for summary, entity, and extraction endpoints. Before returning data associated with a channel, the handler must verify the requesting user can read that channel. +- **io.LimitReader pattern** from integration action handlers prevents oversized request bodies from consuming memory. Apply to all POST and PUT endpoints with a 1MB limit. + +## Deliverables + +### ServeHTTP router + +Implement the `ServeHTTP` hook to route incoming HTTP requests to the appropriate handler based on the request method and path. The router dispatches under the `/api/v1/` prefix (the server has already stripped `/plugins/org-memory`): + +1. **Authentication gate** — Before dispatching to any handler, extract the `Mattermost-User-Id` header. If the header is empty, reject the request with HTTP 401 Unauthorized. Store the user ID for the handler to use. + +2. **Path routing** — Parse the request path and HTTP method to determine which handler to invoke. The router supports these routes: + + - `POST /api/v1/query` → query handler + - `POST /api/v1/query/followup` → follow-up query handler + - `GET /api/v1/threads/{root_id}/summary` → thread summary handler + - `GET /api/v1/channels/{channel_id}/summary` → channel summary handler + - `GET /api/v1/entities/{name}/timeline` → entity timeline handler + - `GET /api/v1/extractions` → extractions list handler + - `PUT /api/v1/extractions/{id}` → extraction edit handler + - `POST /api/v1/admin/backfill` → backfill trigger handler + - `GET /api/v1/admin/status` → pipeline status handler + - `POST /api/v1/webhooks` → webhook create handler + - `GET /api/v1/webhooks` → webhook list handler + - `DELETE /api/v1/webhooks/{id}` → webhook delete handler + +3. **Not found handling** — If no route matches, return HTTP 404 with a JSON error body. + +4. **Method not allowed** — If the path matches but the HTTP method does not, return HTTP 405. + +5. **Path parameter extraction** — For routes with path parameters (e.g., `{root_id}`, `{channel_id}`, `{name}`, `{id}`), extract the parameter value from the path segments. Use standard library path parsing — no external router dependency is required, though `gorilla/mux` is available in the dependency tree if preferred. + +### JSON request/response conventions + +Establish consistent JSON handling across all endpoints: + +1. **Request body parsing** — For POST and PUT endpoints, read the request body through a size-limited reader (1MB max). Decode the JSON body into the appropriate request struct. If the body is malformed, return HTTP 400 with a descriptive error message. + +2. **Response encoding** — All responses use JSON content type (`Content-Type: application/json`). Success responses include a JSON body with the relevant data. Error responses include a JSON body with an `error` field containing a human-readable message and a `status` field with the HTTP status code. + +3. **Content type enforcement** — For POST and PUT endpoints, verify the `Content-Type` header includes `application/json`. If not, return HTTP 415 Unsupported Media Type. + +### Query endpoint (POST /api/v1/query) + +Handles new RAG queries: + +1. **Parse request** — Read the JSON body containing the query text (required, non-empty string). +2. **Call RAG engine** — Invoke the RAG engine's Query operation (step 009) with the authenticated user's ID and the query text. +3. **Return response** — Return the synthesized answer, citation list, and session ID as JSON. The session ID is included so the webapp can send follow-up queries. +4. **Error handling** — If the RAG engine returns an error (Gemini unavailable, etc.), return HTTP 503 with the user-friendly error message. + +### Follow-up query endpoint (POST /api/v1/query/followup) + +Handles multi-turn follow-up queries: + +1. **Parse request** — Read the JSON body containing the query text (required) and session ID (required, non-empty string). +2. **Call RAG engine** — Invoke the RAG engine's FollowUp operation with the user's ID, query text, and session ID. +3. **Return response** — Same response format as the query endpoint, with an updated session ID. +4. **Expired session handling** — If the session has expired, the RAG engine falls back to a fresh query. The response is still successful — the caller does not need to handle session expiry explicitly. + +### Thread summary endpoint (GET /api/v1/threads/{root_id}/summary) + +Returns the thread summary and extractions for a given thread: + +1. **Extract path parameter** — Parse the `root_id` from the URL path. +2. **Permission check** — Fetch the root post via `GetPost` to determine its channel. Check `HasPermissionToChannel(userId, channelId, PermissionReadChannelContent)`. If the user cannot read the channel, return HTTP 403. +3. **Fetch data** — Query the store for the thread summary (from `ai_thread_summaries`) and extractions (from `ai_extractions`) where the post ID matches the root ID. +4. **Return response** — Return the summary text, key takeaways, extractions (decisions, action items, topics), and the summary's generation timestamp. If no summary exists yet, return HTTP 404 with a message indicating the thread has not been summarized. + +### Channel summary endpoint (GET /api/v1/channels/{channel_id}/summary) + +Returns daily or weekly channel summaries: + +1. **Extract path parameter and query params** — Parse the `channel_id` from the path and `type` (daily/weekly) from query parameters. Default to "daily" if not specified. +2. **Permission check** — Check `HasPermissionToChannel(userId, channelId, PermissionReadChannelContent)`. Return HTTP 403 if denied. +3. **Fetch data** — Query the store for the most recent channel summary of the requested type for this channel. +4. **Return response** — Return the summary text, time window (start and end timestamps), and generation timestamp. If no summary exists, return HTTP 404. + +### Entity timeline endpoint (GET /api/v1/entities/{name}/timeline) + +Returns cross-channel mentions of a named entity: + +1. **Extract path parameter** — Parse the entity `name` from the path. URL-decode it to handle spaces and special characters. +2. **Fetch mentions** — Query the store for entity mentions matching the normalized name (lowercased), ordered by creation time descending. +3. **Permission filter** — Filter the results to only include mentions from channels the user can read. Use the permission resolver from step 009 (cached channel set) to filter efficiently rather than checking each mention individually. +4. **Return response** — Return a chronological list of mentions, each with: the entity name, source post ID and permalink, channel name, extraction content, and timestamp. If no mentions are found, return an empty list (not a 404). + +### Extractions list endpoint (GET /api/v1/extractions) + +Returns extractions filtered by channel and/or type: + +1. **Parse query parameters** — Read optional `channel_id` and `type` (decision/action_item/topic) filters from query parameters. Also support `page` and `per_page` for pagination (defaults: page 0, per_page 20). +2. **Permission check** — If a `channel_id` filter is provided, check that the user can read that channel. If no filter is provided, the store query must be restricted to channels the user can access (use the cached channel set from step 009). +3. **Fetch data** — Query the store for extractions matching the filters, paginated. +4. **Return response** — Return the list of extractions with: ID, type, content, metadata, post ID, channel ID, edit status, creation timestamp. Include pagination metadata (total count, page, per_page). + +### Extraction edit endpoint (PUT /api/v1/extractions/{id}) + +Allows users to edit an extraction's content: + +1. **Extract path parameter** — Parse the extraction `id` from the path. +2. **Parse request** — Read the JSON body containing the updated content text. +3. **Fetch and validate** — Load the extraction from the store. If not found, return HTTP 404. Check that the user can read the extraction's channel (permission check). +4. **Update** — Update the extraction's content text, set the `edited` flag to true, record the editor's user ID, and update the timestamp. +5. **Trigger re-embedding** — Create a processing log entry with job type "embed" for the extraction's post to regenerate embeddings with the updated content. The processing worker (step 007) will handle the actual re-embedding. +6. **Return response** — Return the updated extraction. + +### Admin backfill endpoint (POST /api/v1/admin/backfill) + +Triggers, pauses, or resumes historical data backfill (admin only). This single endpoint handles all backfill lifecycle actions via an `action` field: + +1. **Admin guard** — Check `HasPermissionTo(userId, PermissionManageSystem)`. If the user is not a system admin, return HTTP 403. +2. **Parse request** — Read the JSON body containing: an `action` field (required, one of "start", "pause", "resume") and channel ID (required). For the "start" action only: since date (optional, as Unix millisecond timestamp — defaults to all history) and batch size (optional, defaults to 100). +3. **Action: start** — Verify the channel exists. Check that no backfill with status "running" exists for this channel (query the KV store for backfill progress). If one is running, return HTTP 409 Conflict. Otherwise, schedule the backfill job via the `JobOnceScheduler` (step 006) with the `backfill:` key prefix. Store the backfill parameters as job props. Return HTTP 202 Accepted with the backfill job ID for progress tracking. +4. **Action: pause** — Check that a running backfill exists for the channel in the KV store. If not found or not in "running" status, return HTTP 404. Set the backfill progress status to "paused" in the KV store. Cancel the pending scheduled job via `JobOnceScheduler.Cancel(backfill:{channelId})`. Return HTTP 200 with the current progress state (cursor, counters) so the admin knows where it stopped. +5. **Action: resume** — Check that a paused backfill exists for the channel in the KV store. If not found or not in "paused" status, return HTTP 404. Set the backfill progress status back to "running". Schedule the next batch from the existing cursor position via `ScheduleOnce` with key `backfill:{channelId}`. Return HTTP 202 Accepted with the resumed backfill state. + +### Admin status endpoint (GET /api/v1/admin/status) + +Returns pipeline health and operational status (admin only): + +1. **Admin guard** — Same system admin permission check as backfill. +2. **Gather status** — Query the store and in-memory metrics for: + - Processing queue depth (count of pending items in `ai_processing_log`) + - Error count (count of items with "error" status) + - Worker health (whether the processing worker ran recently — check last completion timestamp) + - Circuit breaker state (from KV store — closed/open/half-open) + - Active backfill progress (if any) + - Total knowledge base size (count of embeddings, extractions, summaries) +3. **Return response** — Return the status data as JSON. + +### Webhook create endpoint (POST /api/v1/webhooks) + +Creates a new outbound webhook configuration (admin only): + +1. **Admin guard** — System admin permission check. +2. **Parse request** — Read the JSON body containing: channel ID (required), target URL (required, must be a valid URL), extraction types to send (required, array of "decision" and/or "action_item"), and an optional secret for HMAC signing. +3. **Validate** — Verify the channel exists. Validate the URL format. +4. **Store** — Generate a webhook ID (via `model.NewId()`), store the configuration in the KV store with key `webhook:{channelId}:{webhookId}`. +5. **Return response** — Return the created webhook configuration with its ID. + +### Webhook list endpoint (GET /api/v1/webhooks) + +Lists outbound webhook configurations (admin only): + +1. **Admin guard** — System admin permission check. +2. **Fetch** — Use `KVList` with the `webhook:` prefix to enumerate all webhook configurations. Deserialize each value. +3. **Return response** — Return the list of webhook configurations. + +### Webhook delete endpoint (DELETE /api/v1/webhooks/{id}) + +Deletes an outbound webhook configuration (admin only): + +1. **Admin guard** — System admin permission check. +2. **Extract path parameter** — Parse the webhook `id` from the path. +3. **Find and delete** — Locate the webhook by scanning the KV store with the `webhook:` prefix for a matching ID. Delete the KV entry. If not found, return HTTP 404. +4. **Return response** — Return HTTP 204 No Content on success. + +### Slash command handler + +Implement the `ExecuteCommand` hook for the `/ask` command: + +1. **Parse the query** — Extract the query text from `args.Command` by stripping the `/ask ` trigger prefix. If the remaining text is empty, return an ephemeral response with usage help: "Usage: /ask ". + +2. **Call RAG engine** — Invoke the RAG engine's Query operation with `args.UserId` and the extracted query text. + +3. **Format response** — Build a markdown response containing the synthesized answer with citation links. Citations should be formatted as Mattermost markdown links (`[N](permalink)`) so they are clickable in the client. + +4. **Return ephemeral response** — Return a `CommandResponse` with `ResponseType` set to `CommandResponseTypeEphemeral` and `Text` containing the formatted answer. The response is visible only to the user who invoked the command. + +5. **Error handling** — If the RAG engine returns an error, return an ephemeral response with a user-friendly message ("Knowledge base is temporarily unavailable. Please try again later."). Do not expose internal error details. + +6. **Context from channel** — The slash command inherits the channel context from where it was invoked (`args.ChannelId`). This can be included in the query metadata for relevance weighting in future enhancements, but for now it is logged but not used to bias results. + +### Command registration (extending step 002) + +The `/ask` command was registered during OnActivate in step 002 with basic fields. This step specifies the complete registration: + +1. **Trigger** — `ask` +2. **AutoComplete** — enabled (`true`) +3. **AutoCompleteDesc** — "Ask your workspace knowledge base a question" +4. **AutoCompleteHint** — "[question]" +5. **DisplayName** — "Ask Workspace" +6. **Description** — "Query the AI knowledge base for answers from your workspace conversations" + +## Acceptance Criteria + +- [x] ServeHTTP routes requests to the correct handler based on path and HTTP method +- [x] All endpoints reject unauthenticated requests (missing Mattermost-User-Id header) with HTTP 401 +- [x] Unmatched paths return HTTP 404 with a JSON error body +- [x] Mismatched HTTP methods on known paths return HTTP 405 +- [x] POST and PUT endpoints enforce Content-Type: application/json and reject other types with HTTP 415 +- [x] Request bodies are size-limited to 1MB to prevent memory exhaustion +- [x] All responses use Content-Type: application/json +- [x] Error responses include a JSON body with error message and status code +- [x] POST /api/v1/query calls the RAG engine and returns synthesized answer, citations, and session ID +- [x] POST /api/v1/query/followup passes session ID to the RAG engine for multi-turn context +- [x] GET /api/v1/threads/{root_id}/summary checks channel read permission before returning thread data +- [x] GET /api/v1/channels/{channel_id}/summary checks channel read permission and supports daily/weekly type filter +- [x] GET /api/v1/entities/{name}/timeline returns permission-filtered entity mentions across channels +- [x] GET /api/v1/extractions supports channel_id and type query parameter filters with pagination +- [x] PUT /api/v1/extractions/{id} updates content, sets edited flag and editor user ID, and triggers re-embedding +- [x] POST /api/v1/admin/backfill requires system admin permission and returns HTTP 403 for non-admins +- [x] POST /api/v1/admin/backfill with action "start" rejects with HTTP 409 when a backfill is already running for the channel +- [x] POST /api/v1/admin/backfill with action "pause" sets status to "paused" and cancels the pending scheduler job +- [x] POST /api/v1/admin/backfill with action "resume" resumes a paused backfill from its saved cursor position +- [x] POST /api/v1/admin/backfill with action "pause" or "resume" returns HTTP 404 when no matching backfill exists +- [x] GET /api/v1/admin/status requires system admin permission and returns queue depth, error count, circuit breaker state, and knowledge base size +- [x] POST /api/v1/webhooks requires system admin permission and stores webhook config in KV store +- [x] GET /api/v1/webhooks requires system admin permission and lists all webhook configurations +- [x] DELETE /api/v1/webhooks/{id} requires system admin permission and returns HTTP 204 on success +- [x] The /ask slash command extracts query text by stripping the trigger prefix +- [x] The /ask command with empty text returns usage help as an ephemeral response +- [x] The /ask command calls the RAG engine and returns a formatted ephemeral response with citation links +- [x] The /ask command returns a user-friendly error message when the RAG engine fails +- [x] The /ask command is registered with autocomplete enabled, description, and hint text diff --git a/tasks/0001-org-memory-ai-knowledge-base/011.md b/tasks/0001-org-memory-ai-knowledge-base/011.md new file mode 100644 index 000000000000..3eb982fb099a --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/011.md @@ -0,0 +1,129 @@ +# 011: Channel summary generation worker + +## Context + +Thread summaries (step 008) capture the arc of individual conversations, but users also need a higher-level view: "What happened in #engineering yesterday?" or "What were the key decisions in #product this week?" Channel summaries aggregate thread summaries and extractions from a time window into a coherent narrative — a daily digest or weekly recap of channel activity. Unlike thread summaries which are event-driven (triggered by thread inactivity), channel summaries are clock-driven: they run on a daily and weekly schedule aligned to a configurable hour (default 8 AM), so the summaries are ready before the workday starts. + +This step builds the recurring background worker that generates these summaries. It uses `cluster.Schedule` with a custom wait function that targets the configured `DailySummaryHour` (from step 002), iterates over channels with activity in the relevant time window, gathers existing thread summaries and extractions as source material, sends them to Gemini for synthesis, and stores the results in `ai_channel_summaries`. Weekly summaries aggregate from daily summaries rather than raw posts, creating a hierarchical summarization structure that bounds Gemini token usage. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/pluginapi/cluster/job.go` (lines 60-89) — `MakeWaitForRoundedInterval(interval)` creates a scheduling function that aligns execution to clock-rounded boundaries. For a 24-hour interval, this means the job fires at approximately the same time each day. On first run (`LastFinished.IsZero()`), it fires immediately. Subsequent runs are scheduled to the next rounded interval after the last completion time. +- `server/public/pluginapi/cluster/job.go` (lines 36-58) — `MakeWaitForInterval(interval)` for the weekly summary variant, which runs every 7 days relative to last completion. Alternatively, `MakeWaitForRoundedInterval` with a 7-day interval aligns to weekly boundaries. +- `server/public/plugin/api.go` (line 728) — `GetPostsSince(channelId string, time int64) (*model.PostList, *model.AppError)` returns all posts created after a given Unix millisecond timestamp in a channel. Used to detect whether a channel had activity in the summary window. +- `server/channels/app/recap.go` — The existing recap feature processes channels iteratively: fetches posts via `GetPostsSince`, enriches with usernames, calls LLM for summarization, tracks progress, handles partial failures (some channels succeed, others fail). This iterative-with-partial-failure pattern is directly applicable. +- `server/channels/app/summarization.go` — `SummarizePosts` sends conversation text to an LLM and receives structured highlights and action items. The channel summary worker follows a similar pattern but operates on thread summaries and extractions rather than raw posts. +- `server/channels/jobs/recap/worker.go` — The recap worker iterates over channels, tracks progress as a percentage, publishes WebSocket events on status changes, and handles the case where some channels fail without aborting the entire batch. The plugin's channel summary worker should follow this resilience pattern. +- `server/public/model/utils.go` — `GetMillis()` returns the current Unix millisecond timestamp. `GetMillisForTime(t)` converts a `time.Time` to milliseconds. `GetTimeForMillis(ms)` converts back. Used for computing time window boundaries. +- `server/channels/jobs/base_schedulers.go` — `GenerateNextStartDateTime(now, nextStartTime)` computes the next daily execution time given a target hour and minute. Shows the pattern for handling "should run today or tomorrow?" logic based on the current time. + +## Reuse Opportunities + +- **Custom wait function for daily, MakeWaitForRoundedInterval for weekly** — The daily worker needs a custom `nextWaitInterval` that targets the configured `DailySummaryHour`, following the `GenerateNextStartDateTime` pattern. The weekly worker can use `MakeWaitForRoundedInterval(168 * time.Hour)` since the exact hour matters less for weekly cadence. Two separate `cluster.Schedule` jobs are needed — one for daily, one for weekly. +- **Recap channel iteration pattern** from `recap/worker.go` — iterate over a list of channels, process each one independently, track successes and failures, continue even if individual channels fail. This prevents a problem in one channel from blocking summaries for all other channels. +- **GetPostsSince as activity detector** — before attempting to summarize a channel, call `GetPostsSince` with the window start time. If the result is empty, skip the channel (no activity to summarize). This avoids wasteful Gemini calls for quiet channels. +- **Thread summaries as input** rather than raw posts. The architecture specifies that channel summaries are synthesized from thread summaries and extractions within the window, not from individual post messages. This reduces Gemini token usage dramatically — a channel with 50 threads and 500 posts might have 50 thread summaries of ~100 words each (5,000 words) versus 500 raw messages of ~50 words each (25,000 words). +- **WebSocket notification** via `PublishWebSocketEvent` after a channel summary is stored, so the webapp can offer to display the new summary. Follow the same pattern as `publishRecapUpdate` in the recap worker. + +## Deliverables + +### Daily summary worker + +Create a recurring background worker using `cluster.Schedule` that generates daily channel summaries, aligned to the configured `DailySummaryHour` (from step 002, default 8 = 8:00 AM server time): + +1. **Schedule the worker** — During OnActivate (extending step 002), schedule the daily summary job with a custom `nextWaitInterval` function. Instead of `MakeWaitForRoundedInterval(24h)` (which aligns to epoch boundaries, not a specific hour), implement a custom wait function that calculates the duration until the next occurrence of the configured `DailySummaryHour`. On each invocation, the function computes "today at DailySummaryHour" in the server's local timezone — if that time is in the past, it returns the duration until tomorrow at that hour; if it's in the future, it returns the duration until today at that hour. On first run (`LastFinished.IsZero()`), the function should fire immediately. This follows the `GenerateNextStartDateTime` pattern from `server/channels/jobs/base_schedulers.go`. Store the job handle on the Plugin struct for shutdown. + +2. **Determine the summary window** — Calculate the time window for the daily summary: from 24 hours before the current time to the current time. Express both boundaries as Unix millisecond timestamps. These boundaries define the "yesterday" period being summarized. + +3. **Identify active channels** — Enumerate all channels across all enabled teams. For each channel, check whether it had activity in the summary window. There are two approaches, and the implementer should choose based on performance: + - **Option A (query-based)**: For each channel in the enabled teams, call `GetPostsSince(channelId, windowStart)` and check if any posts were returned. This is simple but requires one API call per channel. + - **Option B (store-based)**: Query the `ai_thread_summaries` table for summaries with `LastActivityAt` within the window, group by channel ID. This leverages the plugin's own data and avoids per-channel API calls. Channels with thread summaries in the window definitely had activity. + + Either approach should skip channels that had no activity — no summary should be generated for quiet channels. + +4. **Process each active channel** — For each channel with activity, generate a daily summary: + + a. **Gather source material** — Query the store for all thread summaries (`ai_thread_summaries`) and extractions (`ai_extractions`) associated with the channel whose timestamps fall within the summary window. Assemble these into a structured text block with each thread summary and its key extractions. + + b. **Fetch channel metadata** — Get the channel name and team name via the plugin API for inclusion in the Gemini prompt and for display purposes. + + c. **Call Gemini** — Send the assembled source material to the Gemini client's synthesis operation (step 004) with a prompt instructing it to: summarize the key activities, decisions, and outcomes for the channel during the time window; organize by topic or theme; cite specific threads where major decisions were made. The prompt should identify this as a daily summary. + + d. **Store the result** — Insert a row in `ai_channel_summaries` (step 003) with: a new ID, the channel ID, type "daily", the summary text, the window start and end timestamps, and the creation timestamp. The unique constraint on channel + type + window start prevents duplicate summaries if the worker runs twice for the same window. + + e. **Embed the summary** — Create a processing log entry with job type "embed" and source type "summary" for the channel summary text, so the processing worker (step 007) generates embeddings for it. This makes channel summaries searchable via the RAG engine. + +5. **Handle channel failures gracefully** — If generating a summary for one channel fails (Gemini error, database error), log the error and continue to the next channel. Do not abort the entire batch. Track the count of successes and failures and log a summary at the end. + +6. **Circuit breaker awareness** — Before processing the batch, check the Gemini circuit breaker state. If the circuit breaker is open, skip the entire run and return — summaries will be generated on the next cycle when the API recovers. If the circuit breaker opens mid-batch (a channel's Gemini call fails and triggers the breaker), stop processing remaining channels and return. The remaining channels will be processed on the next run. + +### Weekly summary worker + +Create a second recurring background worker using `cluster.Schedule` with `MakeWaitForRoundedInterval(168 * time.Hour)` (7 days) for weekly channel summaries: + +1. **Schedule the worker** — During OnActivate, schedule the weekly summary job separately from the daily worker. Store its job handle for shutdown. + +2. **Determine the summary window** — The weekly window spans from 7 days before the current time to the current time. + +3. **Aggregate from daily summaries** — Instead of gathering thread summaries and extractions (which the daily summaries already cover), the weekly worker gathers daily summaries for each channel within the window. This creates a hierarchical summarization: raw posts → thread summaries → daily summaries → weekly summary. Each layer reduces content volume by an order of magnitude. + +4. **Process each channel** — For each channel that has at least one daily summary in the weekly window: + + a. **Gather daily summaries** — Query `ai_channel_summaries` for rows matching the channel, type "daily", and window start timestamps within the weekly window. Order by window start ascending. + + b. **Call Gemini** — Send the assembled daily summaries to the Gemini client with a prompt that requests a weekly synthesis: high-level themes, major decisions, key outcomes across the week, and significant trends or patterns. The prompt should identify this as a weekly summary. + + c. **Store the result** — Insert a row in `ai_channel_summaries` with type "weekly", the summary text, and the weekly window boundaries. The unique constraint prevents duplicates. + + d. **Embed the summary** — Same as daily — create a processing log entry for embedding. + +5. **Same failure handling** — Per-channel failure isolation and circuit breaker awareness, identical to the daily worker. + +### Time window edge cases + +The summary workers must handle several time-related edge cases: + +1. **First run after activation** — Both workers fire immediately on first activation (since `LastFinished` is zero). The daily worker should generate a summary for "the last 24 hours" relative to the current time, even if no previous summary exists. The weekly worker should generate for "the last 7 days." + +2. **Plugin downtime** — If the plugin was deactivated for 3 days and reactivated, the daily worker fires immediately and generates a summary for the last 24 hours. The intervening 2 days are not automatically backfilled — they would need the admin backfill endpoint (step 017) if coverage is desired. + +3. **Duplicate prevention** — The unique constraint on (channel_id, type, window_start) in `ai_channel_summaries` prevents duplicate summaries. If the worker runs twice for the same window (e.g., during a cluster failover), the second insert is a no-op. The worker should handle the duplicate constraint gracefully (not treat it as an error). + +4. **Time zone considerations** — The window boundaries are calculated in UTC. The configured summary hour (e.g., 8 AM) is in the server's local time zone, as determined by the custom wait function's use of `time.Now().Local()`. This means the "daily summary at 8 AM" runs at 8 AM server time, which may not align with all users' time zones. This is an acceptable limitation documented for administrators. + +### WebSocket notification + +After storing a channel summary, publish a WebSocket event to notify connected clients: + +1. **Event name** — Pass `channel_summary_ready` as the event name to `PublishWebSocketEvent`. The plugin API automatically prepends `custom_org-memory_` (the `custom_{pluginid}_` prefix), so the full wire-format event name received by clients is `custom_org-memory_channel_summary_ready`. +2. **Payload** — Include the channel ID, summary type (daily/weekly), and the window's end timestamp. +3. **Broadcast** — Broadcast to the channel so all members can see the new summary indicator. + +### Shutdown + +During OnDeactivate (extending step 002), close both the daily and weekly worker job handles to stop the recurring schedules. Let any in-progress channel summarization complete before stopping. + +## Acceptance Criteria + +- [x] The daily summary worker is scheduled via `cluster.Schedule` with a custom wait function that targets the configured `DailySummaryHour` (default 8 AM server time), and starts during OnActivate +- [x] The weekly summary worker is scheduled via `cluster.Schedule` with `MakeWaitForRoundedInterval(168 * time.Hour)` and starts during OnActivate +- [x] Both workers stop gracefully during OnDeactivate +- [x] The daily worker calculates a 24-hour window (current time minus 24 hours to current time) for each run +- [x] The weekly worker calculates a 7-day window for each run +- [x] Channels with no activity in the summary window are skipped without generating a summary +- [x] Daily summaries are synthesized from thread summaries and extractions within the window, not from raw posts +- [x] Weekly summaries are synthesized from daily summaries within the window, creating a hierarchical summarization +- [x] Each summary is sent to the Gemini client with a prompt that instructs topic-organized synthesis with thread citations +- [x] Summary results are stored in `ai_channel_summaries` with the correct type (daily/weekly) and window timestamps +- [x] The unique constraint on (channel_id, type, window_start) prevents duplicate summaries, and the worker handles duplicates gracefully +- [x] A processing log entry with source type "summary" is created for each channel summary so it gets embedded by the processing worker +- [x] Channel-level failures do not abort the entire batch — the worker continues to the next channel +- [x] Success and failure counts are logged at the end of each batch run +- [x] The worker skips the entire run when the Gemini circuit breaker is open +- [x] The worker stops processing remaining channels if the circuit breaker opens mid-batch +- [x] The first run after plugin activation fires immediately and generates summaries for the most recent window +- [x] A WebSocket event (`custom_org-memory_channel_summary_ready`) is published after each successful summary storage +- [x] The WebSocket event is broadcast to the channel and includes the channel ID, summary type, and window end timestamp +- [x] Both workers use the cluster scheduling to ensure only one cluster node generates summaries at a time diff --git a/tasks/0001-org-memory-ai-knowledge-base/012.md b/tasks/0001-org-memory-ai-knowledge-base/012.md new file mode 100644 index 000000000000..30a303c65f09 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/012.md @@ -0,0 +1,139 @@ +# 012: Outbound webhook dispatcher + +## Context + +The knowledge base extracts structured information — decisions and action items — from conversations. This information is valuable beyond the Mattermost workspace: project management tools, documentation systems, notification platforms, and custom integrations all benefit from receiving real-time signals about team decisions and commitments. The outbound webhook dispatcher bridges this gap by sending HTTP POST notifications to configured external URLs whenever new decisions or action items are extracted. + +This is a lightweight integration layer, not a message queue. Deliveries are best-effort with limited retries (3 attempts with exponential backoff). There is no persistent retry queue — if all retries fail, the delivery is logged and discarded. This design keeps the dispatcher simple and prevents failed webhooks from accumulating unbounded state or blocking the processing pipeline. Webhook configurations are stored in the KV store (not the database) because they are admin-managed, low-volume, and benefit from the KV store's simplicity over a separate SQL table. + +The dispatcher is triggered by the processing worker (step 007) and debounce handler (step 008) after new extractions are stored. It does not run on its own schedule — it executes synchronously (but quickly) as the final step in extraction processing. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/channels/app/webhook.go` (lines 99-227) — Mattermost's outgoing webhook implementation. `TriggerWebhook` dispatches to callback URLs concurrently via goroutines (one per URL) and waits for all to complete. `doOutgoingWebhookRequest` constructs the HTTP POST with `context.WithTimeout`, sets Content-Type and Accept headers, and reads the response with a 1MB size limit. The timeout is configurable via `ServiceSettings.OutgoingIntegrationRequestsTimeout` (default 30 seconds). This is the closest reference pattern for the plugin's own outbound HTTP calls. +- `server/public/shared/httpservice/httpservice.go` — The shared HTTP service that creates outbound HTTP clients with security features: reserved IP filtering (SSRF protection), proxy support, configurable timeouts (connect timeout 3s, request timeout 30s), and TLS configuration. Plugins do not have direct access to this service, but the timeout and transport configuration patterns should be followed when creating the plugin's own HTTP client. +- `server/public/model/outgoing_webhook.go` — The `OutgoingWebhook` model with `CallbackURLs`, `Token`, `TriggerWords`, `ContentType`. The `OutgoingWebhookPayload` struct defines the standard payload format. While the plugin's webhook payload differs (extraction data, not post data), the model structure and naming conventions are worth referencing. +- `server/public/pluginapi/kv.go` — KV store operations for webhook configuration storage: `Set(key, value)` for storing configs, `Get(key, &dest)` for retrieving, `Delete(key)` for removal, `ListKeys(page, perPage, WithPrefix(prefix))` for enumerating all configs by prefix. The `webhook:{channel_id}:{webhook_id}` key pattern from the architecture spec. +- `server/channels/utils/backoff.go` — `ProgressiveRetry` with predefined backoff timeout arrays. The plugin's webhook retry uses a simpler fixed sequence (1s, 2s, 4s) but the general backoff pattern is consistent with this utility. +- `server/public/model/utils.go` — `model.NewId()` for generating webhook configuration IDs. `model.GetMillis()` for timestamps in webhook payloads. +- `server/channels/app/integration_action.go` (line 255) — Response body size limiting via `io.LimitReader(resp.Body, MaxIntegrationResponseSize)`. The webhook dispatcher should apply the same limit to prevent external services from consuming memory with oversized responses. + +## Reuse Opportunities + +- **HTTP client timeout pattern** from `doOutgoingWebhookRequest` — use `context.WithTimeout` with a 30-second timeout for each webhook delivery attempt. This prevents slow or hanging external services from blocking the processing pipeline. +- **Concurrent dispatch with WaitGroup** from `TriggerWebhook` — when multiple webhooks match a channel, dispatch to each concurrently and wait for all to complete. This keeps the total dispatch latency bounded by the slowest webhook rather than the sum of all webhooks. +- **KV store prefix-based enumeration** — use `ListKeys` with the `webhook:` prefix to find all webhook configurations, then filter by channel ID. For the common case of checking webhooks for a specific channel, use the more specific prefix `webhook:{channel_id}:` to narrow the search. +- **model.NewId()** for webhook configuration IDs, consistent with all Mattermost entity IDs. +- **Response size limiting** from the integration action handler — read at most 1MB from the webhook response body to prevent memory exhaustion from malicious or misconfigured external services. + +## Deliverables + +### Webhook configuration type + +Define a type representing a webhook configuration with these fields: + +- **ID** — Unique 26-character identifier (generated via `model.NewId()`) +- **ChannelId** — The channel this webhook monitors for new extractions +- **URL** — The target URL to send HTTP POST requests to (must be a valid HTTPS or HTTP URL) +- **ExtractionTypes** — A list of extraction types that trigger this webhook (e.g., "decision", "action_item"). Only extractions matching one of these types cause a delivery. +- **Secret** — An optional shared secret used to compute an HMAC-SHA256 signature for payload verification. If empty, no signature is included. +- **CreatedAt** — Unix millisecond timestamp of creation +- **CreatedBy** — User ID of the admin who created the webhook + +### Webhook configuration CRUD + +The CRUD operations for webhook configurations are called by the HTTP API handlers (step 010). This step implements the storage layer: + +1. **Create** — Accept a webhook configuration (without ID and timestamps), generate an ID via `model.NewId()`, set the creation timestamp, and store in the KV store with key `webhook:{channelId}:{webhookId}`. Return the complete configuration with the generated ID. + +2. **List** — Enumerate all webhook configurations by scanning KV store keys with the `webhook:` prefix. Deserialize each value into a webhook configuration. Return the full list. For efficiency, paginate the `ListKeys` call (page size 100) and accumulate results. + +3. **Get by ID** — Given a webhook ID, scan the KV store for a matching key. Since the key includes both channel ID and webhook ID, and the caller may not know the channel ID, scan with the `webhook:` prefix and match the ID suffix. Alternatively, maintain a secondary index in a single KV entry that maps webhook IDs to their full keys. + +4. **Delete** — Given a webhook ID, locate the full key (same scan as Get), then delete the KV entry. Return success even if the webhook was already deleted (idempotent). + +5. **Find by channel** — Given a channel ID, enumerate KV store keys with prefix `webhook:{channelId}:` and deserialize all matching configurations. This is the primary lookup used by the dispatcher when checking whether to fire webhooks after an extraction. + +### Dispatch function + +Implement the core dispatch function that is called by the processing worker (step 007) and debounce handler (step 008) after new extractions are stored: + +1. **Input** — Accept a list of newly stored extraction items, each with: extraction type, content text, post ID, channel ID, and metadata. + +2. **Find matching webhooks** — For each unique channel ID in the extraction list, call the "find by channel" operation to get all webhook configurations for that channel. If no webhooks are configured for the channel, return immediately. + +3. **Filter by extraction type** — For each webhook, filter the extraction list to only include types that the webhook is configured to receive. If no extractions match the webhook's type filter, skip that webhook. + +4. **Build payload** — For each matching webhook, construct the outbound payload containing: + - An array of extraction items, each with: type, content text, source post ID, source post permalink (constructed as `{SiteURL}/pl/{postId}`), channel ID, and timestamp + - A metadata block with: the plugin ID, the event type ("extraction_created"), and the delivery timestamp + +5. **Dispatch with retries** — For each webhook, send the payload as an HTTP POST to the configured URL. The dispatch includes retry logic: + - Serialize the payload as JSON + - Set `Content-Type: application/json` + - If the webhook has a secret configured, compute an HMAC-SHA256 signature of the raw JSON payload bytes using the secret as the key, and include it in a `X-Org-Memory-Signature` header as a hex-encoded string + - Send the request with a 30-second timeout + - If the request fails (network error, timeout, or non-2xx response status), retry with exponential backoff: wait 1 second, retry; wait 2 seconds, retry; wait 4 seconds, retry + - After 3 failed retries (4 total attempts including the original), log the failure as a warning and discard the delivery + +6. **Concurrency** — If multiple webhooks match for a batch of extractions, dispatch to each webhook concurrently (separate goroutines) and wait for all to complete before returning. This keeps the total dispatch latency bounded. + +7. **Non-blocking behavior** — The dispatcher must not slow down the processing pipeline. Individual webhook timeouts (30 seconds) and retries (up to ~7 seconds of backoff) are acceptable because the dispatcher runs after extraction storage is already complete — the data is safely persisted regardless of webhook delivery success. + +### HMAC signature verification + +When a webhook configuration includes a secret, the dispatcher computes an HMAC-SHA256 signature to allow the receiver to verify the payload's authenticity and integrity: + +1. **Computation** — Take the raw JSON payload bytes (before any transport encoding), use the webhook's secret as the HMAC key, compute the SHA256 digest, and hex-encode the result. + +2. **Header** — Include the signature in the `X-Org-Memory-Signature` request header. The format is the hex-encoded HMAC-SHA256 digest. + +3. **Verification by receiver** — The external service computes the same HMAC-SHA256 over the received request body using its copy of the shared secret and compares it to the header value. If they match, the payload is authentic and unmodified. This is a widely-used pattern (GitHub, Stripe, Slack all use similar approaches). + +4. **No signature when no secret** — If the webhook's secret field is empty, the signature header is omitted entirely. + +### Integration with processing pipeline + +The dispatcher is called at two points in the processing pipeline: + +1. **From the processing worker (step 007)** — After the extraction job handler stores new extractions for a post, it calls the dispatcher with the new extraction items. This happens within the worker's per-item processing, after the extraction store writes but before the item is marked as "complete". If the dispatcher fails entirely (all webhooks fail), this does not affect the processing log status — extractions are stored regardless. + +2. **From the debounce handler (step 008)** — After thread-level summarization produces new extractions, the debounce handler calls the dispatcher with the thread-level extraction items. Same fire-and-forget semantics. + +The dispatcher should handle being called with an empty extraction list gracefully (return immediately without any KV lookups). + +### Failed delivery tracking + +Track webhook delivery failures for admin visibility via the pipeline status endpoint (step 010): + +1. **In-memory counter** — Maintain a counter of total failed deliveries (after all retries exhausted) per webhook ID. This counter is exposed by the metrics/observability layer (step 014) and the admin status endpoint. + +2. **Log-based audit** — Each failed delivery (after all retries) is logged at the warning level with: webhook ID, target URL (with any query parameters redacted), channel ID, number of extractions in the payload, and the last error message. This provides an audit trail in the server logs. + +3. **No persistent retry queue** — Failed deliveries are discarded after logging. There is no mechanism to re-deliver failed payloads. If an admin needs to replay deliveries, they can use the backfill endpoint to reprocess the relevant posts, which will regenerate extractions and re-trigger webhooks. + +## Acceptance Criteria + +- [x] Webhook configurations are stored in the KV store with key pattern `webhook:{channelId}:{webhookId}` +- [x] Webhook configurations include ID, channel ID, target URL, extraction type filter, optional HMAC secret, creation timestamp, and creator user ID +- [x] The create operation generates a unique ID and stores the configuration +- [x] The list operation enumerates all webhook configurations by scanning the KV store with the `webhook:` prefix +- [x] The delete operation removes a webhook configuration by ID (idempotent) +- [x] The find-by-channel operation returns all webhook configurations for a given channel ID +- [x] The dispatcher accepts a list of new extractions and sends HTTP POST requests to matching webhooks +- [x] Only webhooks whose extraction type filter includes the extraction's type receive the delivery +- [x] The webhook payload includes extraction type, content, source post permalink, channel ID, and timestamp +- [x] Payloads are serialized as JSON with Content-Type: application/json +- [x] When a webhook has a secret, an HMAC-SHA256 signature of the payload is included in the X-Org-Memory-Signature header +- [x] When a webhook has no secret, the signature header is omitted +- [x] Failed deliveries are retried up to 3 times with exponential backoff (1s, 2s, 4s waits between retries) +- [x] After all retries fail, the delivery is logged as a warning and discarded +- [x] Each delivery attempt uses a 30-second HTTP request timeout +- [x] Multiple webhooks for the same channel are dispatched concurrently +- [x] The dispatcher is called by the processing worker after storing new extractions +- [x] The dispatcher is called by the debounce handler after storing thread-level extractions +- [x] An empty extraction list causes the dispatcher to return immediately without KV lookups +- [x] Failed delivery counts are tracked for admin visibility via the status endpoint +- [x] Webhook delivery failures do not affect the processing log status — extractions are stored regardless of delivery success diff --git a/tasks/0001-org-memory-ai-knowledge-base/013.md b/tasks/0001-org-memory-ai-knowledge-base/013.md new file mode 100644 index 000000000000..594512f3dbe8 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/013.md @@ -0,0 +1,111 @@ +# 013: Data retention and lifecycle cleanup + +## Context + +Mattermost's data retention system periodically sweeps the database to delete posts older than a configured retention window. When a post is deleted by retention, all knowledge base artifacts derived from that post — embeddings, extractions, entity mentions, summaries — must be cleaned up too. Without this cleanup, the knowledge base would accumulate stale references to deleted content, waste storage, and potentially return RAG results that reference posts users can no longer access. + +The plugin participates in retention by implementing the `RunDataRetention` hook. When Mattermost's data retention job runs (typically scheduled for off-hours, e.g., 2 AM), it calls this hook on every active plugin with the current timestamp and a batch size. The plugin must delete its derived data in batches, returning the count of deleted rows. Mattermost calls the hook repeatedly until the returned count is zero, indicating all aged-out data has been cleaned. + +This step also handles cascade deletion from the `MessageHasBeenDeleted` hook (step 006) for individual post deletions, and provides a supplementary cleanup mechanism for orphaned data that might accumulate due to edge cases (e.g., plugin downtime during a retention sweep, or posts deleted directly in the database bypassing hooks). + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/hooks.go` (line 306) — `RunDataRetention(nowTime, batchSize int64) (int64, error)` hook signature. `nowTime` is the current timestamp in Unix milliseconds. `batchSize` is the maximum number of records to delete per invocation. Returns the count of deleted rows and any error. Hook ID: `RunDataRetentionID = 24`. Minimum server version: 6.4. +- `server/channels/app/plugin_hooks_test.go` (lines 1333-1376) — Test for the `RunDataRetention` hook. Shows the invocation pattern: `hooks.RunDataRetention(0, 0)` with the server checking the returned count. The server calls `RunMultiHook` which invokes the hook on each active plugin. +- `server/public/model/config.go` (lines 3260-3327) — `DataRetentionSettings` configuration: `MessageRetentionHours` (how long to keep messages), `BatchSize` (default 3000), `TimeBetweenBatchesMilliseconds` (default 100ms), `DeletionJobStartTime` (default "02:00"), `PreservePinnedPosts` (don't delete pinned posts). The `batchSize` parameter passed to the hook comes from this configuration. +- `server/channels/store/sqlstore/post_store.go` (lines 2593-2606) — `PermanentDeleteBatch(endTime int64, limit int64)` shows the core batch deletion pattern: `DELETE FROM table WHERE Id = any (array (SELECT Id FROM table WHERE CreateAt < ? LIMIT ?))`. This subquery-based batch deletion ensures a bounded number of rows are deleted per call, preventing long-running transactions. +- `server/channels/store/sqlstore/retention_policy_store.go` — `genericPermanentDeleteBatchForRetentionPolicies` implements multi-stage batch deletion with cursor tracking. Deletes rows matching retention policies in batches, returns the count and whether more data exists. Uses Squirrel query builder with `sq.Dollar` placeholders. +- `server/channels/db/migrations/postgres/000053_create_retention_policies.up.sql` — Shows `ON DELETE CASCADE` foreign key patterns for linked tables. When a parent row is deleted, child rows are automatically removed. The plugin's schema (step 003) uses similar cascades from `Posts` to plugin tables. +- `server/public/model/utils.go` — `GetMillis()` and `GetTimeForMillis(millis)` for timestamp operations. The `nowTime` parameter in the hook is a Unix millisecond timestamp comparable to `CreateAt` columns. + +## Reuse Opportunities + +- **Batch deletion with subquery pattern** from `PermanentDeleteBatch` — delete rows whose `CreateAt` is older than the retention cutoff, limited by `batchSize`. This prevents the deletion from locking the table for an extended period. The plugin should use the same `DELETE WHERE Id IN (SELECT Id WHERE condition LIMIT batch)` pattern. +- **ON DELETE CASCADE from Posts** — The plugin's schema (step 003) defines foreign keys from `ai_embeddings`, `ai_extractions`, and `ai_entity_mentions` to the `Posts` table with `ON DELETE CASCADE`. When Mattermost's core retention deletes a post, the cascade automatically removes related plugin rows. The `RunDataRetention` hook acts as a secondary cleanup for rows that might not have FK cascades (e.g., `ai_thread_summaries` keyed by root post ID, `ai_channel_summaries` which reference channels rather than individual posts, and `ai_processing_log` entries). +- **DataRetentionSettings.BatchSize** — The `batchSize` parameter from the hook call matches the admin-configured retention batch size (default 3000). The plugin should respect this value rather than defining its own batch size. +- **Squirrel query builder** — Use Squirrel with `sq.Dollar` placeholders for the batch deletion queries, consistent with all store operations from step 003. + +## Deliverables + +### RunDataRetention hook implementation + +Implement the `RunDataRetention` hook that the Mattermost server calls during data retention sweeps: + +1. **Determine the retention cutoff** — The `nowTime` parameter represents the current time. However, the plugin does not independently determine the retention window — it relies on the fact that Mattermost's core has already deleted (or is about to delete) posts older than the retention threshold. The plugin's job is to clean up derived data for posts that no longer exist in the core `Posts` table. The cutoff is therefore determined by checking which posts have been deleted. + +2. **Delete orphaned embeddings** — Query `ai_embeddings` for rows whose `PostId` no longer exists in the `Posts` table (the post was deleted by retention or manually). Delete up to `batchSize` such orphaned rows. If the ON DELETE CASCADE from the Posts FK handles this automatically, this query may return zero rows — which is correct and expected. + +3. **Delete orphaned extractions** — Same pattern for `ai_extractions`: find rows whose `PostId` no longer has a corresponding post, delete up to `batchSize` rows. + +4. **Delete orphaned entity mentions** — Same for `ai_entity_mentions`. + +5. **Delete orphaned thread summaries** — Query `ai_thread_summaries` for rows whose root post ID (stored as `PostId`) no longer exists in the `Posts` table. Thread summaries reference the root post, so when the root is deleted by retention, the thread summary should be removed. + +6. **Delete aged channel summaries** — Channel summaries have their own lifecycle: daily summaries older than a reasonable retention period (e.g., 90 days) and weekly summaries older than a longer period (e.g., 365 days) should be cleaned up even if the channel still exists. These are rolling summaries — older ones become less relevant as newer ones replace them. Delete up to `batchSize` channel summary rows where the window end timestamp is older than the retention threshold. + +7. **Delete completed processing log entries** — Processing log entries with "complete" or "error" status that are older than a retention period (e.g., 30 days) can be safely deleted. They serve no further purpose once the work is done. Pending and processing entries are preserved regardless of age. + +8. **Delete aged query log entries** — Query log entries older than a retention period (e.g., 90 days) can be cleaned up to prevent unbounded growth. The query log is an audit trail, but infinite retention is unnecessary for a plugin-level log. + +9. **Return the total count** — Sum the number of rows deleted across all tables and return the total. If the total is greater than zero, Mattermost will call the hook again to continue cleanup. If the total is zero, all aged-out data has been cleaned and the retention sweep moves on. + +10. **Respect the batch size** — The total number of rows deleted across all tables in a single invocation should not exceed `batchSize`. Distribute the batch budget across tables in a round-robin or proportional fashion. A simple approach: allocate `batchSize / number_of_tables` to each table's deletion query. If one table has fewer rows to delete, the remaining budget can be applied to the next table. + +### Cascade deletion coordination + +The plugin's cleanup strategy has two layers that work together: + +1. **Foreign key cascades (automatic)** — The schema (step 003) defines `ON DELETE CASCADE` from plugin tables to the `Posts` table. When Mattermost core deletes a post (via retention or `DeletePost`), PostgreSQL automatically cascades the deletion to `ai_embeddings`, `ai_extractions`, and `ai_entity_mentions` rows that reference that post. This is the primary cleanup mechanism and handles the vast majority of cases. + +2. **RunDataRetention hook (supplementary)** — The hook catches cases that cascades don't cover: + - `ai_thread_summaries` are keyed by the thread root post ID but may not have a direct FK to `Posts` (depending on schema design choices in step 003) + - `ai_channel_summaries` reference channels, not individual posts — they age out independently + - `ai_processing_log` entries are transient work items that should be cleaned after completion + - `ai_query_log` entries are audit records that should be cleaned after a retention period + - Orphaned rows from any edge case where the cascade didn't fire (e.g., posts deleted by direct SQL rather than through Mattermost's API) + +### MessageHasBeenDeleted coordination + +Step 006's `MessageHasBeenDeleted` hook performs immediate cleanup when an individual post is deleted (not via retention). The retention hook provides a complementary batch cleanup for bulk deletions. The two mechanisms should not conflict: + +1. **Idempotent deletions** — Both the per-post hook and the retention hook may attempt to delete the same rows. All delete operations must be idempotent — deleting a row that doesn't exist (already deleted by the other mechanism) should succeed silently. + +2. **No double-counting** — If the FK cascade already deleted `ai_embeddings` rows when a post was deleted, the retention hook's orphan check will find zero orphaned rows for that post — which is correct. + +### Channel summary retention thresholds + +Unlike post-derived data (which follows the core retention policy), channel summaries have their own retention schedule because they are aggregate artifacts, not directly tied to individual posts: + +1. **Daily summary retention** — Daily summaries older than 90 days are eligible for cleanup. This keeps approximately 3 months of daily digests available. + +2. **Weekly summary retention** — Weekly summaries older than 365 days are eligible for cleanup. This keeps approximately 1 year of weekly recaps. + +3. **These thresholds are compile-time constants**, not user-facing configuration. They represent reasonable defaults for a knowledge base plugin. Administrators who need different values can adjust them in a future configuration update. + +### Processing log and query log retention + +1. **Processing log** — Entries with "complete" or "error" status older than 30 days are deleted. Entries with "pending" or "processing" status are never deleted by retention regardless of age — they represent unfinished work. + +2. **Query log** — Entries older than 90 days are deleted. The query log grows with every user query and can become large on active workspaces. + +## Acceptance Criteria + +- [x] The `RunDataRetention(nowTime, batchSize int64) (int64, error)` hook is implemented +- [x] The hook deletes orphaned `ai_embeddings` rows whose source posts no longer exist, up to the batch size +- [x] The hook deletes orphaned `ai_extractions` rows whose source posts no longer exist +- [x] The hook deletes orphaned `ai_entity_mentions` rows whose source posts no longer exist +- [x] The hook deletes orphaned `ai_thread_summaries` rows whose root posts no longer exist +- [x] The hook deletes `ai_channel_summaries` daily entries older than 90 days +- [x] The hook deletes `ai_channel_summaries` weekly entries older than 365 days +- [x] The hook deletes `ai_processing_log` entries with "complete" or "error" status older than 30 days +- [x] The hook does not delete `ai_processing_log` entries with "pending" or "processing" status regardless of age +- [x] The hook deletes `ai_query_log` entries older than 90 days +- [x] The total number of deleted rows across all tables does not exceed `batchSize` per invocation +- [x] The hook returns the total count of deleted rows (zero means cleanup is complete) +- [x] The hook returns zero when there is no data to clean up +- [x] All delete operations are idempotent — rows already deleted by FK cascades or per-post hooks do not cause errors +- [x] The batch deletion uses the subquery pattern (DELETE WHERE Id IN SELECT Id WHERE condition LIMIT N) to prevent long-running transactions +- [x] FK cascade from Posts to ai_embeddings, ai_extractions, and ai_entity_mentions handles the primary cleanup automatically +- [x] The RunDataRetention hook provides supplementary cleanup for data not covered by FK cascades diff --git a/tasks/0001-org-memory-ai-knowledge-base/014.md b/tasks/0001-org-memory-ai-knowledge-base/014.md new file mode 100644 index 000000000000..10005cd5e176 --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/014.md @@ -0,0 +1,194 @@ +# 014: Metrics and observability + +## Context + +The plugin operates largely in the background — processing posts, generating embeddings, running extraction jobs, synthesizing summaries, answering queries. Without observability into this machinery, administrators have no way to tell whether the plugin is healthy, underperforming, or silently failing. Is the processing queue backing up? Is Gemini rate-limiting us? How many RAG queries are we serving per hour? Is the circuit breaker open? + +This step adds a Prometheus metrics endpoint to the plugin using the `ServeMetrics` hook, which Mattermost routes to `localhost:8067/plugins/{plugin_id}/metrics`. The plugin creates its own `prometheus.Registry`, registers counters, histograms, and gauges covering four domains (processing pipeline, Gemini API, RAG queries, embedding operations), and serves the standard Prometheus text exposition format. Prometheus scrapers (or Grafana Agent, Victoria Metrics, etc.) poll this endpoint at their configured interval and store the time series for dashboards and alerting. + +The metrics are collected at instrumentation points throughout the codebase — in the processing worker (step 007), Gemini client (step 004), RAG query engine (step 009), chunking engine (step 005), channel summary worker (step 011), webhook dispatcher (step 012), and event hooks (step 006). This step defines the metrics collector struct and registration; the instrumentation call sites were identified as "metrics integration points" in each of those steps. + +## Related Code + +Existing files/patterns to reference or extend: + +- `server/public/plugin/hooks.go` (line 366) — `ServeMetrics(c *Context, w http.ResponseWriter, r *http.Request)` hook signature. Hook ID: `ServeMetricsID = 39`. Minimum server version: 9.2. The server routes requests to `localhost:8067/plugins/{plugin_id}/metrics` to this hook. The URL path is stripped to just the portion after `/metrics` before calling the hook, so the hook receives relative paths (e.g., `/` or `/subpath`). +- `server/channels/app/platform/metrics.go` (lines 188-239) — The server-side routing for plugin metrics. The `pluginsMetricsRoute` handler at `/plugins/{plugin_id:[A-Za-z0-9\_\-\.]+}/metrics` strips the path prefix and calls `hooks.ServeMetrics` with an empty `plugin.Context`. The plugin receives a standard `http.ResponseWriter` and `*http.Request`, allowing it to use `promhttp.HandlerFor(registry, opts)` directly. +- `server/enterprise/metrics/metrics.go` (lines 53-254) — Mattermost's own `MetricsInterfaceImpl` struct shows the canonical pattern: a `prometheus.Registry` field, Counter/Histogram/HistogramVec/Gauge/GaugeFunc/GaugeVec fields for each metric, and `MustRegister` calls in the constructor. The struct uses namespace `"mattermost"` and subsystem constants like `MetricsSubsystemPlugin`. The plugin should use a distinct namespace (e.g., `"org_memory"`) to avoid collisions. +- `server/enterprise/metrics/metrics.go` (lines 304-311) — Counter registration pattern: `prometheus.NewCounter(prometheus.CounterOpts{Namespace, Subsystem, Name, Help, ConstLabels})` followed by `Registry.MustRegister(counter)`. Counters are monotonically increasing — they only go up and are reset on process restart. +- `server/enterprise/metrics/metrics.go` (lines 862-903) — HistogramVec registration pattern with labels: `prometheus.NewHistogramVec(prometheus.HistogramOpts{...}, []string{"label1", "label2"})`. The `PluginHookTimeHistogram` uses labels `plugin_id`, `hook_name`, `success`. HistogramVec allows a single metric definition to produce multiple label combinations. The `Observe(elapsed)` method records a duration sample. +- `server/enterprise/metrics/metrics.go` (lines 360-367) — GaugeFunc pattern: `prometheus.NewGaugeFunc(prometheus.GaugeOpts{...}, func() float64 { ... })`. The callback is called each time the metric is scraped, providing a live snapshot (e.g., current connection count). Useful for queue depth and circuit breaker state. +- `server/enterprise/metrics/metrics.go` (lines 276-281) — Registry initialization pattern: `prometheus.NewRegistry()` creates an isolated registry (not the global default), then `MustRegister(collectors.NewProcessCollector(...))` and `MustRegister(collectors.NewGoCollector())` add standard Go runtime metrics (goroutines, memory, GC). +- `server/channels/app/plugin_api_tests/manual.test_serve_metrics_plugin/main.go` — A minimal test plugin implementing `ServeMetrics`. Demonstrates the hook signature: receive `*plugin.Context`, `http.ResponseWriter`, `*http.Request`, and write the response directly. The plugin can route on `r.URL.Path` for sub-paths under the metrics endpoint. +- `server/channels/app/plugin_api_test.go` (lines 2983-3010) — `TestPluginServeMetrics` test. Shows the setup: enable metrics via `MetricsSettings.Enable = true`, set a custom `ListenAddress`, deploy the plugin, and make HTTP requests to `http://localhost:{port}/plugins/{plugin_id}/metrics`. Validates that the response body matches the expected content. +- `server/enterprise/metrics/metrics.go` (lines 1987-2001) — Metric observation methods: `ObservePluginHookDuration` calls `HistogramVec.With(prometheus.Labels{...}).Observe(elapsed)`. `ObservePluginAPIDuration` follows the same pattern. These show how observation methods wrap the raw Prometheus calls with a clean interface. + +## Reuse Opportunities + +- **prometheus.NewRegistry() + promhttp.HandlerFor()** — The plugin creates its own isolated registry (not the global `prometheus.DefaultRegisterer`) and uses `promhttp.HandlerFor(registry, promhttp.HandlerOpts{})` to create an HTTP handler that serves the registry's collected metrics in Prometheus text exposition format. This is the standard approach used by Mattermost's own metrics implementation and avoids conflicts with any global metrics from other plugins. +- **Namespace/Subsystem/Name hierarchy** from Mattermost's metrics. The plugin uses a unique namespace (`org_memory`) with subsystems for each domain (`processing`, `gemini`, `rag`, `embedding`). This produces metric names like `org_memory_processing_items_total` and `org_memory_gemini_request_duration_seconds` that are self-documenting and sortable. +- **HistogramVec with labels** for metrics that need dimensional breakdown. For example, processing latency by job type uses a `job_type` label, Gemini request duration uses an `operation` label. This follows the same pattern as `PluginHookTimeHistogram` which uses `plugin_id`, `hook_name`, and `success` labels. +- **GaugeFunc for live snapshots** — queue depth and circuit breaker state are best represented as GaugeFunc metrics whose callbacks query the current state at scrape time, rather than being updated on every state change. This avoids synchronization complexity and ensures the metric always reflects the latest state. +- **Process and Go collectors** — Register `collectors.NewProcessCollector` and `collectors.NewGoCollector` on the plugin's registry to expose standard Go runtime metrics (goroutine count, heap usage, GC statistics). These are useful for diagnosing plugin memory leaks or goroutine leaks without requiring custom instrumentation. + +## Deliverables + +### Metrics collector struct + +Define a struct that holds all the plugin's Prometheus metric instruments and a `prometheus.Registry`. This struct is created once during `OnActivate` (extending step 002) and passed to all components that need to record metrics: + +1. **Registry** — A `prometheus.Registry` instance created via `prometheus.NewRegistry()`. All metric instruments are registered on this registry. Standard process and Go collectors are also registered. + +2. **Namespace** — All metrics use the namespace `org_memory` to clearly identify them as belonging to this plugin and avoid collisions with Mattermost's own `mattermost` namespace or other plugins. + +3. **Processing pipeline metrics** — Instruments for the processing worker (step 007) and event hooks (step 006): + + - **Items processed counter** (`org_memory_processing_items_total`) — A `CounterVec` with a `job_type` label (`embed`, `extract`). Incremented each time a processing log item completes successfully. Tracks throughput by job type. + - **Items errored counter** (`org_memory_processing_errors_total`) — A `CounterVec` with a `job_type` label. Incremented each time a processing log item transitions to the error state (after all retries exhausted). Tracks the failure rate. + - **Processing latency histogram** (`org_memory_processing_duration_seconds`) — A `HistogramVec` with a `job_type` label. Observes the wall-clock time from when a processing item is picked up by the worker to when it completes (or errors). Bucket boundaries should cover the typical range: 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s, 30s, 60s. + - **Queue depth gauge** (`org_memory_processing_queue_depth`) — A `GaugeFunc` whose callback queries the count of processing log items with "pending" status. Provides a live view of how many items are waiting to be processed. If the queue depth grows consistently, the worker is not keeping up. + - **Stale items gauge** (`org_memory_processing_stale_items`) — A `GaugeFunc` whose callback queries the count of processing log items with "processing" status and a start time older than 5 minutes. Indicates items that are stuck and will be reset by the worker's stale detection. + - **Items ingested counter** (`org_memory_processing_items_ingested_total`) — A `Counter` incremented each time a new processing log entry is created by the event hooks (step 006). Tracks the ingest rate separate from the processing rate, showing how quickly work is arriving versus being consumed. + +4. **Gemini API metrics** — Instruments for the Gemini client (step 004): + + - **Request counter** (`org_memory_gemini_requests_total`) — A `CounterVec` with an `operation` label (`embed`, `extract`, `synthesize`) and a `status` label (`success`, `error`, `rate_limited`). Tracks the total number of Gemini API calls by operation and outcome. + - **Request duration histogram** (`org_memory_gemini_request_duration_seconds`) — A `HistogramVec` with an `operation` label. Observes the wall-clock time of each Gemini API call (including retries within the call). Bucket boundaries for API latency: 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s, 30s. + - **Rate limit hits counter** (`org_memory_gemini_rate_limit_hits_total`) — A `Counter` incremented each time a Gemini API call receives a rate limit response (HTTP 429 or resource exhausted error). Spikes indicate the plugin is exceeding quota and needs throttling. + - **Circuit breaker state gauge** (`org_memory_gemini_circuit_breaker_state`) — A `Gauge` that reflects the current circuit breaker state as a numeric value: 0 = closed (healthy), 1 = half-open (testing), 2 = open (failing). Updated whenever the circuit breaker state changes. This allows alerting on prolonged open states. + - **Tokens processed counter** (`org_memory_gemini_tokens_total`) — A `CounterVec` with an `operation` label and a `direction` label (`input`, `output`). If Gemini's response includes token usage metadata, track the total tokens consumed. This helps estimate API costs. If token counts are not available from the response, this metric is omitted. + +5. **RAG query metrics** — Instruments for the RAG query engine (step 009): + + - **Query counter** (`org_memory_rag_queries_total`) — A `CounterVec` with a `type` label (`new`, `followup`). Incremented on each query processed. Tracks overall usage of the knowledge base. + - **Query duration histogram** (`org_memory_rag_query_duration_seconds`) — A `Histogram` observing end-to-end query latency from receipt to response delivery. Bucket boundaries: 0.1s, 0.25s, 0.5s, 1s, 2.5s, 5s, 10s, 30s, 60s. + - **Search result count histogram** (`org_memory_rag_search_results`) — A `Histogram` observing the number of vector search results returned per query (before permission filtering). Bucket boundaries: 0, 1, 2, 5, 10, 20. Tracks how much relevant content exists for typical queries. + - **Permission cache hit/miss counter** (`org_memory_rag_permission_cache_total`) — A `CounterVec` with a `result` label (`hit`, `miss`). Tracks the effectiveness of the permission cache. A low hit rate suggests the 60-second TTL is too short or query patterns are highly distributed across users. + - **Session hit/miss counter** (`org_memory_rag_session_total`) — A `CounterVec` with a `result` label (`hit`, `miss`, `expired`). Tracks multi-turn session usage. A high miss/expired rate may indicate the 30-minute TTL is too short. + - **No results counter** (`org_memory_rag_no_results_total`) — A `Counter` incremented when a query returns zero vector search results and the canned "no results" response is returned. A high rate indicates content gaps in the knowledge base. + +6. **Embedding operation metrics** — Instruments for the chunking engine (step 005) and embedding storage: + + - **Embeddings generated counter** (`org_memory_embedding_generated_total`) — A `Counter` incremented each time an embedding vector is generated and stored. Tracks the rate of new content entering the vector index. + - **Chunks per post histogram** (`org_memory_embedding_chunks_per_post`) — A `Histogram` observing the number of chunks produced per post by the chunking engine. Bucket boundaries: 1, 2, 3, 5, 10, 15, 20, 32. Tracks the distribution of post complexity — most posts should produce 1-3 chunks; posts hitting the 32-chunk cap may indicate unusually long content. + - **Total embeddings gauge** (`org_memory_embedding_total_stored`) — A `GaugeFunc` whose callback queries the count of rows in `ai_embeddings` where `DeleteAt = 0`. Provides a live view of the knowledge base size. Useful for capacity planning. + +7. **Channel summary metrics** — Instruments for the channel summary worker (step 011): + + - **Summaries generated counter** (`org_memory_summary_generated_total`) — A `CounterVec` with a `type` label (`daily`, `weekly`). Incremented each time a channel summary is successfully generated and stored. + - **Channels skipped counter** (`org_memory_summary_channels_skipped_total`) — A `Counter` incremented for each channel skipped during a summary batch because it had no activity in the window. Tracks how many channels are quiet. + - **Summary batch duration histogram** (`org_memory_summary_batch_duration_seconds`) — A `Histogram` observing the wall-clock time for an entire daily or weekly summary batch (processing all channels). Tracks how long the summary generation takes end to end. + +8. **Webhook delivery metrics** — Instruments for the webhook dispatcher (step 012): + + - **Webhook deliveries counter** (`org_memory_webhook_deliveries_total`) — A `CounterVec` with a `status` label (`success`, `failed`). Incremented for each webhook delivery attempt's final outcome (after all retries). + - **Webhook delivery duration histogram** (`org_memory_webhook_delivery_duration_seconds`) — A `Histogram` observing the time for a single webhook delivery (including retries). Tracks whether external services are responding slowly. + +9. **Data retention metrics** — Instruments for the retention hook (step 013): + + - **Retention rows deleted counter** (`org_memory_retention_rows_deleted_total`) — A `CounterVec` with a `table` label identifying which plugin table the rows were deleted from. Tracks how much data retention is cleaning up per table. + +### ServeMetrics hook implementation + +Implement the `ServeMetrics` hook to serve the plugin's Prometheus metrics: + +1. **Create the HTTP handler** — During `OnActivate`, after registering all metrics on the registry, create a `promhttp.HandlerFor(registry, promhttp.HandlerOpts{})` handler. Store this handler on the Plugin struct for reuse across scrape requests. + +2. **Serve requests** — In the `ServeMetrics` hook method, delegate to the stored `promhttp.HandlerFor` handler. The handler writes the Prometheus text exposition format (`text/plain; version=0.0.4; charset=utf-8`) to the response writer. Each scrape request triggers the registry to collect current values from all registered metrics and format the output. + +3. **No authentication** — The metrics endpoint is served on the internal metrics listener (port 8067 by default), which is separate from the main API port. It does not go through Mattermost's session authentication. This is consistent with how Mattermost's own metrics work — the metrics port is typically only exposed within the internal network or cluster, not to the public internet. + +4. **Path routing** — The hook receives requests with paths relative to `/plugins/{plugin_id}/metrics`. For the standard Prometheus scrape at the root path (`/` or empty), serve the full metrics output. The plugin does not need to implement sub-paths — the root handler is sufficient. + +### Metrics collector initialization + +During `OnActivate` (extending step 002): + +1. **Create the collector** — Instantiate the metrics collector struct with a new `prometheus.Registry`. + +2. **Register standard collectors** — Register `collectors.NewProcessCollector` and `collectors.NewGoCollector` on the registry for Go runtime metrics. + +3. **Register all plugin metrics** — Call `MustRegister` for each metric instrument defined in the collector struct. This is done once during activation and the instruments are reused for the lifetime of the plugin. + +4. **Create the HTTP handler** — Build the `promhttp.HandlerFor` handler from the registry. + +5. **Wire to components** — Pass the metrics collector (or an interface wrapping it) to the processing worker, Gemini client, RAG engine, chunking engine, channel summary worker, webhook dispatcher, and event hooks. Each component calls the appropriate observation methods on the collector. + +6. **No-op fallback** — If metrics initialization fails (e.g., a metric name collision), log a warning and continue with a no-op metrics collector. The plugin should function without metrics rather than failing to activate. The metrics collector interface should have a no-op implementation that silently discards observations. + +### Metrics collector interface + +Define a Go interface for the metrics collector that all components depend on, rather than depending directly on the concrete struct: + +1. **Processing methods** — `IncProcessingItemsComplete(jobType string)`, `IncProcessingItemsError(jobType string)`, `ObserveProcessingDuration(jobType string, seconds float64)`, `IncItemsIngested()` + +2. **Gemini methods** — `IncGeminiRequest(operation, status string)`, `ObserveGeminiDuration(operation string, seconds float64)`, `IncGeminiRateLimitHit()`, `SetCircuitBreakerState(state int)` + +3. **RAG methods** — `IncRAGQuery(queryType string)`, `ObserveRAGDuration(seconds float64)`, `ObserveSearchResultCount(count int)`, `IncPermissionCacheResult(result string)`, `IncSessionResult(result string)`, `IncNoResults()` + +4. **Embedding methods** — `IncEmbeddingsGenerated()`, `ObserveChunksPerPost(count int)` + +5. **Summary methods** — `IncSummaryGenerated(summaryType string)`, `IncChannelsSkipped()`, `ObserveSummaryBatchDuration(seconds float64)` + +6. **Webhook methods** — `IncWebhookDelivery(status string)`, `ObserveWebhookDeliveryDuration(seconds float64)` + +7. **Retention methods** — `IncRetentionRowsDeleted(table string, count int64)` + +The interface enables a no-op implementation for when metrics are disabled or initialization fails, and a mock implementation for testing. + +### GaugeFunc callbacks and store dependency + +Several metrics use `GaugeFunc` with callbacks that query the database at scrape time: + +1. **Queue depth callback** — Queries `SELECT COUNT(*) FROM ai_processing_log WHERE Status = 'pending'`. This runs on each Prometheus scrape (typically every 15-30 seconds). The query is lightweight (single count with index on Status) and should use the replica database. + +2. **Stale items callback** — Queries `SELECT COUNT(*) FROM ai_processing_log WHERE Status = 'processing' AND StartedAt < (now - 5 minutes)`. Also lightweight and replica-safe. + +3. **Total embeddings callback** — Queries `SELECT COUNT(*) FROM ai_embeddings WHERE DeleteAt = 0`. On a large knowledge base, this could be slow. Consider using an estimated count via `pg_class.reltuples` if exact counts prove too expensive, or cache the count with a short TTL. + +4. **Circuit breaker state** — Not a database query. The callback reads the in-memory circuit breaker state from the Gemini client (step 004). The Gemini client should expose a method that returns the current state as an integer (0/1/2). + +All GaugeFunc callbacks must be safe to call from any goroutine (they are called by the Prometheus library during scrape handling). They should not panic — errors should result in a zero value being returned. + +### Integration with admin status endpoint + +The admin status endpoint (step 010) provides a JSON summary of the plugin's health. Several of the metrics defined here can also be surfaced through that endpoint: + +1. **Processing queue depth** — The same "pending items count" used by the GaugeFunc is useful for the status endpoint's queue depth field. + +2. **Circuit breaker state** — The status endpoint reports whether the Gemini circuit breaker is open/closed/half-open. + +3. **Recent error rate** — The status endpoint can compute this from the error counter, but for simplicity it may maintain its own in-memory counter. The Prometheus metrics and the status endpoint are complementary — Prometheus provides time-series history for dashboards, while the status endpoint provides a point-in-time JSON snapshot for quick checks. + +The metrics collector should provide accessor methods (or expose the GaugeFunc callbacks) so the status endpoint handler can reuse the same queries rather than duplicating them. + +### Shutdown + +During `OnDeactivate` (extending step 002): + +1. **No explicit cleanup needed** — Prometheus registries are garbage collected when the metrics collector struct is released. There is no background goroutine to stop or connection to close. + +2. **GaugeFunc callbacks become stale** — After deactivation, the database connections used by GaugeFunc callbacks may be closed. Since the ServeMetrics hook will no longer be called after deactivation, this is not a practical concern. However, if there is a race between deactivation closing the DB and a final scrape calling a GaugeFunc, the callback should handle the error gracefully (return 0). + +## Acceptance Criteria + +- [x] A metrics collector struct is created with a `prometheus.Registry` and all metric instruments +- [x] All metrics use the namespace `org_memory` to avoid collisions with Mattermost's `mattermost` namespace +- [x] Standard Go process and runtime collectors are registered on the plugin's registry +- [x] The `ServeMetrics` hook is implemented and delegates to `promhttp.HandlerFor(registry, opts)` +- [x] The metrics endpoint serves Prometheus text exposition format at `localhost:8067/plugins/{plugin_id}/metrics` +- [x] Processing pipeline metrics include: items processed counter (by job type), items errored counter (by job type), processing duration histogram (by job type), queue depth gauge, stale items gauge, and items ingested counter +- [x] Gemini API metrics include: request counter (by operation and status), request duration histogram (by operation), rate limit hits counter, and circuit breaker state gauge +- [x] RAG query metrics include: query counter (by type), query duration histogram, search result count histogram, permission cache hit/miss counter, session hit/miss counter, and no-results counter +- [x] Embedding metrics include: embeddings generated counter, chunks per post histogram, and total stored embeddings gauge +- [x] Channel summary metrics include: summaries generated counter (by type), channels skipped counter, and batch duration histogram +- [x] Webhook delivery metrics include: deliveries counter (by status) and delivery duration histogram +- [x] Data retention metrics include: rows deleted counter (by table) +- [x] A Go interface for the metrics collector is defined so components can depend on the interface rather than the concrete struct +- [x] A no-op implementation of the metrics interface exists for when metrics initialization fails or for testing +- [x] GaugeFunc callbacks query the replica database and handle errors gracefully (return zero on failure) +- [x] The metrics collector is initialized during OnActivate and the promhttp handler is stored for reuse +- [x] If metrics initialization fails, the plugin continues to function with the no-op metrics implementation +- [x] The circuit breaker state gauge reflects the current Gemini circuit breaker state (0=closed, 1=half-open, 2=open) +- [x] Histogram bucket boundaries are appropriate for each metric's expected range (sub-second to minutes for API calls, 1-32 for chunks per post) +- [x] The metrics collector provides accessor methods for queue depth and circuit breaker state so the admin status endpoint (step 010) can reuse them diff --git a/tasks/0001-org-memory-ai-knowledge-base/015.md b/tasks/0001-org-memory-ai-knowledge-base/015.md new file mode 100644 index 000000000000..77c17d458c5e --- /dev/null +++ b/tasks/0001-org-memory-ai-knowledge-base/015.md @@ -0,0 +1,186 @@ +# 015: Webapp: RHS panel with query interface + +## Context + +This step creates the plugin's webapp bundle — the browser-side code that provides the "Ask your workspace" user interface. The webapp registers with Mattermost during initialization, adds an icon to the app bar that toggles a Right-Hand Sidebar panel, and renders a query interface within that panel. Users type a natural language question, the plugin sends it to the backend API (step 010), and the response is rendered as markdown with clickable citation links pointing to original posts. + +This step focuses on the initial query flow only — one question, one answer, with citation links. Multi-turn follow-up queries, WebSocket event handling, and "summary available" indicators are deferred to step 016. The separation keeps this step focused on the core registration, rendering, and API integration. + +The webapp bundle is built as a single JavaScript file (`webapp/dist/main.js`) that Mattermost loads dynamically via a `