GB-scale module processing: complete, incremental AI generation with full audit/token logging - #14
Merged
Conversation
Introduces src/lib/ai/run.ts as the single choke point every AI request goes through. It normalizes the various AI SDK usage shapes (inputTokens/outputTokens, embedding `tokens`, prompt/completionTokens), always writes the aiUsageLog token ledger, and records a human-readable audit entry (model, feature, token counts in `after`) so every AI action is visible in the audit log — satisfying the requirement to always see when the AI was invoked and how many tokens it used. Routes all existing AI call sites through runAi: - flashcards, quiz, quiz-grading, study-plan, analysis, semester-plan, thesis topics/outline/milestones, chat (streamText onFinish) - OCR (ai_extract) and transcription (ai_transcribe) in media.ts; getTranscriptionModel now returns the model ref for logging Extends AuditOperation with ai_generate/ai_embed/ai_summarize/ ai_transcribe/ai_extract and guards all ai_* ops as non-undoable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
…ding Upload no longer buffers whole files in memory: the client sends the raw file body (metadata in query params + x-file-name header) and the server streams it to disk via saveStream(), computing size + sha256 as it goes and hard-capping at maxUploadMb mid-stream. Enables multi-GB uploads without OOM. Incremental processing / reuse of results: - material gains content_hash, text_storage_path, char_count, summary, extraction_status, chunks_total/embedded; material_chunk gains level + parent_chunk_id (for later summary tree). - Identical re-uploads (same content hash in the module) are skipped, in both the upload route and zip unpacking. - processMaterial extracts text once and stores the full text on disk; re-runs reuse it instead of re-OCR/transcribing (saves tokens). - Embedding runs in bounded batches, skips chunks already embedded for the active model (resumable), tracks chunks_embedded for progress, and aggregates one ai_embed audit entry per material. Query embeddings log usage without audit spam. - embed-material job uses singletonKey + higher retryLimit; processing is idempotent so retries resume. Extraction cap for plain-text files raised from 2 MB to 25 MB (MAX_TEXT_EXTRACT_MB). Migration 0031 is additive; backfills extraction_status='ready' for already-extracted materials. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
…module Replaces top-k-only generation with a coverage-driven map-reduce so flashcards/quizzes cover the ENTIRE material, not a ~6-chunk sample. - summarize.ts: RAPTOR-lite doc/section summaries (map-reduce over a material's chunks), stored as level-1 chunks + material.summary; runs as a summarize-material job after embedding. Skips already-summarized (immutable) content. - outline.ts: buildModuleOutline derives a de-duplicated topic outline over ALL material summaries, versioned + fingerprinted so it is only rebuilt when materials change; topic ids are carried across rebuilds (by normalized title) to preserve coverage. - generate.ts: runCoverageGeneration iterates every topic, grounds on a focused per-topic retrieval (searchChunksInMaterials, ~16 chunks scoped to the topic's source materials), generates items, and de-duplicates (semantic + normalized-string) across the whole run. generation_job tracks progress; generation_coverage (unique per target+topic) enables cross-run reuse — a re-run only fills new/uncovered topics. - New schema: module_outline, outline_topic, generation_job, generation_coverage (migration 0032). New generate-coverage pg-boss queue (long lease, resumable). - UI: "Complete (cover the whole material)" toggle in the deck and quiz generate dialogs, with a live coverage progress bar (generation-progress.tsx) polling generationStatus; i18n de/en. All AI calls (summaries, outline, per-topic generation, dedup/query embeddings) flow through runAi, so tokens + audit are logged throughout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
The audit log now shows every AI action (generate/embed/summarize/ transcribe/extract) with its token count (read from the entry's stored metadata), model and feature, and lets users filter by the new AI operations. Fulfils the requirement to always see, in the audit log, when the AI was invoked and how many tokens it used. i18n de/en. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
…bedding Retrieval quality at scale (Phase 2, core): - Adds a generated tsvector column (content_tsv) + GIN index on material_chunk and fuses vector cosine ranking with Postgres full-text ranking via Reciprocal Rank Fusion. Catches exact terms/acronyms/formulae the embedding model misses (and vice-versa), improving both chat retrieval and per-topic generation grounding. searchChunks and searchChunksInMaterials now share one hybridSearch path. - Contextual retrieval: each chunk is embedded with a short document-context header (title + summary when available) prepended, stored in contextual_header; the raw chunk text is kept for storage and lexical search. Improves precision on ambiguous chunks. Migration 0033 is additive (generated column + GIN index). Note: the pgvector HNSW ANN index is intentionally deferred — it requires a pinned embedding dimension (the column is deliberately dimensionless) and a live DB to build safely; the sequential cosine scan remains the default, as the schema chose for personal scale. Tracked as a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
Coverage generation now checks the monthly token limit before each topic and stops cleanly when it is reached; uncovered topics stay pending so a later run resumes exactly where it left off. Adds estimateTokens helpers for pre-flight sizing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
veniplex
added
the
release-candidate
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ziel
Ein Modul soll mehrere GB an Daten aller Dateitypen verarbeiten können, und die KI soll daraus vollständige Lernpläne, Karteikarten und Quizze erzeugen — nicht nur eine gesampelte Teilmenge. Kernproblem heute: Generierung gründet auf semantischem Top‑k (
searchChunks, ~6 Chunks) → das Modell sieht nie mehr als eine Handvoll Auszüge. Vollständigkeit erfordert Vollkorpus‑Abdeckung statt Top‑k.Zwei durchgängige Pflicht‑Anforderungen (vom Nutzer):
Umsetzung in Phasen, alle nacheinander vollständig. Detailplan:
/root/.claude/plans/…(im PR‑Verlauf zusammengefasst).Enthalten in diesem PR (laufend erweitert)
Phase 0 – Fundament
src/lib/ai/run.ts(runAi): einziger Choke‑Point für alle KI‑Calls; schreibt immer den Token‑Ledger (aiUsageLog) und einen Audit‑Eintrag mit Modell/Feature/Tokens. Alle bestehenden Aufrufsites (Flashcards, Quiz, Grading, Studienplan, Analyse, Semesterplan, Thesis, Chat, OCR, Transkription, Embedding) laufen jetzt darüber. (Anforderung A)saveStreamauf Platte, berechnet Größe + sha256, hart gedeckelt beimaxUploadMb.content_hash+ Skip identischer Re‑Uploads (Upload + Zip); Volltext auf Platte (text_storage_path) → keine erneute Extraktion/OCR/Transkription; batched, resumierbares Embedding mitextraction_status/chunks_embedded. (Anforderung B, Material‑Ebene)0031(rein additiv, mit Backfill).Geplant (Folge‑Commits in diesem PR)
Verifikation
🤖 Generated with Claude Code
Generated by Claude Code