Skalierungs-Add-ons: S3-Storage-Treiber + optionale Batch-API - #18
Merged
Merged
Conversation
Put storage behind a backend-agnostic driver interface selected by the
STORAGE_DRIVER env var: `local` (default, unchanged on-disk behaviour) or
`s3` for S3 / S3-compatible object storage (AWS S3, MinIO, Cloudflare R2,
Hetzner). Additive and opt-in — without STORAGE_DRIVER the local driver is
used and behaviour is byte-for-byte the same as before.
- src/lib/storage/ replaces the single storage.ts:
- driver.ts: StorageDriver interface, shared pure helpers (safeInlineMime,
sanitizeName, StorageLimitError), the canonical rel-path key builder, and
the shared size+sha256+maxBytes upload meter.
- local.ts: LocalStorageDriver (the previous fs implementation, verbatim).
- s3.ts: S3StorageDriver — AWS SDK imported lazily so the local path never
loads it; PutObject/GetObject/HeadObject/DeleteObject + lib-storage
multipart Upload for streams; ranged GetObject for HTTP range requests;
credentials via the standard AWS chain.
- index.ts: facade keeping every existing named export + new readFileBuffer,
caches the selected driver.
- fileStream is now async (S3 GetObject is async); the file route and the
zip-unpacker await it.
- extract.ts and media.ts now read stored binaries through readFileBuffer
instead of reading local disk directly, so OCR/transcription/PDF/DOCX
extraction work against S3 too.
- No schema change: the stored key is backend-agnostic.
- Tests: mocked S3 driver (put/get/head/delete/range/key-prefix/safe-key) and
a real LocalStorageDriver round-trip (save/read/size/delete, streaming with
size+hash+maxBytes, byte-range).
- Docs: STORAGE_DRIVER / S3_* documented in .env.example, docker-compose.yml
and docs/self-hosting.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
Run the coverage-generation MAP step through the provider's async Batch API (Anthropic Message Batches / OpenAI Batch) — ~50% cheaper for large modules. Additive and opt-in via the `ai.useBatchApi` admin flag (default off); with the flag off, or for non-Anthropic/OpenAI providers, the synchronous live path is unchanged and is also the fallback if a batch submit fails. - src/lib/ai/generation/batch-adapter.ts (new): pure, unit-tested builders and parsers (Zod->JSON schema, Anthropic tool-use requests, OpenAI json_schema tasks, result parsing with usage) plus thin network calls; the vendor SDKs (@anthropic-ai/sdk, openai) are imported lazily so non-batch deploys never load them. resolveBatchProvider reads credentials from settings + BYOK. - generate.ts: extract the two MAP prompts into shared buildCardPrompt/ buildQuestionPrompt and the dedup+insert into persistCards/persistQuestions, so live and batch paths produce identical output. runCoverageGeneration submits one batch for all uncovered topics when the flag is on and returns; grounding/retrieval still runs live. - src/lib/ai/generation/batch-poll.ts (new): cron-driven poller that, once a vendor batch ends, fetches results, de-dupes + inserts, records token usage per topic (requirement A), and completes the job. Failed items are marked in coverage so a re-run regenerates only them. - jobs: new poll-batches queue on a */5 schedule (matches send-reminders). - schema: generation_job gains nullable batch_ref + batch_model (migration 0034); no other data-model change. - settings: ai.useBatchApi flag + admin toggle (Switch) with i18n. The batch path is asynchronous: results arrive over minutes to hours and the coverage view fills gradually. Submit/poll/fetch against the real vendor APIs is verified on the production stand; CI covers the pure builder/parser logic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
veniplex
changed the title
veniplex
added
the
release-candidate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Überblick
Zwei der zurückgestellten Skalierungs-Bausteine (nach PR #14 Kern und PR #17 Worker + HNSW), beide additiv und opt-in — ohne die neuen Env-Variablen/Flags läuft alles byte-gleich wie bisher:
local(Default) unds3.1 — S3-/Object-Storage-Treiber (
feat(storage))Für mehrere GB an Modul-Material braucht Self-Hosting Object Storage statt einer einzelnen Festplatte. Per Env umschaltbar zwischen
local(Default, unverändert) unds3(AWS S3, MinIO, Cloudflare R2, Hetzner). Keine Schema-/DB-Änderung.src/lib/storage/ersetztstorage.ts:driver.ts(Interface + treiber-agnostische Reinfunktionen + geteilter Upload-Meter für Größe/sha256/maxBytes),local.ts(bisherigenode:fs-Implementierung 1:1),s3.ts(AWS-SDK lazy importiert;PutObject/GetObject/HeadObject/DeleteObject+lib-storageUploadfürs Streaming; ranged GetObject für HTTP-Range/206),index.ts(Fassade + neureadFileBuffer).fileStreamist jetzt async (Datei-Route + Zip-Entpackerawaiten es);extract.ts/media.tslesen Binärdateien überreadFileBufferstatt direkt von der Disk — damit funktionieren OCR/Transkription/PDF-/DOCX-Extraktion auch gegen S3.STORAGE_DRIVER=local|s3; für S3:S3_BUCKET(Pflicht),S3_REGION/AWS_REGION,S3_ENDPOINT,S3_FORCE_PATH_STYLE,S3_KEY_PREFIX. Credentials über die Standard-AWS-Chain.2 — Optionale Batch-API für vollständige Generierung (
feat(generation))Die „Complete"-Generierung macht pro Thema einen LLM-Call (MAP) — bei 100+ Themen viele Live-Calls. Anthropic Message Batches / OpenAI Batch erledigen dieselben Calls asynchron zu ~50 % der Kosten. Per Admin-Flag
ai.useBatchApi(Default aus); ohne Flag oder bei Nicht-Anthropic/OpenAI-Providern bleibt der synchrone Live-Pfad aktiv (auch als Fallback, falls ein Batch-Submit fehlschlägt).batch-adapter.ts(neu): reine, unit-getestete Bausteine (Zod→JSON-Schema, Anthropic-Tool-Use-Requests, OpenAI-json_schema-Tasks, Result-Parsing inkl. Usage) + dünne Netz-Calls; Vendor-SDKs (@anthropic-ai/sdk,openai) lazy importiert.resolveBatchProviderliest Credentials aus Settings + BYOK.generate.ts: MAP-Prompts in geteiltebuildCardPrompt/buildQuestionPromptund Dedup+Insert inpersistCards/persistQuestionsextrahiert — Live- und Batch-Pfad erzeugen identische Ausgabe.runCoverageGenerationsubmittet bei aktivem Flag einen Batch für alle offenen Themen und kehrt zurück; Grounding/Retrieval bleibt live.batch-poll.ts(neu): cron-gesteuerter Poller (*/5-Queuepoll-batches), holt nach Batch-Abschluss die Ergebnisse, dedupt + inserted, erfasst pro Thema den Token-Verbrauch (Pflicht-Anforderung A) und schließt den Job ab. Fehlgeschlagene Items werden in der Coverage markiert, sodass ein Re-Run nur sie neu erzeugt.generation_jobbekommt nullbarebatch_ref+batch_model(Migration0034); sonst keine Datenmodell-Änderung.ai.useBatchApi+ Admin-Toggle (Switch) mit i18n (de/en).Tests / Verifikation
src/lib/storage/storage.test.ts(S3-Treiber gegenaws-sdk-client-mock+ echter LocalStorageDriver-Round-Trip) undsrc/lib/ai/generation/batch-adapter.test.ts(Schema-Konvertierung, Request-Builder, Result-Parser, Provider-Resolution).npm run typecheck(0 Fehler),npm run lint(sauber),npm run test(102 bestanden, 3 skipped),npm run build(erfolgreich).STORAGE_DRIVERbzw. mituseBatchApiaus verhält sich alles exakt wie bisher.Noch offen (Folge-PR)
E — tus-Resumable-Uploads (tus-
s3-storebaut auf dem S3-Treiber auf).🤖 Generated with Claude Code
https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH