Resumable (tus) Uploads für sehr große Dateien - #19
Merged
Merged
Conversation
Add an opt-by-size resumable upload path so a dropped connection or page reload continues a large upload from the last byte offset instead of restarting. Additive — files below the threshold keep the existing direct streaming upload unchanged. - src/app/api/materials/tus/[[...slug]]/route.ts (new): a @tus/server endpoint (handleWeb) with @tus/file-store staging under TUS_DIR. Auth on every request via onIncomingRequest; onUploadCreate validates module/folder ownership, size and quota and stamps the authenticated userId into the upload metadata; onUploadFinish enqueues a finalize job. - src/lib/materials/tus-finalize.ts (new): streams the staged file through storage.saveStream (computing size + sha256, works for local AND S3) then registers the material; removes the staging file when done. - src/lib/materials/ingest.ts (new): registerUploadedFile — the folder/quota/ zip/dedup/insert/embed sequence extracted from the upload route so the direct route and the tus finalizer behave identically. upload/route.ts now calls it. - jobs: new finalize-upload queue + enqueueFinalizeUpload (singletonKey on the tus id; the finalizer removes the staging file so retries are safe). - client: upload-client.ts routes files >= 50 MB through tus-js-client (resume via findPreviousUploads), small files keep the xhr path; shouldUseTus is a pure, unit-tested helper (src/components/materials/upload-transport.ts). - docs: TUS_DIR + staging volume in .env.example, docker-compose.yml, docs/self-hosting.md. No schema change: the tus file-store + pg-boss hold the in-flight state, so an interrupted upload resumes across restarts as long as the staging volume persists. tus staging is local even in S3 mode (one scratch pass); a native tus-s3-store is a later optimization. End-to-end resume is verified on the production stand; CI covers typecheck/lint/build and the transport helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Überblick
Letzter der zurückgestellten Skalierungs-Bausteine (nach PR #14 Kern, #17 Worker+HNSW, #18 S3+Batch-API): resumable Uploads über das tus-Protokoll. Bei einem Verbindungsabbruch oder Reload während eines sehr großen Uploads läuft dieser am letzten Byte-Offset weiter, statt von vorn zu beginnen. Additiv & opt-by-size — Dateien unter der Schwelle nehmen weiter den unveränderten Direkt-Upload.
Kernidee: tus staged lokal →
saveStreamfinalisiert (treiber-agnostisch)Statt tus an ein S3-spezifisches Store-Format zu koppeln, schreibt tus in ein lokales Staging-Verzeichnis (
@tus/file-store); ein Hintergrund-Job streamt die fertige Datei dann durchstorage.saveStreamin das konfigurierte Backend (local oder S3) und bekommt dabei Größe + sha256 (für Dedup, Pflicht B). Ein Storage-Pfad, funktioniert mit beiden Treibern.Änderungen
src/app/api/materials/tus/[[...slug]]/route.ts(neu):@tus/server-Endpoint (handleWeb) mit@tus/file-storeunterTUS_DIR. Auth pro Request überonIncomingRequest;onUploadCreateprüft Modul-/Ordner-Ownership, Größe + Quota und stempelt die authentifizierteuserIdin die Metadaten (nicht clientseitig manipulierbar);onUploadFinishreiht den Finalize-Job ein.src/lib/materials/tus-finalize.ts(neu): streamt die Staging-Datei viasaveStreamin den Storage (local oder S3), registriert das Material, löscht die Staging-Datei.src/lib/materials/ingest.ts(neu):registerUploadedFile— die Sequenz Ordner/Quota/Zip/Dedup/Insert/Embed aus der Upload-Route extrahiert, sodass Direkt-Upload und tus-Finalizer denselben Pfad nutzen (identisches Verhalten).upload/route.tsruft sie jetzt auf.finalize-upload+enqueueFinalizeUpload(singletonKey= tus-ID; der Finalizer entfernt die Staging-Datei, Retries sind gefahrlos).upload-client.tsschickt Dateien ≥ 50 MB übertus-js-client(Resume viafindPreviousUploads/resumeFromPreviousUpload), kleine Dateien behalten den xhr-Pfad. Die Transport-Wahl ist eine reine, unit-getestete Funktion (upload-transport.ts).TUS_DIR+ Staging-Volume in.env.example,docker-compose.yml(App- und Worker-Service),docs/self-hosting.md.Datenmodell
Keine Schema-Änderung: tus-
file-store(Staging) + pg-boss halten den Zustand persistent — ein unterbrochener Upload lässt sich über Neustarts hinweg fortsetzen, solange das Staging-Volume bleibt.Tests / Verifikation
src/components/materials/upload-transport.test.ts: Transport-Schwelle (tus ab 50 MB, xhr darunter, tus aus wenn Browser es nicht unterstützt).npm run typecheck(0 Fehler),npm run lint(sauber),npm run test(105 bestanden, 3 skipped),npm run build(erfolgreich; die tus-Route ist registriert).registerUploadedFile-Pfad weiter.Abschluss der Skalierungs-Add-ons
Damit sind alle geplanten Add-ons (A Worker · B HNSW · C S3 · D Batch-API · E tus) umgesetzt.
🤖 Generated with Claude Code
https://claude.ai/code/session_0191JikNRN8Q2HBtf6fpLXmH
Generated by Claude Code