Android chunked-upload engine - #33
Merged
Merged
Conversation
This was referenced Aug 24, 2026
dmurphy5
changed the title
dmurphy5
force-pushed
the
dylan/v9-5-android
branch
2 times, most recently
from
August 26, 2026 20:26
5bfc439 to
d6be9e4
Compare
dmurphy5
force-pushed
the
dylan/v9-5-android
branch
from
August 27, 2026 19:46
d6be9e4 to
fc7d5e8
Compare
dmurphy5
marked this pull request as ready for review
August 27, 2026 19:51
Per docs/design/chunked-uploads.md. A chunked startUpload renames the source into a library-owned directory and persists a durable manifest (parts, per-part status, accept rules, expiresAt) before enqueueing work. The manifest is the single resume mechanism: re-calling with the same id reconciles — identical parts resume (stored headers replaced, accepted parts skipped), covering crash recovery, stop/re-start, and reauth alike; a different parts array while stopped is a recreate (blob kept, parts replaced, statuses reset). The worker runs 3 parts in flight (library constant), one coroutine per part index, each streaming its byte range straight from the moved file through a RandomAccessFile-backed request body. Part success is 2xx or an accept-rule match (status + bodyIncludes); transient failures back off exponentially with expiresAt as the only cap; a non-accepted response out of its bounded retries journals an 'error' carrying the part index and stalls the upload. Terminal events reuse the journal-before-emit path. Bytes are deleted only when a 'completed' event is acked; removeUpload releases manifest and bytes after any other terminal, and is a safe no-op on simple upload ids. Accept rules also replace the dead acceptStatus parsing for simple uploads (the JS surface already sends them), and the notification/transport helpers move to shared files so both workers use one implementation. The library-wide transfer semaphore stays at 1; widening to the design's global cap of 4 belongs to the hardening slice. Review fixes: - F1: normalize Upload after Gson fromJson so a replayed v8 job (acceptStatus, no accept) can't NPE post-transmit and re-send the file up to 6 times. - F2: startUpload's load→reconcile→save is one atomic store operation (ChunkedManifestStore.compute), so a running worker's markAccepted can't be clobbered on disk. - F3: enqueue with APPEND_OR_REPLACE instead of KEEP — a resume arriving while the failing run's row is still RUNNING is queued, never silently dropped; a trailing run over a settled manifest is a clean no-op. - F4: a worker that finds no manifest exits success silently (completed+acked or removed — both settled), and a completed ack cancels the id's queued work. - F5: recreate semantics per the amended design doc — same id + different parts accepted while not running, blob kept, statuses reset, new ranges must tile [0, blob size) exactly; rejected while running. - F6: part range past blob EOF is a terminal 'file' error immediately; empty response bodies stay empty (no reason-phrase substitute); getAllUploads returns one row per id with iOS's state vocabulary; terminals for removed ids are suppressed; ChunkedWorkerGate serializes cancel-then-start so two workers can't PUT the same part concurrently. Final review fixes: - W1: the worker returns Result.success() even after journaling a terminal error — the journal + manifest are the outcome record, never the row state. A FAILED prerequisite makes WorkManager mark appended dependents FAILED without running them, silently destroying a resume enqueued during the failing run's teardown. APPEND_OR_REPLACE comment corrected to match. - W2: a refused foreground start (API 31+ background restriction, incl. the Android 15 dataSync time-limit analog) no longer fails the run — both workers continue without foreground priority. - W3: a trailing run whose completion was already acked reports nothing instead of minting a duplicate 'completed' (ChunkedEngine.completionReport). - W4: a completed ack releases manifest/bytes and cancels queued work only when the manifest is still allAccepted — never a recreate running under the same id. - W5: the init-failure path removes the upload's progress entry. - W6: cancelUpload emits the module-side 'cancelled' whenever no row is RUNNING (BLOCKED appended rows included), not only for ENQUEUED. - W7: skip the chunked enqueue when a queued successor row already exists; the manifest reconcile still lands. - W8: samePartsAs is order-independent (sorted by range start), matching tilesExactly, and accepted flags follow the range, not the index. - W9: a first-call (create) validates the parts tile [0, blobSize) exactly, rejecting before any manifest is saved so the blob stays adoptable. - JVM tests for W1, W3, W4, W6, W7, W8, W9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dmurphy5
force-pushed
the
dylan/v9-5-android
branch
from
August 28, 2026 16:49
fc7d5e8 to
90addbb
Compare
elliottkember
approved these changes
Sep 2, 2026
This was referenced Sep 3, 2026
dmurphy5
added a commit
that referenced
this pull request
Sep 3, 2026
This PR moves the merged chunked-upload work onto master. It replaces #39, which conflicted. The stacked PRs #32–#35 were reviewed and merged, but their stack was rooted on `dylan/v9-3-configure`, so their four squash commits landed on that branch and not on master. This branch carries those same four commits, cherry-picked onto the current master. The tree at the tip is byte-identical to the merged stack. There are no new changes, and there are no conflicts: the branch is a fast-forward of master. - Chunked JS surface — types, codegen spec, chunkPlan (#32) - Android chunked-upload engine (#33) - iOS chunked-upload engine (#34) - Global cap, example app, docs, v9.0.0 (#35) Prefer "Rebase and merge", so the four commits keep their identities on master. After the merge, master is v9.0.0 and gets the `v9.0.0` tag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the Android engine for chunked uploads.
At start, the library moves the file into its own directory. The library then owns the file. The library writes a durable manifest to disk. The manifest holds the parts, the status of each part, and the options.
The engine sends the parts with WorkManager. A RandomAccessFile-backed body streams each part directly from the file. There are no chunk copy files, and disk usage does not double. A maximum of 3 parts transmit at one time (the window). The engine applies the accept rules and the expiry time.
The engine writes each event to a journal before it emits the event. The library deletes the file only when the consumer acknowledges the
completedevent.startUploadwith the same id resumes the upload. The engine skips the parts that the server accepted. The same id with different parts recreates the upload. The engine permits a recreate only when the upload is not running, and the new parts must tile the file exactly.Review fixes in this slice:
acceptStatusfield. A test uses the literal v8 JSON shape.APPEND_OR_REPLACE, and a worker returns success after it journals a terminal error. Thus a failing chain cannot silently drop a resume.84 JVM tests.
🤖 Generated with Claude Code