Skip to content

lfs: require proof of upload for presigned storage - #31

Open
unknwon wants to merge 2 commits into
lfs-libraryfrom
lfs-upload-proof
Open

unknwon wants to merge 2 commits into
lfs-libraryfrom
lfs-upload-proof

Conversation

@unknwon

@unknwon unknwon commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Stacked on #23.

An LFS OID is not secret: every pointer file carries the OID and size in plain text. Presigned verify only HEADed the shared <bucket>/<oid> key, so a writer of repository B could skip the upload, call verify with an OID and size from someone else's pointer file, and then download the content through B. CrossRepoDedup (on in lfsd) linked such objects at batch time by design.

Changes

  • Batch adds a UUIDv7 upload ID as ?upload=<id> to the upload and verify action hrefs of presigned storage. The Git LFS client uses action hrefs verbatim.
  • Upload redirects to a presigned, checksum-signed PUT for uploads/<id>/<oid>. A missing or invalid upload ID gets 400.
  • Verify HEADs the upload key (422 when missing or of another size), records the pending object, copies the upload to <oid> when it is not stored yet, links the object, and deletes the upload key. The bucket keeps one copy per object at <oid>.
  • Presigner gains UploadURI, Copy (CopyObject, or a multipart UploadPartCopy above 5 GiB), and DeleteUploads. PresignPut takes the target URI.
  • Service.CrossRepoDedup, Store.GetObjects, and UploadResult.Existed are removed. Service.Upload and Service.Verify take the upload ID, and Serve passes the upload query parameter.
  • The janitor deletes uploads/ keys older than a day.
  • Proxied (filesystem) storage keeps its behavior, which already required matching content.

Test plan

  • moon run :lint :test. New tests include the cross-repository regression (fails against the old verify), two repositories sharing one copy with nothing left in uploads/, upload ID validation on upload and verify, distinct upload IDs per object and per batch, Serve query handling, Copy, byte range splitting for multipart copy, and DeleteUploads and the janitor keeping recent uploads.
  • go test -race in lfs, GOOS=windows go vet.
  • Manual e2e with git-lfs 3.6.1 against lfsd, PostgreSQL 17, and gofakes3 or filesystem storage: two repositories push the same file (the second uploads it again), fresh clones match byte for byte, a third repository gets 422 on verify with an unused upload ID and 400 without one, then 404 on download. The bucket holds only <oid> keys afterward.
  • Integration (R2) now also pushes the object from a second repository served by a localhost SKIP_AUTH forge on the same storage, and checks the bucket.
  • Not run: multipart copy of an object over 5 GiB against R2. It is reachable only when MAX_OBJECT_SIZE is raised above its 5 GiB default.

An OID and its size are public in pointer files, so knowing them must not
link an object to a repository. Presigned verify only checked that the
object existed at its shared location, so a writer of one repository could
skip the upload and gain an object stored by another repository.

Each presigned upload now goes to a one-time location under uploads/,
named by an upload ID that batch adds to the upload and verify actions.
Verify requires the content at that location, copies it to the shared
location when missing, links the object, and deletes the upload. The
janitor deletes abandoned uploads after a day.

Service.CrossRepoDedup and Store.GetObjects are removed, so every
repository uploads an object before it is linked.
Clone each repository fresh so that git-lfs downloads through the smudge
filter, check that the second repository must upload the object, and check
that R2 holds one copy with no upload left behind.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant