Repository navigation
Conversation
An OID and its size are public in pointer files, so knowing them must not link an object to a repository. Presigned verify only checked that the object existed at its shared location, so a writer of one repository could skip the upload and gain an object stored by another repository. Each presigned upload now goes to a one-time location under uploads/, named by an upload ID that batch adds to the upload and verify actions. Verify requires the content at that location, copies it to the shared location when missing, links the object, and deletes the upload. The janitor deletes abandoned uploads after a day. Service.CrossRepoDedup and Store.GetObjects are removed, so every repository uploads an object before it is linked.
Clone each repository fresh so that git-lfs downloads through the smudge filter, check that the second repository must upload the object, and check that R2 holds one copy with no upload left behind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #23.
An LFS OID is not secret: every pointer file carries the OID and size in plain text. Presigned verify only HEADed the shared
<bucket>/<oid>key, so a writer of repository B could skip the upload, call verify with an OID and size from someone else's pointer file, and then download the content through B.CrossRepoDedup(on in lfsd) linked such objects at batch time by design.Changes
?upload=<id>to the upload and verify action hrefs of presigned storage. The Git LFS client uses action hrefs verbatim.uploads/<id>/<oid>. A missing or invalid upload ID gets 400.<oid>when it is not stored yet, links the object, and deletes the upload key. The bucket keeps one copy per object at<oid>.PresignergainsUploadURI,Copy(CopyObject, or a multipart UploadPartCopy above 5 GiB), andDeleteUploads.PresignPuttakes the target URI.Service.CrossRepoDedup,Store.GetObjects, andUploadResult.Existedare removed.Service.UploadandService.Verifytake the upload ID, andServepasses theuploadquery parameter.uploads/keys older than a day.Test plan
moon run :lint :test. New tests include the cross-repository regression (fails against the old verify), two repositories sharing one copy with nothing left inuploads/, upload ID validation on upload and verify, distinct upload IDs per object and per batch,Servequery handling,Copy, byte range splitting for multipart copy, andDeleteUploadsand the janitor keeping recent uploads.go test -raceinlfs,GOOS=windows go vet.<oid>keys afterward.SKIP_AUTHforge on the same storage, and checks the bucket.MAX_OBJECT_SIZEis raised above its 5 GiB default.