fix(uploads): underscore-led keys, bounded bucket names, dedup only on live objects - #1757
Merged
Conversation
…nly live objects Three independent static-deploy upload defects: - Custom keys rejected a leading underscore, so every Next.js export failed at its first _next/static/** file. Validation moves to a shared module and admits the underscore while keeping traversal, leading-slash, NUL and length guards. - Physical S3 bucket names were prefix-key-databaseId, which overran the 63-char limit and could carry illegal characters. Names are now sanitized, truncated to a readable head and suffixed with a digest of the full logical identity. Existing buckets are untouched: physical_name is authoritative once stored. - Deduplication treated any row matching a content hash as a live object, so an upload that was only requested, or was rejected or expired, could be reported as already stored while no bytes existed. Dedup now requires a confirmed status; a stale row is deleted and the upload restarted.
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This comment has been minimized.
This comment has been minimized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three independent defects that each break static-site deploys, reported as constructive-planning #1754, #1755 and #1756.
#1754 — a leading underscore was an invalid custom key. Every Next.js export dies on its first
_next/static/**file. Validation moves out ofplugin.tsintocustom-key.ts(exported, so callers can validate before requesting a URL) and admits the underscore; traversal, leading slash, NUL and the 1024-char bound are unchanged.#1755 — physical bucket names were unbounded and could be illegal. The name was
${prefix}-${bucketKey}-${databaseId}: a 36-char UUID plus any prefix overruns S3's 63-char limit, and neither component was case- or charset-constrained. Each component is now sanitized to[a-z0-9-], truncated to a readable budget, and closed with a 12-hex digest of the full logical identity, so two keys that truncate to the same head still get distinct names:No migration:
platform_buckets.physical_nameis authoritative once stored, and both the lazy and eager lanes only call the resolver when it is NULL — already-provisioned buckets keep their names.#1756 — dedup treated any row matching the content hash as a stored object. A row whose upload was only
requested, or wasrejected/expired, carries no bytes, but both the direct-presigned and managed paths answereddeduplicated: trueand handed back its key — a deploy that silently references nothing. A row is live only when the files table carries the confirm-upload lifecycle at all and its status isuploaded/processed:hasConfirmUploadcomes fromstorage_modules.has_confirm_upload; when the lifecycle is absent there is nothing to read, so every row stays live as before. A stale row is deleted rather than reused: content-addressed keys equal the hash,(bucket_id, key)is unique so a replacement cannot be inserted alongside it, and reusing a rejected row would not re-run the lifecycle from the start.Bucket-name expectations in the settings and server-test suites move to the digest form;
upload.integration.test.tscontinues to assert a storedphysical_nameis honored verbatim.Note:
s3-signer.integration.test.tsfails locally only because no MinIO is running on this box.Link to Devin session: https://app.devin.ai/sessions/924488078ac34f83bc9c1af8e192af0f
Requested by: @pyramation