fix(backup): stream the off-host copy so it fits a small container - #927
Merged
Conversation
The off-host job was said to benefit from the streaming work the weekly in-database backup got, and it did not. Only the payload builder was shared: `runOffhostBackup` called `buildFullBackupJson` for the whole document, `gzipSync` over that whole string, a whole-buffer `createCipheriv` pass over the result, and handed the finished buffer to a single `PutObject`. Four full copies of the record alive at once, on the same arithmetic the weekly pass had already been fixed for. Configuring an S3 target on a long-lived record therefore restarted the container. On an account of 445 000 measurements the JSON alone is 242 MB, and against a 1 GB container's 524 MB heap the run died of heap exhaustion seventeen seconds in — taking every signed-in session with it, because the job shares the app process. The write path is a stream end to end now: `streamFullBackupJson` produces the document a page at a time, gzip and an incremental AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage` puts it up in two 8 MB parts. What the process holds is fixed by the pipeline's shape rather than by the record. Measured on that same 445 000-measurement account under `--max-old-space-size=450`, with the heap pre-loaded to what a warm server holds: the old path dies with `Ineffective mark-compacts near heap limit`, the new one finishes with a peak 79 MB above its baseline and writes a 9.1 MB object. The stored envelope gains version 3, which moves the GCM tag from in front of the ciphertext to the end. That is the only change the stream required — the tag exists only once the last block is in, so a leading one means the whole object must exist before its first byte can be sent. Nothing about the authentication moves: the tag still covers every ciphertext byte and is verified before a byte of plaintext comes back. Versions 1 and 2 still read, so every object already in a bucket restores through the same reader with no flag to tell them apart. The incremental writer is the crypto module's, split into its byte-level half so the base64 form the in-database blob uses and the raw form an object needs are one implementation under two framings. The off-host key handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or 32-byte base64, passed in rather than looked up. A run that uploaded nothing for anybody is a failed job now, not `ok: true` with an empty bucket behind it. Wrong credentials, a missing bucket and an unreachable endpoint all fail every account rather than one, and the target's own sentence rides out as the cause. A run where some account got a copy still succeeds; retrying the whole cohort over one object would re-upload everybody's. The one ceiling left is structural rather than a memory bound, and it is counted rather than discovered: a multipart upload carries 10 000 parts, so an object past 80 GB is refused for that account with a clear message instead of failing halfway through with an SDK error about part numbers. `tests/integration/offhost-backup-streaming-memory.test.ts` pins the budget from both sides — what the streaming uploader holds, and that the materialising path does not fit the same budget on the same fixture in the same process, so the number cannot pass by measuring nothing.
Nothing calls it any more: the weekly pass and the off-host job both walk the payload through the streaming writer. Keeping an exported function that builds the entire record as one string invites the next caller to reintroduce the memory profile this branch removes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first ever configured run of the nightly off-host backup took production down:
FATAL ERROR: Reached heap limitseventeen seconds in, container restarted. The destination was blanked again the same minute. This is what lets it be switched back on.The job had shared only the payload builder with the v1.38.6 streaming work; everything after it still materialised: the whole JSON string,
gzipSyncover it, a whole-buffer cipher pass, onePutObject. Four copies of the record. The commit message says so plainly rather than repeating the earlier claim that the job "benefited".Reproduced under the real ceiling with a comparable record (445 000 measurements, 30 000 mood entries, 60 000 intake events) against MinIO, from a process already holding part of its heap, since a cold script survives and a warm server does not. At 120 MB retained the old path dies inside the JSON builder; the new one completes at a peak of 319 MB against a 183 MB baseline. At 300 MB retained, closest to production, it still completes, 79 MB above baseline. Streaming holds 14 MB live where materialising held 128.
The stored envelope gains version 3 with the authentication tag at the end, which is the only place a stream can put it. Versions 1 and 2 still read: a unit test writes one of each and restores both, and a version 2 and a version 3 object of the seeded account were restored end to end through the restore script with identical counts, tombstones included. The cipher stays one implementation under two framings.
Bad credentials, a missing bucket and a refused connection each upload nothing, leave no orphaned multipart parts, keep the process up, and turn zero uploads for a non-empty cohort into a job failure that names the cause. A refused upload that rejected mid-write used to leave the producer waiting on a drain nobody delivered; that arm tears the pipeline down too, and the test for it hangs to timeout if the line is reverted.
Five mutations, each red on its own.
@aws-sdk/lib-storageis pinned exactly to the installed client's line. Multipart needsAbortMultipartUploadin the bucket grant; the runbook says so.