Skip to content

JAMES-4209 Improve S3 data recovery handling + JAMES-4224 entropy 256 -> 128 bits - #3153

Open
chibenwa wants to merge 7 commits into
apache:masterfrom
chibenwa:recovery
Open

JAMES-4209 Improve S3 data recovery handling + JAMES-4224 entropy 256 -> 128 bits#3153
chibenwa wants to merge 7 commits into
apache:masterfrom
chibenwa:recovery

Conversation

@chibenwa

@chibenwa chibenwa commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Key ideas:

  • don't pay for the 256 bit of entropy that are mathematically not needed. Let it for a grab for paranoids.
  • don't use an extra side car object for S3 recovery. Use directly a metadata of the header blob
  • better centralize blobId generation logic and encode it in BlobId.Factory

What this achieves:

  • ~8% size reduction of amortized tiered per message cost in Cassandra ( ~ 560 byte per message)
  • dramatic reduction of Rados omap metadata (~40% for the side car and ~18% for shorter names)

Key design decision:
 - DUPLICATE headers: needed for recovery info unicity
 - Use of BlobStoreDAO: apply the above AND allow passing metadata

Impact:
 - full scan of the generation needed
 - Need to explicitly ask to cache headers
 - Save-in-sequence body-then-header is needed...
 - 2 object instead of 3 thus limiting dramatically pressure on S3 store metadata
 - Simplify GC: no side car handling

@quantranhong1999 quantranhong1999 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I rebased to solve conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants