Chunk-merge compact concurrency

[rkistner]

This adds a new storage.chunk_compaction_concurrency option that allows chunk-merge compaction (specific to storage_version: 4) to run concurrently on different buckets.

The main motivation is S3 latency: If we process each bucket sequentially, the S3 latency can dominate the overhead, significantly slowing down compaction. This specifically has an effect for the chunk-merge compaction running after initial replication, since it blocks the activation of the new sync config, and can currently run significantly longer than the initial replication itself.

An existing workaround is running multiple concurrent compaction workers, but this approach is much simpler, especially for the compaction after initial replication.

Full compaction continues processing buckets sequentially, since it is typically much heavier in terms of processing. This may change in the future.

The default concurrency is 2 for inline MongoDB storage 4 when using object storage. This can likely be increased further, but we're keeping the defaults conservative for now. From testing, a concurrency of 4x is sufficient to make the impact of the chunk-merge low on the overall replication time.

AI Usage

Implemented using Codex gpt-6-astra.

[changeset-bot]

🦋 Changeset detected

Latest commit: 02c1732

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72268b06bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[stevensJourney]

The changes look logical and good to me.