Remove global MongoDB storage replication lock

[rkistner]

Currently, only one replication stream at a time can flush changes. This is due to:

1. A global in-process replicationLock.
2. A shared global op_id sequence, that only a single replication stream can update at a time.

These can become a bottleneck if there is high replication throughput while also doing initial replication. And we don't need that: We don't need absolute ordering of operations between replication streams.

This changes the implementation to pre-allocate large ranges of op_ids, separate from the flush transaction, to allow concurrent flushes.

This affects all MongoDB storage versions.

This does not affect concurrency within a replication stream.

This does not change the storage format, although it does tweak how it is used. It remains compatible with older service versions on the same storage version.

Latency within a stream can be slightly improved by this PR: For a small commit, storage round-trips on MongoDB 8, storage v4, can be reduced from 12 to 9 in some cases. Most of this gain is from combining flush and checkpoint commit into one transaction. Bulk replication throughput (initial replication or replication catch-up) is not affected by this.

Pre-allocated ranges

A replication job now pre-allocates ranges of 64K op_ids at a time. When it hits a 16K low-water mark, it requests another range. If a single batch manages to exhausts the available ids before the low-water mark logic can kick in, it fails the transaction, requests another range, then retries. That case should be rare.

A single range is used across multiple batches in a replication job, until the range is exhausted or the job is stopped. It does not return unused ranges back to the db.

Note where we maintained a single global sequence before, different replication streams can now have oeprations "out-of-order" on this global sequence. This is not an issue - we only need to maintain ordering within each individual replication stream.

The pre-allocated ranges also allows future optimizations, since we can now allocate the op_id sequences before we starting writing to the database, allowing for better pipelining of replication work (see #797).

Fencing

Previously, we had two mechanisms to avoid consistency issues in case of multiple concurrent replication jobs on the same stream:

1. Replication stream lease (sync rules lock): This takes a lease before starting a job, and periodically refreshes it. It covers most cases, but did not have strong protection against cases like stalling jobs: A job could continue writing after its lease expired.
2. Transaction on op_id sequence: This ensures that only one job can flush changes at a time. Does not provide strong protection against out-of-order operations when multiple jobs compete.

In practice we've never observed cases where those were not sufficient, but it did not provide strong guarantees. Furthermore, the lock on the op_id sequence was global, which is what we want to remove in this PR.

So this replaces the lock on op_id sequence with "fencing" on the replication stream lease: Every write confirms that the replication job still holds the lease.

Details on the fencing implementation and the guarantees it provides are added to a doc.

This PR also contains some db operation restructuring, to ensure that the new fencing implementation does not add significant overhead in terms of db round-trips.

TODO

• Check new fields: sync_rules.writer_transaction; sync_rules.last_persisted_op: Do we need them? If yes, what does backwards-compatibility look like?
• Review the writer fence.
• Check overhead of additional db operations.
• Check transaction usage.
• Check signal usage.
• Update tests where relevant.

AI Usage

Implemented with Codex gpt-6-astra. Manually checked the approach and reviewed the changes, as well as confirming the performance impact.

[changeset-bot]

🦋 Changeset detected

Latest commit: 5b50b8a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR