From ef5734944b1f547dba5c4140959ee6fd29319d3f Mon Sep 17 00:00:00 2001 From: stevensJourney Date: Wed, 2 Sep 2026 10:36:56 +0200 Subject: [PATCH 1/8] Custom Checkpoint Request rollout docs --- client-sdks/advanced/checkpoint-requests.mdx | 2 +- handling-writes/custom-write-checkpoints.mdx | 69 ++++++++++++++------ 2 files changed, 50 insertions(+), 21 deletions(-) diff --git a/client-sdks/advanced/checkpoint-requests.mdx b/client-sdks/advanced/checkpoint-requests.mdx index 8b5d723b..c5686354 100644 --- a/client-sdks/advanced/checkpoint-requests.mdx +++ b/client-sdks/advanced/checkpoint-requests.mdx @@ -118,7 +118,7 @@ The upload response remains the authority on whether your backend accepted, chan The managed flow assumes that `uploadData()` returns only after your backend commits the uploaded changes to the source database. If your backend queues uploads for later processing, use custom checkpoint requests. This feature is available for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. -Follow the [Custom Write Checkpoints source-side setup](/handling-writes/custom-write-checkpoints#sync-rules-requirements), including its `checkpoint_requests` event definition. The `checkpoint` column stores the checkpoint request ID generated by the client. +Follow the [Custom Write Checkpoints source-side setup](/handling-writes/custom-write-checkpoints#sync-rules-requirements), including its `checkpoint_requests` event definition. The `checkpoint` column stores the checkpoint request ID generated by the client. If older app versions still use legacy Custom Write Checkpoints, follow the [rolling rollout guidance](/handling-writes/custom-write-checkpoints#rolling-out-checkpoint-requests) to preserve those records while allowing checkpoint requests to expire. The difference on the client is that the PowerSync Client SDK generates the checkpoint request ID and sends it to your backend through `CustomCheckpointRequestConnector`. diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 3af866e7..91eecf6b 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,16 +1,18 @@ --- -title: "Data Pipelines" -sidebarTitle: "Data Pipelines" -description: "Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client." +title: 'Data Pipelines' +sidebarTitle: 'Data Pipelines' +description: 'Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client.' --- -**Availability**: -Custom Write Checkpoints are available for customers on our [Team and Enterprise](https://www.powersync.com/pricing) plans. + **Availability**: Custom Write Checkpoints are available for customers on our [Team and + Enterprise](https://www.powersync.com/pricing) plans. -The alpha [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API uses the term "custom checkpoint requests" for asynchronous upload backends. Client support is currently available for Swift. This page retains the previous "Custom Write Checkpoints" name for the source-side configuration. + The alpha [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API uses the term "custom checkpoint + requests" for asynchronous upload backends. Client support is currently available for Swift. This page retains the + previous "Custom Write Checkpoints" name for the source-side configuration. To ensure [consistency](/architecture/consistency), PowerSync relies on Write Checkpoints. These checkpoints ensure that clients have uploaded their own local changes/mutations to the server before applying downloaded data from the server to the local database. @@ -21,12 +23,12 @@ The default Write Checkpoints implementation relies on uploads being acknowledge Problems occur if the persistence in the source database happens _asynchronously_. If the client's upload is meant to mutate the source database (and eventually does), but this is delayed, it will effectively seem as if the client's uploaded changes were reverted on the server, and then applied again thereafter. -Chained *data pipelines* are a common example of asynchronous uploads -- e.g. data uploads are first written to a different upstream database, or a separate queue for processing, and then finally replicated to the 'source database' (to which PowerSync is connected). +Chained _data pipelines_ are a common example of asynchronous uploads -- e.g. data uploads are first written to a different upstream database, or a separate queue for processing, and then finally replicated to the 'source database' (to which PowerSync is connected). For example, consider the following data pipeline: 1. The client makes a change locally and the local database is updated. -2. The client uploads this change to the server. +2. The client uploads this change to the server. 3. The server resolves the request and writes the change into an intermediate database (not the source database yet). 4. The client thinks the upload is complete (i.e. persisted into the source database). It requests a Write Checkpoint from the PowerSync Service. 5. The PowerSync Service increments the replication `HEAD` in the source database, and creates a Write Checkpoint for the client. The Write Checkpoint number is returned and recorded in the client. @@ -49,8 +51,11 @@ The PowerSync Service then needs to process the (ordered) replication events and A self-hosted Node.js demo with Postgres is available here: - - + ## Implementation Details @@ -93,9 +98,16 @@ create publication powersync for table public.lists, public.todos, public.checkp ### Sync Rules Requirements -For clients using the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API, enable the `checkpoint_requests` event in your sync configuration. This event maps rows from the `checkpoints` table to the `CheckpointPayload` payload. +With [storage version 4](/sync/advanced/compatibility#storage-version), each sync configuration can use only one event definition to produce custom checkpoints. Choose the event based on the clients that the configuration supports: + +- Use `checkpoint_requests` to support the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API. These records are temporary and can expire. +- Use `write_checkpoints` only if every client uses the legacy Custom Write Checkpoints flow. These records are retained because legacy clients do not retry expired checkpoints automatically. + +For `checkpoint_requests`, the `is_legacy` field is optional. When it is omitted, PowerSync treats the payload as a checkpoint request that can expire. If the configuration supports only Checkpoint Requests, omit it: ```yaml +# sync-rules.yaml + event_definitions: # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. checkpoint_requests: @@ -104,23 +116,40 @@ event_definitions: - SELECT user_id, checkpoint, client_id FROM checkpoints ``` -Use the `write_checkpoints` event only for clients using the legacy Custom Write Checkpoints flow: +For a configuration that supports only legacy clients, use `write_checkpoints` without an `is_legacy` field: ```yaml # sync-rules.yaml -# Register the custom write_checkpoints event event_definitions: + # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. write_checkpoints: payloads: - # This defines where the replicated Custom Write Checkpoints should be extracted from + # This defines where the replicated legacy Custom Write Checkpoints should be extracted from - SELECT user_id, checkpoint, client_id FROM checkpoints +``` + +#### Rolling Out Checkpoint Requests + +During a rolling rollout, older app versions may continue to create legacy checkpoints while newer versions create checkpoint requests. Configure only the `checkpoint_requests` event and mark the legacy payloads so the PowerSync Service can apply the correct retention behavior: + +- Set `is_legacy` to `true` for legacy checkpoint records. These records cannot expire because older clients do not retry them automatically. +- Omit `is_legacy` from checkpoint request records. The PowerSync Service can expire and delete these temporary records. -# Define Sync Rules as usual -bucket_definitions: - global: - data: - ... +You can store both record types in one table or in separate tables. We recommend separate tables because they keep the different retention behavior explicit. Mark only the legacy table's payload: + +```yaml +# sync-rules.yaml + +event_definitions: + # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. + checkpoint_requests: + payloads: + # Legacy Custom Write Checkpoints must be retained + - SELECT user_id, checkpoint, client_id, true AS is_legacy + FROM legacy_checkpoints + # Checkpoint Requests can expire, so is_legacy is omitted + - SELECT user_id, checkpoint, client_id FROM checkpoint_requests ``` ### Application @@ -147,7 +176,7 @@ Your client backend connector should make a call to the application backend to c async function getCheckpoint(clientId: string): string { /** - * Should perform a request to the application backend which should create the + * Should perform a request to the application backend which should create the * Write Checkpoint record and return the corresponding checkpoint number. */ return "the Write Checkpoint number from the request"; From 5adebfa532fac75ab606b332292054555ddda19c Mon Sep 17 00:00:00 2001 From: stevensJourney <51082125+stevensJourney@users.noreply.github.com> Date: Wed, 9 Sep 2026 16:15:13 +0200 Subject: [PATCH 2/8] Update handling-writes/custom-write-checkpoints.mdx Co-authored-by: benitav --- handling-writes/custom-write-checkpoints.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 91eecf6b..08a18c49 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,5 +1,5 @@ --- -title: 'Data Pipelines' +title: "Data Pipelines" sidebarTitle: 'Data Pipelines' description: 'Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client.' --- From 62b6819780d9b8d97a07f9eaf03e28d7da4ec6a4 Mon Sep 17 00:00:00 2001 From: stevensJourney <51082125+stevensJourney@users.noreply.github.com> Date: Wed, 9 Sep 2026 16:15:23 +0200 Subject: [PATCH 3/8] Update handling-writes/custom-write-checkpoints.mdx Co-authored-by: benitav --- handling-writes/custom-write-checkpoints.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 08a18c49..652c1a23 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,6 +1,6 @@ --- title: "Data Pipelines" -sidebarTitle: 'Data Pipelines' +sidebarTitle: "Data Pipelines" description: 'Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client.' --- From e6a34dfb96bcbb510887828020f0957875f3d149 Mon Sep 17 00:00:00 2001 From: stevensJourney <51082125+stevensJourney@users.noreply.github.com> Date: Wed, 9 Sep 2026 16:15:32 +0200 Subject: [PATCH 4/8] Update handling-writes/custom-write-checkpoints.mdx Co-authored-by: benitav --- handling-writes/custom-write-checkpoints.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 652c1a23..00e7e7b9 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,7 +1,7 @@ --- title: "Data Pipelines" sidebarTitle: "Data Pipelines" -description: 'Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client.' +description: "Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client." --- From 54a5c1f99b633a9eda8ef047f52d599b5c8f7284 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 11:05:53 +0200 Subject: [PATCH 5/8] Mention that postgres bucket storage doesn't support V4 --- .../self-hosted-instances.mdx | 4 ++++ sync/advanced/compatibility.mdx | 2 ++ sync/advanced/storage-version-4.mdx | 22 ++++++++++++++----- 3 files changed, 23 insertions(+), 5 deletions(-) diff --git a/configuration/powersync-service/self-hosted-instances.mdx b/configuration/powersync-service/self-hosted-instances.mdx index 2e57d460..d86ff020 100644 --- a/configuration/powersync-service/self-hosted-instances.mdx +++ b/configuration/powersync-service/self-hosted-instances.mdx @@ -467,6 +467,10 @@ The PowerSync Service requires a storage database to store the data and metadata Storage version to use when deploying a Sync Config that does not specify a storage version. To opt in to [storage version 4](/sync/advanced/storage-version-4) (Beta), set `config.storage_version: 4` in your Sync Config. An explicit `config.storage_version` takes precedence over this default. Available since Service version 1.26.0. + + + Postgres bucket storage does not currently support version 4. + #### MongoDB Storage diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index 98517bca..444e6e9a 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -86,6 +86,8 @@ Set `storage_version` when you need to: ### Available Versions +All PowerSync Cloud instances use MongoDB bucket storage, so they are compatible with all available storage versions. Self-hosted instances with Postgres bucket storage can use versions 1 and 2 only. + | Version | Bucket storage | Status | | --- | --- | --- | | `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 0521834d..56d5922b 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -13,7 +13,9 @@ Compared to version 2, it provides: ## Availability -The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). +Storage version 4 is compatible with all PowerSync Cloud instances, which already use MongoDB [bucket storage](/architecture/powersync-service#bucket-storage). Self-hosted instances must also use MongoDB bucket storage. Postgres bucket storage is not currently supported. + +The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it will become the default for all supported instances. S3 object storage is then also enabled on all PowerSync Cloud instances. For self-hosted deployments, follow the [S3 setup instructions](#self-hosted-s3-setup). | | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | | --- | --- | --- | --- | --- | @@ -21,8 +23,6 @@ The PowerSync Cloud and self-hosted columns below apply during the Beta only. On | Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | | S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | -Storage version 4 requires MongoDB as the [bucket storage database](/architecture/powersync-service#bucket-storage). PowerSync Cloud always uses MongoDB. Self-hosted deployments with Postgres bucket storage cannot use version 4. - Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). ## Opt In @@ -53,6 +53,10 @@ streams: ### Self-Hosted + + Postgres bucket storage is not supported with version 4. + + Add `storage_version: 4` to the `config` block of each Sync Config as shown above, then deploy or redeploy it to use version 4. To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. @@ -61,7 +65,11 @@ To also enable S3 object storage, follow the [self-hosted S3 setup instructions] ## Incremental Reprocessing -Incremental reprocessing requires a MongoDB source database, Sync Streams, and storage version 4. It is active whenever all three apply. +Incremental reprocessing is active when you use a MongoDB source database, Sync Streams, and storage version 4. + + + Self-hosted instances with Postgres bucket storage are not supported. + Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. @@ -111,7 +119,7 @@ For self-hosted instances, offloading bucket data to S3 can reduce storage and d Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. -S3 object storage requires MongoDB bucket storage and storage version 4. It works with Sync Streams and legacy Sync Rules. +S3 object storage requires storage version 4 and works with Sync Streams and legacy Sync Rules. It is compatible with all PowerSync Cloud instances. S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). @@ -123,6 +131,10 @@ During the Beta, PowerSync enables S3 object storage per instance. [Contact us]( ### Self-Hosted S3 Setup + + Self-hosted instances with Postgres bucket storage are not supported. + + Create a bucket. Use the same region as the PowerSync Service where possible, to keep latency low and avoid cross-region data transfer charges. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. From 25f54398bc1a7300ee4406c38a4541fb9a11af4b Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 12:50:33 +0200 Subject: [PATCH 6/8] Restructure the Data Pipelines page according to writing standards --- client-sdks/advanced/checkpoint-requests.mdx | 8 +- handling-writes/custom-write-checkpoints.mdx | 357 +++++++++---------- 2 files changed, 171 insertions(+), 194 deletions(-) diff --git a/client-sdks/advanced/checkpoint-requests.mdx b/client-sdks/advanced/checkpoint-requests.mdx index 9e7bcf95..fa00976e 100644 --- a/client-sdks/advanced/checkpoint-requests.mdx +++ b/client-sdks/advanced/checkpoint-requests.mdx @@ -235,9 +235,13 @@ The upload response remains the authority on whether your backend accepted, chan ## Asynchronous Upload Backends -The managed flow assumes that `uploadData()` returns only after your backend commits the uploaded changes to the source database. If your backend queues uploads for later processing, use custom checkpoint requests. This feature is available for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. + + This feature is available for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. + + +The managed flow assumes that `uploadData()` returns only after your backend commits the uploaded changes to the source database. If your backend queues uploads for later processing, use custom checkpoint requests. -Follow the [Custom Write Checkpoints source-side setup](/handling-writes/custom-write-checkpoints#sync-rules-requirements), including its `checkpoint_requests` event definition. The `checkpoint` column stores the checkpoint request ID generated by the client. If older app versions still use legacy Custom Write Checkpoints, follow the [rolling rollout guidance](/handling-writes/custom-write-checkpoints#rolling-out-checkpoint-requests) to preserve those records while allowing checkpoint requests to expire. +Follow [Setting Up Custom Checkpoint Requests](/handling-writes/custom-write-checkpoints#setting-up-custom-checkpoint-requests) for the source database, Sync Config, and backend changes. If older app versions still use legacy Custom Write Checkpoints, see [Migrating to Custom Checkpoint Requests](/handling-writes/custom-write-checkpoints#migrating-to-custom-checkpoint-requests). The difference on the client is that the PowerSync Client SDK generates the checkpoint request ID and sends it to your backend through `CustomCheckpointRequestConnector`. diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 00e7e7b9..9e6bd8d8 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -1,246 +1,219 @@ --- title: "Data Pipelines" sidebarTitle: "Data Pipelines" -description: "Use Custom Write Checkpoints to track asynchronous data uploads through chained data pipelines and confirm write completion on the client." +description: "Use custom checkpoints to keep client data stable when your backend applies uploads to the source database asynchronously." --- - **Availability**: Custom Write Checkpoints are available for customers on our [Team and - Enterprise](https://www.powersync.com/pricing) plans. +**Availability**: Custom checkpoints are available for customers on our [Team and Enterprise](https://www.powersync.com/pricing) plans. - - The alpha [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API uses the term "custom checkpoint - requests" for asynchronous upload backends. Client support is currently available for Swift. This page retains the - previous "Custom Write Checkpoints" name for the source-side configuration. - +PowerSync uses write checkpoints to keep client data [consistent](/architecture/consistency). After the client uploads its local writes, it obtains a checkpoint that marks the source database position after those writes. The client applies downloaded data only once the sync checkpoint includes that write checkpoint. This is why your [write endpoint must be synchronous](/handling-writes/writing-client-changes): the default checkpoint marks the source database position when `uploadData()` returns, so the uploaded changes must already be in the source database at that moment. -To ensure [consistency](/architecture/consistency), PowerSync relies on Write Checkpoints. These checkpoints ensure that clients have uploaded their own local changes/mutations to the server before applying downloaded data from the server to the local database. +Some backends cannot process uploads synchronously. In a chained data pipeline, uploads first go to a queue or an intermediate database and reach the source database later. With the default checkpoints, this makes client data flicker: -The essential requirement is that the client must get a Write Checkpoint after uploading its last write/mutation. Then, when downloading data from the server, the client checks whether the Write Checkpoint is part of the largest [sync checkpoint](https://github.com/powersync-ja/powersync-service/blob/main/docs/specs/sync-protocol.md) received from the server (i.e. from the PowerSync Service). If it is, the client applies the server-side state to the local database. +1. The client uploads a change. Your backend accepts it and queues it, and `uploadData()` returns. +2. The client obtains a write checkpoint. The PowerSync Service marks the current source database position, which does not include the queued change. +3. The client receives that checkpoint and applies the server state. The change is missing, so the client reverts it locally. +4. The pipeline writes the change to the source database. The Service syncs it, and the client applies it again. -The default Write Checkpoints implementation relies on uploads being acknowledged _synchronously_, i.e. the change persists in the source database (to which PowerSync is connected) before the [`uploadData` call](/configuration/app-backend/client-side-integration) completes. +Custom checkpoints solve this. Instead of the Service marking the source database position when the client asks, your backend writes a checkpoint record into a table in the source database at the end of the pipeline. The record replicates to the Service through the same replication stream as your data, so the checkpoint always follows the uploaded changes. An event definition in your Sync Config tells the Service how to read the record. -Problems occur if the persistence in the source database happens _asynchronously_. If the client's upload is meant to mutate the source database (and eventually does), but this is delayed, it will effectively seem as if the client's uploaded changes were reverted on the server, and then applied again thereafter. +## Choosing a Flow -Chained _data pipelines_ are a common example of asynchronous uploads -- e.g. data uploads are first written to a different upstream database, or a separate queue for processing, and then finally replicated to the 'source database' (to which PowerSync is connected). +PowerSync supports two custom checkpoint flows. Use custom checkpoint requests for new implementations. -For example, consider the following data pipeline: +| | Custom checkpoint requests | Legacy Custom Write Checkpoints | +| --- | --- | --- | +| Checkpoint ID | Generated by the PowerSync Client SDK | Generated by your backend | +| Client integration | Implement `postCheckpointRequest()` on your connector. The SDK calls it after each upload. | Pass the checkpoint number to `transaction.complete()` in `uploadData()`. | +| Event definition | `checkpoint_requests` | `write_checkpoints` | +| Records in the Service | Expire after a retention period | Retained | +| Status | Alpha. Requires PowerSync Service 1.24.0 or later. | Stable | -1. The client makes a change locally and the local database is updated. -2. The client uploads this change to the server. -3. The server resolves the request and writes the change into an intermediate database (not the source database yet). -4. The client thinks the upload is complete (i.e. persisted into the source database). It requests a Write Checkpoint from the PowerSync Service. -5. The PowerSync Service increments the replication `HEAD` in the source database, and creates a Write Checkpoint for the client. The Write Checkpoint number is returned and recorded in the client. -6. The PowerSync Service replicates past the previous replication `HEAD` (but the changes are still not present in the source database). -7. It should be fine for the client to apply the state of the server to the local database. But the server state does not include the client's uploaded changes mentioned in #2. This is the same as if the client's uploaded changes were rejected (not applied) by the server. This results in the client reverting the changes in its local database. -8. Eventually the change is written to the source database, and increments the replication `HEAD`. -9. The PowerSync Service replicates this change and sends it to the client. The client then reapplies the changes to its local database. +Custom checkpoint requests are part of the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API. See that page for the supported client SDKs. The legacy flow continues to work and is documented in [Legacy Custom Write Checkpoints](#legacy-custom-write-checkpoints). -In the above case, the client may see the Write Checkpoint before the data has been replicated. This will cause the client to revert its changes, then apply them again later when it has actually replicated, causing data to "flicker" in the app. +A Sync Config can define only one of the two events. If you have app versions in production that use the legacy flow, see [Migrating to Custom Checkpoint Requests](#migrating-to-custom-checkpoint-requests). -For these use cases, Custom Write Checkpoints should be implemented. +## Setting Up Custom Checkpoint Requests -## Custom Write Checkpoints +With custom checkpoint requests, the PowerSync Client SDK generates an increasing checkpoint request ID and sends it to your backend after each upload. Your backend writes the ID into a checkpoints table in the source database. When the Service replicates the record, the client knows that its uploads are in the source database. -_Custom Write Checkpoints_ allow the developer to define Write Checkpoints and insert them into the replication stream directly, instead of relying on the PowerSync Service to create and return them. An example of this is having the backend persist Write Checkpoints to a dedicated table which is processed as part of the replication stream. + + + Create a table in your source database that stores the latest checkpoint request ID for each PowerSync client: -The PowerSync Service then needs to process the (ordered) replication events and correlate the checkpoint table changes to Write Checkpoint events. + ```sql + CREATE TABLE checkpoints ( + user_id TEXT NOT NULL, + client_id TEXT NOT NULL, + checkpoint BIGINT NOT NULL, + PRIMARY KEY (user_id, client_id) + ); + ``` -## Example Implementation + - `user_id` is the authenticated user. + - `client_id` is the PowerSync client ID. Each local database has its own client ID, so one user can have many clients. + - `checkpoint` is the checkpoint request ID. IDs are 64-bit integers, so use a `BIGINT` or equivalent column. -A self-hosted Node.js demo with Postgres is available here: + Column names can differ. The event definition in the next steps maps your columns to these fields. + + + For Postgres, add the table to the PowerSync [publication](/configuration/source-db/setup): - + ```sql + CREATE PUBLICATION powersync FOR TABLE lists, todos, checkpoints; + ``` -## Implementation Details + For other source databases, the Service replicates every table that your Sync Config references, including tables in event definitions. Complete the same [table setup](/configuration/source-db/setup) as for your other tables, such as enabling CDC for a SQL Server table. + + + Add a `checkpoint_requests` event definition to your Sync Config. Its payload query must return the fields `user_id`, `client_id`, and `checkpoint`: -This outlines what a Custom Write Checkpoints implementation entails. + ```yaml + config: + edition: 3 -### Custom Write Checkpoint Table + event_definitions: + checkpoint_requests: + payloads: + - SELECT user_id, client_id, checkpoint FROM checkpoints -Create a dedicated `checkpoints` table, which should contain the following checkpoint payload information in some form: + streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() + ``` -```TypeScript -export type CheckpointPayload = { - /** - * The user account id - */ - user_id: string; - /** - * The client id relating to the user account. - * A single user can have multiple clients. - * A client is analogous to a device session. - * Checkpoints are tracked separately for each `user_id` + `client_id`. - */ - client_id: string; - /** - * A strictly increasing Write Checkpoint identifier. - * This number is generated by the application backend. - */ - checkpoint: bigint; -} -``` + Use aliases if your column names differ, for example `SELECT owner AS user_id, device AS client_id, request_id AS checkpoint FROM checkpoints`. + + + Add an endpoint that receives the client ID and checkpoint request ID from the client. Take the user ID from your session or token. The endpoint must: -### Replication Requirements + 1. Store the greater of the submitted ID and the stored ID for that user and client. + 2. Return that value. If the submitted ID was stale, the client uses the returned ID to continue counting from there. -Replication events for the Custom Write Checkpoint table (`checkpoints` in this example) need to enabled. + Write the record through the same pipeline as the uploads, so that it reaches the source database after the changes it confirms. If your backend writes the checkpoint record directly while the uploads are still queued, the client sees the checkpoint before its changes and reverts them. -For Postgres, this involves adding the table to the [PowerSync logical replication publication](/configuration/source-db/setup), for example: + For Postgres, one statement handles both new and existing rows: -```SQL -create publication powersync for table public.lists, public.todos, public.checkpoints; -``` + ```sql + INSERT INTO checkpoints (user_id, client_id, checkpoint) + VALUES ($1, $2, $3) + ON CONFLICT (user_id, client_id) DO UPDATE + SET checkpoint = GREATEST(checkpoints.checkpoint, EXCLUDED.checkpoint) + RETURNING checkpoint; + ``` + + Return the value as a string in JSON to avoid precision loss in JavaScript clients. See [Checkpoint Request IDs](/client-sdks/advanced/checkpoint-requests#checkpoint-request-ids) for the full reconciliation rules. + + + Connect with checkpoint requests enabled and add `postCheckpointRequest()` to your backend connector to call your endpoint. The SDK calls this method after each upload, so `uploadData()` needs no checkpoint handling of its own. See [Prerequisites](/client-sdks/advanced/checkpoint-requests#prerequisites) and [Connector Changes](/client-sdks/advanced/checkpoint-requests#connector-changes) for how to declare the method in each SDK. + + + +### Record Retention + +The Service keeps a replicated checkpoint request for `checkpoint_request_retention_minutes` after it stores the record. The default is 60 minutes. The next compact job then removes it. Clients send their current request ID again when they reconnect, so an expired record is recreated when a client still needs it. On self-hosted instances, you can change the period with [`api.parameters.checkpoint_request_retention_minutes`](/configuration/powersync-service/self-hosted-instances#param-checkpoint-request-retention-minutes). + +Retention applies only to the Service's copy. Rows in your checkpoints table are yours to keep or delete. While a row exists, return its value from your endpoint so that a reconnecting client can resume from it. + +## Legacy Custom Write Checkpoints + +In the legacy flow, your backend generates an increasing checkpoint number for each client, and the client passes that number to `transaction.complete()` after each upload. The Service retains these records because legacy clients wait for a specific number and do not request it again. + + + + Use the same table and replication setup as for [custom checkpoint requests](#setting-up-custom-checkpoint-requests). + + + Add a `write_checkpoints` event definition to your Sync Config: + + ```yaml + config: + edition: 3 + + event_definitions: + write_checkpoints: + payloads: + - SELECT user_id, client_id, checkpoint FROM checkpoints + + streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() + ``` + + + Add an endpoint that increments and returns the checkpoint number for the user and client. Write the record through the same pipeline as the uploads. For Postgres: + + ```sql + INSERT INTO checkpoints (user_id, client_id, checkpoint) + VALUES ($1, $2, 1) + ON CONFLICT (user_id, client_id) DO UPDATE + SET checkpoint = checkpoints.checkpoint + 1 + RETURNING checkpoint; + ``` + + + In `uploadData()`, request a checkpoint from your backend after uploading the transaction and pass it to `complete()`: + + ```typescript + async function uploadData(database: CommonPowerSyncDatabase): Promise { + const transaction = await database.getNextCrudTransaction(); + if (!transaction) { + return; + } -### Sync Rules Requirements + for (const operation of transaction.crud) { + // Upload the operation to your backend + } -With [storage version 4](/sync/advanced/compatibility#storage-version), each sync configuration can use only one event definition to produce custom checkpoints. Choose the event based on the clients that the configuration supports: + const clientId = await database.getClientId(); + const checkpoint = await requestWriteCheckpoint(clientId); + await transaction.complete(checkpoint); + } -- Use `checkpoint_requests` to support the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API. These records are temporary and can expire. -- Use `write_checkpoints` only if every client uses the legacy Custom Write Checkpoints flow. These records are retained because legacy clients do not retry expired checkpoints automatically. + async function requestWriteCheckpoint(clientId: string): Promise { + // Call your backend endpoint. It creates the checkpoint record + // and returns the new checkpoint number as a string. + } + ``` + + -For `checkpoint_requests`, the `is_legacy` field is optional. When it is omitted, PowerSync treats the payload as a checkpoint request that can expire. If the configuration supports only Checkpoint Requests, omit it: +## Migrating to Custom Checkpoint Requests -```yaml -# sync-rules.yaml +While you roll out an updated app version, older versions that still use the legacy flow write legacy checkpoint numbers while updated versions write checkpoint request IDs. Because a Sync Config cannot define both `write_checkpoints` and `checkpoint_requests`, support both kinds of records by defining only `checkpoint_requests` and adding an `is_legacy` field to the payload: -event_definitions: - # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. - checkpoint_requests: - payloads: - # This defines where the replicated custom Write Checkpoints should be extracted from - - SELECT user_id, checkpoint, client_id FROM checkpoints -``` +- Set `is_legacy` to `true` for legacy records. The Service retains them. +- Omit `is_legacy`, or set it to `false`, for checkpoint request records. The Service can expire them. -For a configuration that supports only legacy clients, use `write_checkpoints` without an `is_legacy` field: +{/* TODO: Before publishing, confirm (1) the PowerSync Service version from which only one checkpoint event per Sync Config is allowed, and (2) what a developer observes when both are defined. The restriction comes with incremental custom checkpoints in Service 1.26.0 and the custom checkpoints module update (journeyapps-platform/powersync PR 401, merged 2026-09-01), which throws an assertion error when both events are present. */} -```yaml -# sync-rules.yaml +Keep both record types in separate tables where possible. Separate tables make the difference in retention visible in the Sync Config: +```yaml event_definitions: - # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. - write_checkpoints: + checkpoint_requests: payloads: - # This defines where the replicated legacy Custom Write Checkpoints should be extracted from - - SELECT user_id, checkpoint, client_id FROM checkpoints + # Legacy checkpoints must be retained + - SELECT user_id, client_id, checkpoint, true AS is_legacy FROM legacy_checkpoints + # Checkpoint requests can expire + - SELECT user_id, client_id, checkpoint FROM checkpoint_requests ``` -#### Rolling Out Checkpoint Requests - -During a rolling rollout, older app versions may continue to create legacy checkpoints while newer versions create checkpoint requests. Configure only the `checkpoint_requests` event and mark the legacy payloads so the PowerSync Service can apply the correct retention behavior: - -- Set `is_legacy` to `true` for legacy checkpoint records. These records cannot expire because older clients do not retry them automatically. -- Omit `is_legacy` from checkpoint request records. The PowerSync Service can expire and delete these temporary records. - -You can store both record types in one table or in separate tables. We recommend separate tables because they keep the different retention behavior explicit. Mark only the legacy table's payload: +If both flows write to one table, distinguish the rows with a column. For example, your checkpoint request endpoint can set a `checkpoint_requested_at` timestamp that the legacy endpoint leaves `NULL`: ```yaml -# sync-rules.yaml - event_definitions: - # Note this event is only supported for customers on [Team and Enterprise](https://www.powersync.com/pricing) plans. checkpoint_requests: payloads: - # Legacy Custom Write Checkpoints must be retained - - SELECT user_id, checkpoint, client_id, true AS is_legacy - FROM legacy_checkpoints - # Checkpoint Requests can expire, so is_legacy is omitted - - SELECT user_id, checkpoint, client_id FROM checkpoint_requests + - SELECT user_id, client_id, checkpoint, checkpoint_requested_at IS NULL AS is_legacy FROM checkpoints ``` -### Application +Once no clients use the legacy flow, remove the legacy payload or the `is_legacy` field. With [storage version 4](/sync/advanced/storage-version-4#incremental-reprocessing), changing an event definition reprocesses only that event's data. -Your application should handle Custom Write Checkpoints on both the frontend and backend. - -#### Frontend - -Your client backend connector should make a call to the application backend to create a Custom Write Checkpoint record after uploading items in the `uploadData` method. The Write Checkpoint number should be supplied to the CRUD transactions' `complete` method. - -```TypeScript - async function uploadData(database: CommonPowerSyncDatabase): Promise { - const transaction = await database.getNextCrudTransaction(); - // Get the unique client ID from the PowerSync Database SQLite storage - const clientId = await db.getClientId(); - - for (const operation of transaction.crud) { - // Upload the items to application backend - // .... - } - - await transaction.complete(await getCheckpoint(clientId)); - } - - async function getCheckpoint(clientId: string): string { - /** - * Should perform a request to the application backend which should create the - * Write Checkpoint record and return the corresponding checkpoint number. - */ - return "the Write Checkpoint number from the request"; - } -``` - -#### Backend - -The backend should create a Write Checkpoint record when the client requests it. The record should automatically increment the Write Checkpoint number for the associated `user_id` and `client_id`. - -#### Postgres Example - -With the following table defined in the database... - -```SQL -CREATE TABLE checkpoints ( - user_id VARCHAR(255), - client_id VARCHAR(255), - checkpoint INTEGER, - PRIMARY KEY (user_id, client_id) -); -``` - -...the backend should have a route which creates `checkpoints` records: - -```TypeScript -router.put('/checkpoint', async (req, res) => { - if (!req.body) { - res.status(400).send({ - message: 'Invalid body provided' - }); - return; - } - - const client = await pool.connect(); - -// These could be obtained from the session - const { user_id = 'UserID', client_id = '1' } = req.body; - - const response = await client.query( - ` - INSERT - INTO - checkpoints - (user_id, client_id, checkpoint) - VALUES - ($1, $2, '1') - ON - CONFLICT (user_id, client_id) - DO - UPDATE - SET checkpoint = checkpoints.checkpoint + 1 - RETURNING checkpoint; - `, - [user_id, client_id] - ); - client.release(); - - // Return the Write Checkpoint number - res.status(200).send({ - checkpoint: response.rows[0].checkpoint - }); -}); - -``` +## Example Implementations -An example implementation can be seen in the [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/api/data.js), including examples for [MongoDB](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/persistance/mongo/mongo-persistance.js) and [MySQL](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo/blob/main/src/persistance/mysql/mysql-persistance.js). +- [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo): implements both a checkpoint request endpoint and a legacy checkpoint endpoint, with Postgres, MongoDB, and MySQL persistence. +- [Swift custom checkpoint demo](https://github.com/powersync-ja/powersync-swift/tree/main/Demos/CustomCheckpointDemo): a client that uses custom checkpoint requests with the Node.js backend demo. +- [Self-hosted custom checkpoints demo](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-custom-checkpoints): a Docker Compose setup with Postgres that uses the legacy flow. From 9dbfd8b73d9a04981f1bf951fff329dd3e1cd079 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 12:56:15 +0200 Subject: [PATCH 7/8] Fix todos --- handling-writes/custom-write-checkpoints.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/handling-writes/custom-write-checkpoints.mdx b/handling-writes/custom-write-checkpoints.mdx index 9e6bd8d8..106eec80 100644 --- a/handling-writes/custom-write-checkpoints.mdx +++ b/handling-writes/custom-write-checkpoints.mdx @@ -33,7 +33,7 @@ PowerSync supports two custom checkpoint flows. Use custom checkpoint requests f Custom checkpoint requests are part of the [Checkpoint Requests](/client-sdks/advanced/checkpoint-requests) API. See that page for the supported client SDKs. The legacy flow continues to work and is documented in [Legacy Custom Write Checkpoints](#legacy-custom-write-checkpoints). -A Sync Config can define only one of the two events. If you have app versions in production that use the legacy flow, see [Migrating to Custom Checkpoint Requests](#migrating-to-custom-checkpoint-requests). +From PowerSync Service 1.26.0, a Sync Config can define only one of the two events. If you have app versions in production that use the legacy flow, see [Migrating to Custom Checkpoint Requests](#migrating-to-custom-checkpoint-requests). ## Setting Up Custom Checkpoint Requests @@ -182,13 +182,11 @@ In the legacy flow, your backend generates an increasing checkpoint number for e ## Migrating to Custom Checkpoint Requests -While you roll out an updated app version, older versions that still use the legacy flow write legacy checkpoint numbers while updated versions write checkpoint request IDs. Because a Sync Config cannot define both `write_checkpoints` and `checkpoint_requests`, support both kinds of records by defining only `checkpoint_requests` and adding an `is_legacy` field to the payload: +While you roll out an updated app version, older versions that still use the legacy flow write legacy checkpoint numbers while updated versions write checkpoint request IDs. Because a Sync Config cannot define both `write_checkpoints` and `checkpoint_requests`, support both kinds of records by defining only `checkpoint_requests` and adding an `is_legacy` field to the payload. The field is available since Service version 1.26.0. - Set `is_legacy` to `true` for legacy records. The Service retains them. - Omit `is_legacy`, or set it to `false`, for checkpoint request records. The Service can expire them. -{/* TODO: Before publishing, confirm (1) the PowerSync Service version from which only one checkpoint event per Sync Config is allowed, and (2) what a developer observes when both are defined. The restriction comes with incremental custom checkpoints in Service 1.26.0 and the custom checkpoints module update (journeyapps-platform/powersync PR 401, merged 2026-09-01), which throws an assertion error when both events are present. */} - Keep both record types in separate tables where possible. Separate tables make the difference in retention visible in the Sync Config: ```yaml @@ -214,6 +212,6 @@ Once no clients use the legacy flow, remove the legacy payload or the `is_legacy ## Example Implementations -- [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo): implements both a checkpoint request endpoint and a legacy checkpoint endpoint, with Postgres, MongoDB, and MySQL persistence. - [Swift custom checkpoint demo](https://github.com/powersync-ja/powersync-swift/tree/main/Demos/CustomCheckpointDemo): a client that uses custom checkpoint requests with the Node.js backend demo. +- [Node.js backend demo](https://github.com/powersync-ja/powersync-nodejs-backend-todolist-demo): implements both a checkpoint request endpoint and a legacy checkpoint endpoint, with Postgres, MongoDB, and MySQL persistence. - [Self-hosted custom checkpoints demo](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-custom-checkpoints): a Docker Compose setup with Postgres that uses the legacy flow. From 928bd6917e980f71672141d2cf2cc364cbadea51 Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 13:02:21 +0200 Subject: [PATCH 8/8] Add SDK version numbers for checkpoint requests --- client-sdks/advanced/checkpoint-requests.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/client-sdks/advanced/checkpoint-requests.mdx b/client-sdks/advanced/checkpoint-requests.mdx index 9da6ffab..bbd613dd 100644 --- a/client-sdks/advanced/checkpoint-requests.mdx +++ b/client-sdks/advanced/checkpoint-requests.mdx @@ -42,10 +42,13 @@ Before creating a checkpoint request: ```typescript JavaScript/TypeScript +// Requires @powersync/web 2.3.0, @powersync/react-native 2.2.0, +// @powersync/node 1.0.0, or @powersync/capacitor 0.9.0 or later await db.connect(connector, { checkpointMode: 'requests' }); ``` ```dart Dart +// Requires powersync 2.4.0 or later await db.connect( connector: connector, options: SyncOptions(checkpointMode: .requests()), @@ -53,6 +56,7 @@ await db.connect( ``` ```kotlin Kotlin +// Requires PowerSync Kotlin SDK 1.15.0 or later database.connect( connector, options = SyncOptions( @@ -62,6 +66,7 @@ database.connect( ``` ```swift Swift +// Requires PowerSync Swift SDK 1.16.0 or later try await database.connect( connector: connector, options: ConnectOptions(checkpointMode: .requests()) @@ -69,9 +74,10 @@ try await database.connect( ``` ```cs .NET +// Requires PowerSync.Common 0.1.5 or later await database.Connect( connector: connector, - options: new PowerSyncConnectOptions(checkpointMode: new CheckpointMode.Requests()) + options: new PowerSyncConnectionOptions(checkpointMode: new CheckpointMode.Requests()) ); ```