From 7f40fdd803411379073843852ce58447cf11baca Mon Sep 17 00:00:00 2001 From: Benita Volkmann Date: Thu, 10 Sep 2026 15:10:49 +0200 Subject: [PATCH] New Sync Streams nav --- .../app-backend/client-side-integration.mdx | 2 +- .../source-db/postgres-maintenance.mdx | 2 +- debugging/error-codes.mdx | 2 +- docs.json | 61 ++-- snippets/binary-type.mdx | 2 +- sync/advanced/overview.mdx | 2 +- sync/advanced/storage-version-4.mdx | 2 +- sync/grammar/sync-streams/index.mdx | 2 +- sync/overview.mdx | 90 ------ sync/rules/client-parameters.mdx | 2 +- .../migrate-to-sync-streams.mdx} | 42 +-- sync/rules/overview.mdx | 2 +- sync/streams/client-usage.mdx | 6 +- sync/streams/examples.mdx | 2 +- sync/streams/overview.mdx | 297 +++--------------- sync/streams/quickstart.mdx | 276 ++++++++++++++++ 16 files changed, 374 insertions(+), 418 deletions(-) delete mode 100644 sync/overview.mdx rename sync/{streams/migration.mdx => rules/migrate-to-sync-streams.mdx} (69%) create mode 100644 sync/streams/quickstart.mdx diff --git a/configuration/app-backend/client-side-integration.mdx b/configuration/app-backend/client-side-integration.mdx index 8768f56b6..c8c76618c 100644 --- a/configuration/app-backend/client-side-integration.mdx +++ b/configuration/app-backend/client-side-integration.mdx @@ -10,7 +10,7 @@ After you've [instantiated](/intro/setup-guide#instantiate-the-powersync-databas | Purpose | Description | |---------|-------------| -| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams or Sync Rules (legacy)](/sync/overview), and the _client-to-server write path_ goes via your backend. | +| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), and the _client-to-server write path_ goes via your backend. | | **Authentication integration:** (optional) | PowerSync uses JWTs for authentication between the Client SDK and PowerSync Service. Some [authentication providers](/configuration/auth/overview#common-authentication-providers) generate JWTs for users which PowerSync can verify directly. For others, some code must be [added to your application backend](/configuration/auth/custom) to generate the JWTs. | diff --git a/configuration/source-db/postgres-maintenance.mdx b/configuration/source-db/postgres-maintenance.mdx index f5ea3e3ef..263486cc4 100644 --- a/configuration/source-db/postgres-maintenance.mdx +++ b/configuration/source-db/postgres-maintenance.mdx @@ -7,7 +7,7 @@ description: "Manage Postgres replication slots and WAL lag for reliable PowerSy Postgres logical replication slots are used to keep track of [replication](/architecture/powersync-service#replication-from-the-source-database) progress (recorded as a [LSN](https://www.postgresql.org/docs/current/datatype-pg-lsn.html)). -Every time a new version of [Sync Streams or Sync Rules](/sync/overview) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. +Every time a new version of [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) is deployed, PowerSync creates a new replication slot. Once the new version is fully processed, PowerSync switches to use the new slot and deletes the old one. The Service logs these steps and, during a snapshot, how much WAL budget remains. See [Postgres Replication Slots and WAL Budget](/debugging/log-reference#postgres-replication-slots-and-wal-budget) in the Log Reference. The replication slots can be viewed using this query: diff --git a/debugging/error-codes.mdx b/debugging/error-codes.mdx index affbf86b9..4bf69b795 100644 --- a/debugging/error-codes.mdx +++ b/debugging/error-codes.mdx @@ -9,7 +9,7 @@ This reference documents PowerSync error codes organized by component, with trou ## PSYNC_Rxxxx: Sync Config issues - **PSYNC_R0001**: - Catch-all [Sync Config](/sync/overview) parsing error, if no more specific error is available + Catch-all [Sync Config](/sync/streams/quickstart#defining-streams) parsing error, if no more specific error is available - **PSYNC_R2201**: A table or schema wildcard (`%`) is not supported by the configured source connector. diff --git a/docs.json b/docs.json index c57a8c129..2d20dbcd9 100644 --- a/docs.json +++ b/docs.json @@ -183,42 +183,23 @@ ] }, { - "group": "Sync Streams & Rules", + "group": "Sync Streams", "icon": "arrows-rotate", "pages": [ - "sync/overview", - { - "group": "Sync Streams", - "pages": [ - "sync/streams/overview", - "sync/streams/parameters", - "sync/streams/queries", - "sync/streams/ctes", - "sync/streams/bucket-count", - "sync/streams/examples", - "sync/streams/client-usage", - "sync/streams/migration" - ] - }, - { - "group": "Sync Rules (Legacy)", - "pages": [ - "sync/rules/overview", - "sync/rules/organize-data-into-buckets", - "sync/rules/global-buckets", - "sync/rules/parameter-queries", - "sync/rules/data-queries", - "sync/rules/many-to-many-join-tables", - "sync/rules/client-parameters" - ] - }, + "sync/streams/overview", + "sync/streams/quickstart", + "sync/streams/parameters", + "sync/streams/queries", + "sync/streams/ctes", + "sync/streams/bucket-count", + "sync/streams/client-usage", "sync/types", + "sync/streams/examples", { "group": "Supported SQL", "pages": [ "sync/supported-sql", - "sync/grammar/sync-streams/index", - "sync/grammar/sync-rules/index" + "sync/grammar/sync-streams/index" ] }, { @@ -237,6 +218,20 @@ "sync/advanced/partitioned-tables", "sync/advanced/sharded-databases" ] + }, + { + "group": "Sync Rules (Legacy)", + "pages": [ + "sync/rules/migrate-to-sync-streams", + "sync/rules/overview", + "sync/rules/organize-data-into-buckets", + "sync/rules/global-buckets", + "sync/rules/parameter-queries", + "sync/rules/data-queries", + "sync/rules/many-to-many-join-tables", + "sync/rules/client-parameters", + "sync/grammar/sync-rules/index" + ] } ] }, @@ -784,6 +779,14 @@ "source": "/usage/sync-streams", "destination": "/sync/streams/overview" }, + { + "source": "/sync/overview", + "destination": "/sync/streams/overview" + }, + { + "source": "/sync/streams/migration", + "destination": "/sync/rules/migrate-to-sync-streams" + }, { "source": "/usage/sync-rules/types", "destination": "/sync/types" diff --git a/snippets/binary-type.mdx b/snippets/binary-type.mdx index ace29f838..121b61735 100644 --- a/snippets/binary-type.mdx +++ b/snippets/binary-type.mdx @@ -1,3 +1,3 @@ - Binary data can be accessed in the Sync Streams / Sync Rules, but cannot be used as [parameters](/sync/overview#how-it-works). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). + Binary data can be accessed in Sync Streams, but cannot be used as [parameters](/sync/streams/parameters). To sync binary columns/fields to clients, those columns need to be converted to hex or base64 representation using the relevant [functions](/sync/supported-sql#functions). \ No newline at end of file diff --git a/sync/advanced/overview.mdx b/sync/advanced/overview.mdx index de09cbf17..74a5d0707 100644 --- a/sync/advanced/overview.mdx +++ b/sync/advanced/overview.mdx @@ -1,6 +1,6 @@ --- title: "Advanced Topics" -description: "Advanced Sync Streams and Sync Rules topics." +description: "Advanced Sync Streams topics." sidebarTitle: Overview --- diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx index 56d5922ba..2b6b4b9ed 100644 --- a/sync/advanced/storage-version-4.mdx +++ b/sync/advanced/storage-version-4.mdx @@ -23,7 +23,7 @@ The PowerSync Cloud and self-hosted columns below apply during the Beta only. On | Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | | S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Set up S3 object storage](#self-hosted-s3-setup) | -Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/rules/migrate-to-sync-streams). ## Opt In diff --git a/sync/grammar/sync-streams/index.mdx b/sync/grammar/sync-streams/index.mdx index cc3e7f591..a4aab0fcb 100644 --- a/sync/grammar/sync-streams/index.mdx +++ b/sync/grammar/sync-streams/index.mdx @@ -1,5 +1,5 @@ --- -title: "Grammar Reference (Sync Streams)" +title: "Grammar Reference" description: "Railroad diagram reference for the SQL grammar supported in Sync Streams queries." --- diff --git a/sync/overview.mdx b/sync/overview.mdx deleted file mode 100644 index 8754bd87c..000000000 --- a/sync/overview.mdx +++ /dev/null @@ -1,90 +0,0 @@ ---- -title: "Sync Streams and Sync Rules" -sidebarTitle: "Overview" -description: "PowerSync Sync Streams and the legacy Sync Rules allow developers to control which data syncs to which clients/devices (i.e. they enable partial sync)." ---- - - -## Sync Streams — Recommended - -With [Sync Streams](/sync/streams/overview), you write SQL-like queries to define streams of data. Clients subscribe to the streams they need, either on-demand or automatically on connect. Sync Streams are the recommended path to achieve partial sync for both new and existing projects. - -Key improvements in Sync Streams over legacy Sync Rules include: -- **On-demand syncing**: You define Sync Streams on the PowerSync Service, and a client can then subscribe to them one or more times with different parameters, on-demand. You still have the option of auto-subscribing streams when a client connects, for "sync data upfront" behavior. -- **Temporary caching-like behavior**: Each subscription includes a configurable TTL that keeps data active after the client unsubscribes, acting as a warm cache for re-subscribing. -- **Simpler developer experience**: Simplified syntax and mental model, and capabilities such as your UI components automatically managing subscriptions (for example, React hooks). - -If you're on Sync Rules, you can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. - - - - -## Sync Rules (Legacy) - -Sync Rules is the legacy approach for controlling data sync. It remains available and supported for existing projects: - - - -If you're currently using Sync Rules and want to migrate to Sync Streams, see our [migration docs](/sync/streams/migration). - - -## How It Works - -You may also find it useful to look at the [PowerSync Service architecture](/architecture/powersync-service) for background. - -Each [PowerSync Service](/architecture/powersync-service) instance has a deployed _Sync Streams_ (or legacy _Sync Rules_) configuration. This takes the form of a YAML file which contains: -- **In the case of Sync Streams:** Definitions of the streams that exist, with a SQL-like query (which can also contain limited subqueries), which defines the data in the stream, and references the necessary parameters. -- **In the case of Sync Rules:** Definitions of the different [buckets](/architecture/powersync-service#bucket-system) that exist, with SQL-like queries to specify the parameters used by each bucket (if any), as well as the data contained in each bucket. - - -A _parameter_ is a value that can be used in Sync Streams (or legacy Sync Rules) to create dynamic sync behavior for each user/client. Each client syncs only the relevant [_buckets_](/architecture/powersync-service#bucket-system) based on the parameters for that client. -* Sync Streams can make use of _authentication parameters_ from the JWT token (such as the user ID or other JWT claims), _connection parameters_ (specified at connection), and _subscription parameters_ (specified by the client when it subscribes to a stream at any time). See [Using Parameters](/sync/streams/parameters). -* Sync Rules can make use of _authentication parameters_ from the JWT token, as well as [_client parameters_](/sync/rules/client-parameters) (passed directly from the client when it connects to the PowerSync Service). - -It is also possible to have buckets/streams with no parameters. In the case of Sync Rules, these buckets sync to all users/clients automatically. - - - -The concept of _buckets_ is core to PowerSync and key to its performance and scalability. The [PowerSync Service architecture overview](/architecture/powersync-service) provides more background on this. -* In _Sync Streams_, buckets and parameters are implicit — they are automatically created based on the streams, their queries and subqueries. You don't need to explicitly define the buckets that exist. -* In legacy _Sync Rules_, buckets and their parameters are [explicitly defined](/sync/rules/overview#bucket-definition). - - -There are limitations on the SQL syntax and functionality that is supported in Sync Streams and Sync Rules. See [Supported SQL](/sync/supported-sql) for details and limitations. - -In addition to filtering data based on parameters, Sync Streams and Sync Rules also enable: - -* Selecting only specific tables/collections and columns/fields to sync. -* Filtering data based on static conditions. -* Transforming column/field names and values. - - -### Sync Streams/Rules Determine Replication From the Source Database - -A PowerSync Service instance [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) relevant data from your backend source database according to your Sync Streams/Sync Rules. During replication, data and metadata are persisted in [buckets](/architecture/powersync-service#bucket-system) on the PowerSync Service. Buckets are incrementally updated so that they contain the latest state as well as a history of changes (operations). This is key to how PowerSync achieves efficient delta syncing — having the operation history for each bucket allows clients to sync only the deltas that they need to get up to date (see [Protocol](/architecture/powersync-protocol#protocol) for more details). - - -As a practical example, let's say you have a bucket named `user_todo_lists` that contains the to-do lists for a user, and that bucket utilizes a `user_id` parameter (which will be embedded in the JWT). Now let's say users with IDs `A` and `B` exist in the source database. PowerSync will then replicate data from the source database and create individual buckets with IDs `user_todo_lists["A"]` and `user_todo_lists["B"]`. When the user with ID `A` connects, they can efficiently sync just the bucket with ID `user_todo_lists["A"]`. - - - - - - - -### Sync Streams/Rules Determine Real-Time Streaming Sync to Clients - -Whenever buckets are updated (buckets added or removed, or operations added to existing buckets), these changes are [streamed in real-time](/architecture/powersync-service#streaming-sync) to clients based on the Sync Streams/Sync Rules. - -This syncing behavior can be highly dynamic: in the case of Sync Streams, syncing will dynamically adjust based on the stream subscriptions (which can make use of _subscription parameters_), as well as _connection parameters_ and _authentication parameters_ (from the JWT). In the case of Sync Rules, syncing will dynamically adjust based on changes in _client parameters_ and _authentication parameters_. - -The bucket data is persisted in SQLite on the client-side, where it is easily queryable based on the [client-side schema](/intro/setup-guide#define-your-client-side-schema), which corresponds to the Sync Streams/Rules. - -For more information on the client-side SQLite database structure, see [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure). - - - - - - - diff --git a/sync/rules/client-parameters.mdx b/sync/rules/client-parameters.mdx index 08a814109..cdd33224b 100644 --- a/sync/rules/client-parameters.mdx +++ b/sync/rules/client-parameters.mdx @@ -16,7 +16,7 @@ PowerSync already supports using **token parameters** in parameter queries. An e [Sync Streams](/sync/streams/overview) make it easier to manage dynamic parameters, especially for apps where parameters are managed across different UI components and tabs. Sync Streams offer _subscription parameters_ (specified when subscribing to a stream) and _connection parameters_ (the equivalent of client parameters). - We recommend Sync Streams for new projects, and [migrating](/sync/streams/migration) existing projects. + We recommend Sync Streams for new projects, and [migrating](/sync/rules/migrate-to-sync-streams) existing projects. ### Usage diff --git a/sync/streams/migration.mdx b/sync/rules/migrate-to-sync-streams.mdx similarity index 69% rename from sync/streams/migration.mdx rename to sync/rules/migrate-to-sync-streams.mdx index 365cebf2b..31bbc0a0c 100644 --- a/sync/streams/migration.mdx +++ b/sync/rules/migrate-to-sync-streams.mdx @@ -1,41 +1,27 @@ --- -title: "Migrating from Sync Rules" -description: "Migrate existing projects from legacy Sync Rules to Sync Streams." +title: "Migrate to Sync Streams" +description: "Migrate an existing project from legacy Sync Rules to Sync Streams." --- import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; -## Why Migrate? - -PowerSync's original Sync Rules system was optimized for offline-first use cases where you want to "sync everything upfront" when the client connects, so data is available locally if the user goes offline. - -However, many developers are building apps where users are mostly online, and you don't want to make users wait to sync a lot of data upfront. This is especially true for **web apps**: users are mostly online, you often want to sync only the data needed for the current page, and users frequently have multiple browser tabs open — each needing different subsets of data. - -### The Problem with Client Parameters - -[Client Parameters](/sync/rules/client-parameters) in Sync Rules partially support on-demand syncing — for example, using a `project_ids` array to sync only specific projects. However, manually managing these arrays across different browser tabs becomes painful: +Sync Streams do everything Sync Rules do, and more. A stream with `auto_subscribe: true` syncs when the client connects, the same way a bucket definition does, so apps that sync all relevant data upfront for offline use keep working the same way after migrating. The [migration tool](#migration-tool) sets `auto_subscribe: true` on every generated stream, so no client-side changes are required when you first deploy. -- You need to aggregate IDs across all open tabs -- You need additional logic for different data types (tables) -- If you want to keep data around after a tab closes (caching), you need even more management - -### How Sync Streams Solve This - -Sync Streams address these limitations: +## Why Migrate? -1. **On-demand syncing**: Define streams once, then subscribe from your app one or more times with different parameters. No need to manage arrays of IDs — each subscription is independent. +Beyond matching Sync Rules, Sync Streams add: -2. **Multi-tab support**: Each subscription manages its own lifecycle. Open the same list in two tabs? Each tab subscribes independently. Close one? The other keeps working. +1. **More expressive queries**: Stream queries support JOINs, [CTEs](/sync/streams/ctes), subqueries, and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream), with syntax closer to plain SQL. [Parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write one query instead of separate `parameters:` and `data:` blocks. -3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data cached after unsubscribing. When users return to a screen, data may already be available — no loading state needed. +2. **On-demand syncing**: Define a stream once, then subscribe from your app one or more times with different parameters. Each subscription has its own lifecycle, so two screens or browser tabs can subscribe to the same stream independently. With Sync Rules, [Client Parameters](/sync/rules/client-parameters) approximate this, but you have to aggregate the parameter values yourself across screens and tabs, and remove them when they are no longer needed. -4. **Simpler, more powerful syntax**: Stream queries support JOINs, CTEs, subqueries, and multiple queries per stream, and the syntax is closer to plain SQL. For example, [parameter queries become inline subqueries](#data-with-subqueries-replaces-parameter-queries), so you write a single query instead of separate `parameters:` and `data:` blocks. +3. **Built-in caching**: Each subscription has a configurable `ttl` that keeps data on the device after unsubscribing. When users return to a screen, the data is often already available. -5. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let your UI components automatically manage subscriptions based on what's rendered. +4. **Framework integration**: [React hooks, Vue composables, TanStack Query, and Kotlin Compose extensions](/sync/streams/client-usage#framework-integrations) let UI components manage subscriptions based on what is rendered. -### Still Need Offline-First? +5. **Access to new features**: Newer PowerSync Service features such as [wildcard schemas](/sync/advanced/schemas-and-connections) and [incremental reprocessing](/sync/advanced/storage-version-4) require Sync Streams. -If you want "sync everything upfront" behavior (like Sync Rules), set [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) on your Sync Streams and clients will subscribe automatically when they connect. +You can migrate incrementally. Deploy the generated streams with `auto_subscribe: true` first, then convert individual streams to on-demand subscriptions where that benefits your app. ## Requirements @@ -117,7 +103,7 @@ The output uses `auto_subscribe: true` by default, preserving your existing sync ### Global Data (No Parameters) -In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/overview#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. +In Sync Rules, a ["global" bucket](/sync/rules/global-buckets) syncs the same data to all users. In Sync Streams, you achieve this with queries that have no parameters. Add [`auto_subscribe: true`](/sync/streams/quickstart#using-auto-subscribe) to maintain the Sync Rules behavior where data syncs automatically on connect. **Sync Rules:** ```yaml @@ -236,7 +222,7 @@ const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); ## Client-Side Changes -After updating your Sync Config, update your client code to use subscriptions: +Streams generated by the migration tool with `auto_subscribe: true` need no client changes. When you convert a stream to on-demand syncing, replace connect-time parameters with a subscription: ```js // Before (Sync Rules with Client Parameters) @@ -249,4 +235,6 @@ await db.connect(connector); const sub = await db.syncStream('project_data', { project_id: projectId }).subscribe(); ``` +If you want to keep passing values at connect time instead, use [connection parameters](/sync/streams/parameters#connection-parameters). + See [Client-Side Usage](/sync/streams/client-usage) for detailed examples. diff --git a/sync/rules/overview.mdx b/sync/rules/overview.mdx index 6d85e0bcc..17921d0e2 100644 --- a/sync/rules/overview.mdx +++ b/sync/rules/overview.mdx @@ -11,7 +11,7 @@ Sync Rules are PowerSync's original system for results in fewer sync buckets. +If multiple streams share the same filtering logic, consider using [CTEs](/sync/streams/ctes) to avoid repetition and [multiple queries per stream](/sync/streams/queries#multiple-queries-per-stream) so the client only needs to manage one subscription instead of multiple. This is more efficient and results in fewer sync buckets. ### User's Default or Primary Item diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 3ec4013ca..096b8fc3a 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -1,287 +1,66 @@ --- title: "Sync Streams" -description: "Introduction to Sync Streams, the recommended way to define which data syncs to each client with SQL-based stream definitions." -sidebarTitle: "Quickstart" +description: "Sync Streams define which data syncs to each client. Learn what a stream is and how PowerSync replicates and streams the data." +sidebarTitle: "Overview" --- -import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; +With Sync Streams, you write SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync keeps subscribed data synced in real-time to a client-side SQLite database, where it stays available when the device is offline. -With Sync Streams, you write simple SQL-like queries to define streams of data, and your client app subscribes to the streams it needs. This enables _partial sync_: each client syncs only the relevant subset of data, instead of the entire database. PowerSync handles the rest, keeping subscribed data synced in real-time to a client-side SQLite database, where it stays available even when the device is offline. - -For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Offline-first apps that need all relevant data available upfront can use `auto_subscribe: true` so streams sync automatically when clients connect. +For example, you might define a stream that syncs only the current user's to-do items, another for shared projects they have access to, and another for reference data that everyone needs. Your app subscribes to these streams on demand, and only that data syncs to the device. Apps that need all relevant data available upfront can set `auto_subscribe: true` so streams sync automatically when clients connect. **Are you still using Sync Rules?** Sync Streams support everything Sync Rules do, plus more expressive queries (including JOIN support), on-demand syncing, and a simpler developer experience (e.g. React hooks that manage subscriptions automatically). -You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See the [migration guide](/sync/streams/migration) for details. +You can migrate in a few clicks. Click **Migrate to Sync Streams** in the PowerSync Dashboard, or run `powersync migrate sync-rules` in the CLI to generate a draft from your current config. See [Migrate to Sync Streams](/sync/rules/migrate-to-sync-streams) for details. -## Defining Streams - -Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/overview#how-it-works) like the authenticated user's ID to personalize what each user receives. - - - -In the [PowerSync Dashboard](https://dashboard.powersync.com/): - -1. Select your project and instance -2. Go to **Sync Streams** -3. Edit the YAML directly in the dashboard -4. Click **Deploy** to validate and deploy - -```yaml -config: - edition: 3 - -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` - - - -Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: - -```yaml sync-config.yaml -config: - edition: 3 +## How It Works -streams: - todos: - query: SELECT * FROM todos WHERE owner_id = auth.user_id() -``` +Each PowerSync Service instance has a deployed Sync Streams configuration: a YAML file that defines the streams that exist. Each stream has a name and a SQL-like query that selects the tables and columns to sync, filters rows by static conditions or by parameters, and can rename or transform columns. The Service uses this configuration in two places: when it replicates data from your source database into buckets, and when it streams those buckets to clients. -You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. - - +See the [PowerSync Service architecture](/architecture/powersync-service) for more background. -Available stream options: +### Buckets and Parameters - +PowerSync groups replicated data into [buckets](/architecture/powersync-service#bucket-system): partitions of data that are synced as a unit. A stream creates one bucket for each unique value of its filter, such as each user ID matched by `auth.user_id()` or each `list_id` that a client subscribes with. A stream without parameters creates a single bucket that syncs the same data to every subscriber. -## Basic Examples +Buckets are implicit in Sync Streams. The Service creates them from your stream queries, parameters, and subqueries, and you do not define or name them yourself. See [Bucket Count](/sync/streams/bucket-count) for how queries determine the number of buckets. -There are two independent concepts to understand: +### Replication From the Source Database -- _What_ data the stream returns. For example: - - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). - - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. -- _When_ the client syncs the data - - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) - - *On-demand*: Client explicitly subscribes when needed (default behavior) - -### Global Data - -Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: - -```yaml -config: - edition: 3 - -streams: - # Same categories for everyone - categories: - query: SELECT * FROM categories - - # Same active products for everyone - products: - query: SELECT * FROM products WHERE active = true -``` +The Service [replicates and transforms](/architecture/powersync-service#replication-from-the-source-database) data from your source database according to your stream queries, and persists the data and metadata in buckets. Buckets are updated incrementally, so they contain the latest state as well as a history of changes (operations). This operation history allows clients to sync only the deltas they need to get up to date. See [Protocol](/architecture/powersync-protocol#protocol) for details. -Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` +For example, a stream `user_lists` with the query `SELECT * FROM lists WHERE owner_id = auth.user_id()` creates one bucket per user. If users `A` and `B` exist in the source database, the Service creates a bucket for each of them. When user `A` connects and subscribes, they sync only their own bucket. -### Filtering Data by User - -Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: - -```yaml -config: - edition: 3 - -streams: - # Each user gets their own lists - my_lists: - query: SELECT * FROM lists WHERE owner_id = auth.user_id() - - # Each user gets their own orders - my_orders: - query: SELECT * FROM orders WHERE user_id = auth.user_id() -``` - -### Filtering Data Based on Subscription Parameters - -Use `subscription.parameter()` for data that clients subscribe to explicitly: - -```yaml -config: - edition: 3 - -streams: - # Sync todos for a specific list when the client subscribes with a list_id - list_todos: - query: | - SELECT * FROM todos - WHERE list_id = subscription.parameter('list_id') - AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) -``` - -```js -// Client subscribes with the list they want to view -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); -``` - -### Using Auto-Subscribe - -Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: -- Reference data that all users need, or that are needed in many screens in the app. -- User data that should always be available offline -- Maintaining [Sync Rules](/sync/rules/overview) default behavior ("sync everything upfront") when migrating to Sync Streams - -```yaml -config: - edition: 3 - -streams: - # Global data, synced automatically - categories: - auto_subscribe: true - query: SELECT * FROM categories - - # User-scoped data, synced automatically - my_orders: - auto_subscribe: true - query: SELECT * FROM orders WHERE user_id = auth.user_id() - - # Parameterized data, subscribed on-demand (no auto_subscribe) - order_items: - query: | - SELECT * FROM order_items - WHERE order_id = subscription.parameter('order_id') - AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) -``` - - -## Client-Side Usage - -Subscribe to streams from your client app: - - - -```js -const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) - .subscribe({ ttl: 3600 }); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - -**React hooks:** - -```jsx -const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); -// Check download progress or subscription information -stream?.progress; -stream?.subscription.hasSynced; -``` - -The `useQuery` hook can wait for Sync Streams before running queries: - -```jsx -const { data } = useQuery( - 'SELECT * FROM todos WHERE list_id = ?', - [listId], - { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } -); -``` - - - -```dart -final sub = await db - .syncStream('list_todos', {'list_id': 'abc123'}) - .subscribe(ttl: const Duration(hours: 1)); - -// Wait for this subscription to have synced -await sub.waitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.unsubscribe(); -``` - - - -```kotlin -val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) - .subscribe(ttl = 1.0.hours) - -// Wait for this subscription to have synced -sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -sub.unsubscribe() -``` - - - - -```swift -let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) - .subscribe(ttl: 60 * 60, priority: nil) // 1 hour - -// Wait for this subscription to have synced -try await sub.waitForFirstSync() - -// When the component needing the subscription is no longer active... -try await sub.unsubscribe() -``` - - - -```csharp -var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) - .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); - -// Wait for this subscription to have synced -await sub.WaitForFirstSync(); - -// When the component needing the subscription is no longer active... -sub.Unsubscribe(); -``` - - - -### TTL (Time-To-Live) - -Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior — when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. - -```js -// Set TTL in seconds when subscribing -const sub = await db.syncStream('todos', { list_id: 'abc' }) - .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe -``` -## Developer Notes - -- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details on joins, multiple queries per stream, and other features. - -- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. - -- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). - -- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + + + -- **Bucket Limits**: PowerSync uses internal partitions called [buckets](/architecture/powersync-service#bucket-system) to efficiently sync data. Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. +### Streaming Sync to Clients -- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user — you can see which buckets the user has and what data is being synced. +Whenever buckets change (buckets are added or removed, or operations are added to existing buckets), the Service [streams these changes in real-time](/architecture/powersync-service#streaming-sync) to the subscribed clients. The set of buckets a client receives adjusts as it subscribes to and unsubscribes from streams, and depends on its subscription, connection, and authentication parameters. -## Examples & Demos +On the client, bucket data is persisted in SQLite, where you query it through your [client-side schema](/intro/setup-guide#define-your-client-side-schema). See [Client Architecture](/architecture/client-architecture#client-side-schema-and-sqlite-database-structure) for the database structure. -See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns. + + + -## Migrating from Legacy Sync Rules +## Next Steps -If you have an existing project using legacy Sync Rules, see the [Migration Guide](/sync/streams/migration) for step-by-step instructions, syntax changes, and examples. + + + Define your first streams and subscribe to them from your app. + + + Filter data with auth, subscription, and connection parameters. + + + Query syntax, joins, subqueries, and multiple queries per stream. + + + Manage subscriptions in each SDK and UI framework. + + diff --git a/sync/streams/quickstart.mdx b/sync/streams/quickstart.mdx new file mode 100644 index 000000000..c8ae8cd68 --- /dev/null +++ b/sync/streams/quickstart.mdx @@ -0,0 +1,276 @@ +--- +title: "Sync Streams Quickstart" +description: "Define your first Sync Streams and subscribe to them from your client app." +sidebarTitle: "Quickstart" +--- + +import StreamDefinitionReference from '/snippets/stream-definition-reference.mdx'; + +This page shows how to define streams and subscribe to them from your app. For what streams are and how PowerSync replicates and syncs them, see the [Sync Streams overview](/sync/streams/overview). + +## Defining Streams + +Streams are defined in a YAML configuration file. Each stream has a **name** and a **query** that specifies which rows to sync using SQL-like syntax. The query can reference [parameters](/sync/streams/parameters) like the authenticated user's ID to personalize what each user receives. + + + +In the [PowerSync Dashboard](https://dashboard.powersync.com/): + +1. Select your project and instance +2. Go to **Sync Streams** +3. Edit the YAML directly in the dashboard +4. Click **Deploy** to validate and deploy + +```yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + + + +Add a `sync_config` section to your `service.yaml`. Using a **separate file** is recommended (e.g. `sync_config: path: sync-config.yaml`). Put the stream definition in that file: + +```yaml sync-config.yaml +config: + edition: 3 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +You can also use inline `sync_config: content: |` with the YAML nested in your main config. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances#sync_config) for both options. + + + +Available stream options: + + + +## Basic Examples + +There are two independent concepts to understand: + +- _What_ data the stream returns. For example: + - *Global data*: No parameters. Same data for all users (e.g. reference tables like categories). + - *Filtered data*: Filters the data by a parameter value. This can make use of _auth parameters_ from the JWT token (such as the user ID or other JWT claims), _subscription parameters_ (specified by the client when it subscribes to a stream at any time), or _connection parameters_ (specified at connection). Different users will get different sets of data based on the parameters. See [Using Parameters](/sync/streams/parameters) for the full reference. +- _When_ the client syncs the data + - *Auto-subscribe*: Client automatically subscribes on connect (`auto_subscribe: true`) + - *On-demand*: Client explicitly subscribes when needed (default behavior) + +### Global Data + +Data without parameters is "global" data, meaning the same data goes to all users/clients. This is useful for reference tables: + +```yaml +config: + edition: 3 + +streams: + # Same categories for everyone + categories: + query: SELECT * FROM categories + + # Same active products for everyone + products: + query: SELECT * FROM products WHERE active = true +``` + + +Global data streams still require clients to subscribe explicitly unless you set `auto_subscribe: true` + + +### Filtering Data by User + +Use `auth.user_id()` or other [JWT claims](/sync/streams/parameters#auth-parameters) to return different data per user: + +```yaml +config: + edition: 3 + +streams: + # Each user gets their own lists + my_lists: + query: SELECT * FROM lists WHERE owner_id = auth.user_id() + + # Each user gets their own orders + my_orders: + query: SELECT * FROM orders WHERE user_id = auth.user_id() +``` + +### Filtering Data Based on Subscription Parameters + +Use `subscription.parameter()` for data that clients subscribe to explicitly: + +```yaml +config: + edition: 3 + +streams: + # Sync todos for a specific list when the client subscribes with a list_id + list_todos: + query: | + SELECT * FROM todos + WHERE list_id = subscription.parameter('list_id') + AND list_id IN (SELECT id FROM lists WHERE owner_id = auth.user_id()) +``` + +```js +// Client subscribes with the list they want to view +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }).subscribe(); +``` + +### Using Auto-Subscribe + +Set `auto_subscribe: true` to sync data automatically when clients connect. This is useful for: +- Reference data that all users need, or that is needed in many screens in the app. +- User data that should always be available offline. +- Keeping the "sync everything upfront" behavior of legacy [Sync Rules](/sync/rules/overview) when migrating to Sync Streams. + +```yaml +config: + edition: 3 + +streams: + # Global data, synced automatically + categories: + auto_subscribe: true + query: SELECT * FROM categories + + # User-scoped data, synced automatically + my_orders: + auto_subscribe: true + query: SELECT * FROM orders WHERE user_id = auth.user_id() + + # Parameterized data, subscribed on-demand (no auto_subscribe) + order_items: + query: | + SELECT * FROM order_items + WHERE order_id = subscription.parameter('order_id') + AND order_id IN (SELECT id FROM orders WHERE user_id = auth.user_id()) +``` + + +## Client-Side Usage + +Subscribe to streams from your client app: + + + +```js +const sub = await db.syncStream('list_todos', { list_id: 'abc123' }) + .subscribe({ ttl: 3600 }); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + +**React hooks:** + +```jsx +const stream = useSyncStream({ name: 'list_todos', parameters: { list_id: 'abc123' } }); +// Check download progress or subscription information +stream?.progress; +stream?.subscription.hasSynced; +``` + +The `useQuery` hook can wait for Sync Streams before running queries: + +```jsx +const { data } = useQuery( + 'SELECT * FROM todos WHERE list_id = ?', + [listId], + { streams: [{ name: 'list_todos', parameters: { list_id: listId }, waitForStream: true }] } +); +``` + + + +```dart +final sub = await db + .syncStream('list_todos', {'list_id': 'abc123'}) + .subscribe(ttl: const Duration(hours: 1)); + +// Wait for this subscription to have synced +await sub.waitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.unsubscribe(); +``` + + + +```kotlin +val sub = database.syncStream("list_todos", mapOf("list_id" to JsonParam.String("abc123"))) + .subscribe(ttl = 1.0.hours) + +// Wait for this subscription to have synced +sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +sub.unsubscribe() +``` + + + + +```swift +let sub = try await db.syncStream(name: "list_todos", params: ["list_id": JsonValue.string("abc123")]) + .subscribe(ttl: 60 * 60, priority: nil) // 1 hour + +// Wait for this subscription to have synced +try await sub.waitForFirstSync() + +// When the component needing the subscription is no longer active... +try await sub.unsubscribe() +``` + + + +```csharp +var sub = await db.SyncStream("list_todos", new() { ["list_id"] = "abc123" }) + .Subscribe(new SyncStreamSubscribeOptions { Ttl = TimeSpan.FromHours(1) }); + +// Wait for this subscription to have synced +await sub.WaitForFirstSync(); + +// When the component needing the subscription is no longer active... +sub.Unsubscribe(); +``` + + + +### TTL (Time-To-Live) + +Each subscription has a `ttl` that keeps data cached after unsubscribing. This enables warm cache behavior: when users return to a screen and you re-subscribe to relevant streams, data is already available on the client. Default TTL is 24 hours. See [Client-Side Usage](/sync/streams/client-usage) for details. + +```js +// Set TTL in seconds when subscribing +const sub = await db.syncStream('todos', { list_id: 'abc' }) + .subscribe({ ttl: 3600 }); // Cache for 1 hour after unsubscribe +``` + +## Developer Notes + +- **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can select specific columns, filter on static conditions, rename or transform columns, and use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details and [Supported SQL](/sync/supported-sql) for the operators and functions you can use. + +- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. + +- **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT *, _id as id FROM your_collection`). + +- **Case Sensitivity**: To avoid issues across different databases and platforms, use **lowercase identifiers** for all table and column names in your Sync Streams. If your backend uses mixed case, see [Case Sensitivity](/sync/advanced/case-sensitivity) for how to handle it. + +- **Bucket Limits**: Each user has a limit on how many buckets they can sync ([1,000 by default](/resources/performance-and-limits)), and the query pattern determines how many each stream creates. See [Bucket Count](/sync/streams/bucket-count) for how buckets are counted, and [Reducing Bucket Count](/sync/advanced/reducing-bucket-count) to resolve `PSYNC_S2305` errors. + +- **Troubleshooting**: If data isn't syncing as expected, the [Sync Diagnostics Client](/tools/diagnostics-client) helps you inspect what's happening for a specific user. You can see which buckets the user has and what data is being synced. + +## Examples & Demos + +See [Examples & Demos](/sync/streams/examples) for working demo apps and complete application patterns.