Use proper result set when evaluating parameters by simolus3 · Pull Request #782 · powersync-ja/powersync-service

simolus3 · 2026-09-01T15:30:52Z

When querying buckets for Sync Streams, we generally try to resolve parameters independently to form a cartesian product in the end. This is correct for most streams, but goes wrong when a parameter index has more than one column. For example, in SELECT a.* FROM a, b WHERE a.c1 = b.c1 AND a.c2 = b.c2 AND b.u = auth.user_id(), the two parameters are b.c1 and b.c2. If we encounter multiple rows of b through a lookup result, we can't assume those to be independent parameters though! We can only pair parameters that originate from the same row.

This is currently implemented by tracking provenance for each parameter value back to the lookup this originally came from. When we build the cartesian product in the end, we ignore values with incompatible provenance from different rows. Unfortunately, tracking provenance is both kind of expensive and very tricky to get right.

Semantically, evaluating bucket parameters involves:

Forming a result set by adding lookups through joins or a cartesian product.
Once we have the full result set, bucket parameters are a projection.

This replaces the previous querier logic with an actual result set implementation: We start out with a unit set of one row without columns, then go through added lookups that are cross-joined (table-valued functions) or inner-joined (parameter lookups). If we end up with an empty intermediate result set at any point, we know there won't be any buckets and bail out early.

Intersection parameters require special consideration now, but can be implemented by going through the result set and deleting rows where the columns don't match.

AI use: The approach is manual, most tests and some implementation details are generated with Claude Code.

changeset-bot · 2026-09-01T15:30:58Z

⚠️ No Changeset found

Latest commit: b5cf4eb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

simolus3 · 2026-09-01T15:36:36Z

+      }

-    for (let i = 0; i < this.lookupStages.length; i++) {
-      // Within a stage, we can resolve lookups concurrently.


With the implementation from this PR, joins are no longer processed concurrently. It's possible to add that back with minor added complexity, but:

this is only relevant for complex sync streams

we already evaluate other users / queriers concurrently, to the point where bucket storage is likely the bottleneck and not JS

So I don't think this is necessarily something worth doing, but I can change this here / in a follow-up PR if needed.

simolus3 · 2026-09-01T15:39:58Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68a4c2dc8b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af866d1014

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

rkistner

Codex picked up a performance regression here with intersecting results. Example queries:

-- In this case we can intersect results
SELECT * FROM issues
  WHERE id IN auth.parameter('allowed_issue_ids')
    AND id IN subscription.parameter('visible_issue_ids')
    AND id IN subscription.parameter('selected_issue_ids');

-- In this case we can compute the checks independently
SELECT * FROM issues
  WHERE 'issues' IN auth.parameter('tables')
    AND 'read' IN auth.parameter('permissions');

With the old implementation, the intersection was computed first. The new implementation builds up the Cartesian product of the arrays before filtering them. In a cases like the above, the number of results can explode even with modestly-sized arrays, leading to slow performance and/or OOM-crashes.

simolus3 · 2026-09-08T13:23:53Z

That is a good point. Since the columns of intersections are known statically, we can evaluate them for each row added to the result set instead of removing rows after building the cartesian product.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 849a419d50

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

rkistner

I like the approach of using a ResultSet here, but Codex picked up another couple of performance regressions with the implementation (manually confirmed the findings).

rkistner · 2026-09-10T13:39:46Z

+    for (const row of rows) {
+      if (!filter(originalRow, row)) {
+        continue;
+      }


The memory usage is now only O(N) instead of the O(N^2) before, but the comparisons are still O(N^2) here.

This affects the same query example mentioned before:

SELECT * FROM issues WHERE id IN auth.parameter('allowed_issue_ids') AND id IN subscription.parameter('visible_issue_ids') AND id IN subscription.parameter('selected_issue_ids');

This is tricky to optimize in ResultSet because it also needs to work for async joins from parameters (where we at least know we have a constant upper bound due to the paramer limit...).

To avoid pathological cases for parameter-based intersections, I'll look into optimizing those specifically in the evaluator.

rkistner · 2026-09-10T13:46:51Z

+        for (const toDelete of deletedRows) {
+          this.#rows.splice(toDelete - offset, 1);
+          offset++;
+        }


Array.splice() is O(N) on the array size, making this loop O(N^2).

simolus3 commented Sep 1, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

simolus3 commented Sep 1, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts

simolus3 mentioned this pull request Sep 1, 2026

Fix provenance for lookups with multiple inputs #780

Merged

simolus3 marked this pull request as ready for review September 2, 2026 08:41

chatgpt-codex-connector Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

simolus3 requested a review from rkistner September 2, 2026 09:11

rkistner reviewed Sep 8, 2026

View reviewed changes

simolus3 added 9 commits September 8, 2026 13:47

Represent instantiated parameters as result set

f577834

Fix most evaluator tests

46d9374

Support intersections

0454ab0

Test empty joins

a17c768

Fix for falsy resolved values

a31864a

Freeze rows before adding them

69caf10

AI feedback

b6ff8f3

Evaluate intersection on table-valued functions eagerly

d5500fe

Build intersection row-by-row

849a419

simolus3 force-pushed the update-evaluator branch from dade362 to 849a419 Compare September 8, 2026 13:22

chatgpt-codex-connector Bot reviewed Sep 8, 2026

View reviewed changes

Comment thread packages/sync-rules/src/sync_plan/evaluator/parameter_evaluator.ts Outdated

Comment thread packages/sync-rules/src/sync_plan/evaluator/result_set.ts Outdated

AI feedback

b5cf4eb

rkistner reviewed Sep 10, 2026

View reviewed changes

Uh oh!

Conversation

simolus3 commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

simolus3 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

simolus3 commented Sep 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

simolus3 commented Sep 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

rkistner left a comment

Choose a reason for hiding this comment

Uh oh!

rkistner Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

simolus3 Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rkistner Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simolus3 commented Sep 1, 2026 •

edited

Loading

changeset-bot Bot commented Sep 1, 2026 •

edited

Loading

simolus3 Sep 10, 2026 •

edited

Loading