Managed raw tables

[simolus3]

This adds a new type of table next to the existing ones based on JSON data + views (easy to manage and migrate, slower to query) and raw tables (fast queries and full customizability at the expense of a higher setup complexity). Direct tables (name tbd) have the shape of raw tables with proper SQLite columns that can be queried without decoding JSON on reads. At the same time, they still store the original JSON object they're derived from, which means they can be managed by the core extension.

A direct table has a shape like this:

CREATE TABLE users (
  id TEXT NOT NULL PRIMARY KEY,
 _rest TEXT, -- synced data not covered by declared columns
  -- Rest of this depends on schema set on PowerSync database
  name TEXT,
  email TEXT,
  points INTEGER
);

The sync client treats this table like raw tables, automatically inferring statements to sync into it. There is no view to generate, but we generate triggers to forward writes into ps_crud. For updates, the trigger also updates the __data column.

As an alternative to this table shape, I have considered keeping a ps_data__-like table that just adds GENERATED ALWAYS AS columns. This might seem simpler, but then you wouldn't be able to update these columns directly so we'd have to keep generating views and INSTEAD OF triggers. The current pattern is closer to the shape one would expect SQLite tables to have.

Direct tables are currently an opt-in, but we could make them the default in a future release.

TODOs

• Support creating direct tables, move existing contents from ps_untyped
• Support schema changes where direct tables are removed.
• Support migrating from json-based to direct tables.
• Support migrating from direct to json-based tables.
• Support indices on direct tables.
• Support local-only direct tables.
• Handle schema changes on direct tables.
• Support migrating between synced and local-only direct tables.
• Investigate using a rest column instead of storing the full __data set.

[rkistner]

I think this feature will be great, but I wonder if we could remove the _data / _rest column completely to reduce the overhead - same reasoning as this comment? #189 (comment)

Or would that make the migration paths with JSON-based tables too difficult? In that case, maybe we can consider another change later (potentially breaking or opt-in) to remove ps_untyped and _rest usage?

[simolus3]

I like that idea. Perhaps there could be a single flag (on the entire schema, not per-table) that does both.

1. When enabled for the first time, we create direct tables (copying rows from old ps_data__ tables and ps_untyped). We then delete old tables, clear ps_untyped and stop using it.
   1. If there is synced data for a table not covered by the schema, the client could emit a log. Otherwise things like typos between client/server state are very tricky to debug (we already had reports where people add aliases to their sync streams and then complain about data not syncing when it's in ps_untyped).
2. If a developer wants to disable this flag later, we can reconstruct ps_data__ and ps_untyped by forming a json structure of existing columns (read via pragma_table_info). That can miss columns, so a resync trigger is still necessary.