Add opt-in incremental (append-semantics) element data delivery

[mrubash1]

I build Sigma plugins against large sources — think detail grids and
data-dense visualizations over hundreds of thousands of rows — and I've hit a
scaling wall in the SDK's data delivery contract that makes any element larger
than one page quadratically expensive to load. This PR adds a minimal,
fully backward-compatible, opt-in path to linear delivery. It defines the
plugin half of the protocol only; it is a safe no-op until the host implements
the other half, and it degrades gracefully in the meantime.

Problem

The current contract forces cumulative delivery:

• WorkbookElementData is bare column arrays — { [colId: string]: any[] }.
  There is no field where an offset, page index, completion flag, or total row
  count could even be expressed today. (This also means "the plugin author is
  holding it wrong" isn't available as an explanation: the types give a plugin
  no way to do better.)
• subscribeToElementData delivers via a callback whose payload consumers use
  to replace their entire state — the SDK's own useElementData and
  usePaginatedElementData hooks do exactly this (setData). Because
  delivery is replace-semantics, the host must ship the full accumulated array
  on every page; sending only the new rows would discard all prior data.
• fetchMoreElementData posts wb:plugin:element:fetch-more carrying only
  the configId. The client holds no pagination state and cannot request a
  slice.
• The contract's cost is measurable end-to-end, including the network leg.
  Instrumenting the host page during one initial load of the same 546k×58
  source described below: the host translates each SDK fetch-more into a
  fresh POST /api/v2/db/ir/eval with outputIR.pager.offset = 0 and a
  limit that grows by 25,000 per round (25001 → 50001 → 75001 → …). Across 17
  captured consecutive requests, a field-level diff of the decompressed
  request bodies (~690 leaf fields) shows only three fields changing:
  pager.limit (+25000 each round) and two wall-clock timestamps. The offset
  is present and pinned at 0 in every request. The server returns exactly
  what it is asked for — the full accumulated prefix, every round.

The consequence is that loading n rows at page size P delivers
Σ k·P for k = 1..⌈n/P⌉ ≈ n²/2P row-deliveries — O(n²) bytes serialized,
structure-cloned, and parsed across the iframe boundary.

Concretely, as a Sigma user: on a 546k-row × 58-column source, the element
loads in ~22 pages of 25k rows. I instrumented the data callback and each
delivery carried the entire accumulated set — callback #9 carried 200,001
rows, #13 carried 300,001, #17 carried 400,001. Total delivery cost was ~6.4M
row-deliveries to convey 546k rows — roughly ~4 GB of cumulative
serialize/clone/parse to deliver a ~295 MB final dataset. Per-page latency
degraded from ~2.6s for early pages to ~7.3s as the accumulated payload grew,
for a total load time of 5–6 minutes. The same plugin logic against a sub-25k
source is instant.

Measured cost of the network leg for one (truncated, 17-of-~19-round) load:
~119 MB compressed on the wire, ~2.07 GB decoded, to deliver a ~294 MB final
dataset; extrapolated full-load wire cost ~147 MB. Per-page wall time degraded
from ~2s to ~20s+ as the limit grew. This establishes that the quadratic
behavior exists on both sides of the plugin boundary — warehouse→browser
(host re-queries the full prefix) and browser→plugin (host re-delivers the
full accumulated array) — and that the host's grow-the-limit pager is the
implementation the replace-semantics contract naturally induces: since the
SDK obligates delivery of the full accumulated set on every page, re-querying
it wholesale is the consistent implementation. The contract is the root
cause; the network waste is its symptom.

Why this design

Additive envelope + capability handshake. The change adds one type, one
client method, and one hook — nothing existing is touched:

• WorkbookElementDataChunk — { data, offset, isComplete, totalRows? },
  where data contains only the chunk's rows and offset is the absolute row
  offset to apply them at.
• client.elements.subscribeToIncrementalElementData(configId, callback) —
  identical to subscribeToElementData except the subscribe message carries a
  capability option: wb:plugin:element:subscribe:data is posted with args
  [configId, { mode: 'incremental' }]. Hosts that don't recognize the extra
  argument ignore it and keep sending cumulative payloads.
• useIncrementalElementData(configId) — returns
  [data, loadMore, { rowCount, isComplete, totalRows }]. The first two
  positions match usePaginatedElementData, so migrating a plugin is a
  one-line hook swap.

Why append-semantics must be opt-in. Every existing plugin is written
against replace-semantics — the entire ecosystem's callbacks do
setData(payload). Changing the meaning of the existing subscription would
silently truncate every existing plugin's data to the last page. Opt-in via a
new method keeps the blast radius at exactly zero.

Why the hook accumulates internally. Chunk assembly (splicing rows at
offsets, tracking completion) is mechanical and easy to get subtly wrong.
Doing it inside the hook keeps the migration to a one-line swap and gives
every consumer the same well-tested assembly logic. Plugins that want raw
chunks (e.g., to stream into WebGL buffers or a worker) can use the client
method directly.

Why degraded mode matters. The host and this SDK ship on independent
cadences, and plugin authors don't control which Sigma version renders their
iframe. A plugin built against the new API must work correctly today,
against hosts that have never heard of the capability flag. The design makes
this fall out naturally rather than requiring branching code:

• Disambiguation rule: every value in a legacy WorkbookElementData payload
  is a column array, so a payload with a numeric offset, boolean
  isComplete, and non-array object data can only be the chunk envelope.
  The client checks exactly that.
• Normalization: legacy cumulative payloads are delivered to the incremental
  callback as { data: payload, offset: 0, isComplete: false } — a
  replace-everything chunk.
• Idempotent, defensive assembly: the hook trusts offset — rows before it
  are kept, rows at or after it are overwritten, and rows are padded to their
  absolute offset if a column first appears mid-stream. A re-sent or
  overlapping chunk applies idempotently; a chunk that omits a column (or an
  empty chunk that only flips isComplete) carries the accumulated columns
  forward instead of dropping them; and a normalized legacy payload at offset
  0 becomes an exact wholesale replace, which is precisely today's behavior.
  One code path handles both hosts. Column ids are applied own-property-safely
  (__proto__ is skipped; ids like constructor cannot collide with
  inherited members), and envelopes with malformed offsets (negative,
  fractional, NaN) are rejected by the type guard rather than corrupting
  assembly.

One honest limitation of degraded mode: legacy hosts never signal completion,
so isComplete remains false against them — the same (absence of)
information plugins have today. Because that is true of 100% of hosts until
the host half ships, the README, JSDoc, and hook docs all warn explicitly:
never drive an auto-load loop or a "load more" affordance from isComplete
alone; use rowCount to detect whether a fetch made progress.

One scoping note: the delivery mode belongs to the (plugin, element)
subscription — both subscription styles share the element's data channel — so
mixing subscribeToElementData and subscribeToIncrementalElementData on
the same config element is documented as unsupported (JSDoc and README). A
plugin migrates an element wholesale, which the one-line hook swap makes
easy.

Alternatives considered

• Raise the 25k page size. Doesn't change the asymptotics — delivery is
  still O(n²) in bytes, just with a different constant — and it inflates the
  cost of every individual clone/parse and worsens peak memory on both sides
  of the boundary.
• Make cumulative delivery faster (transferables, structured/columnar
  formats, compression). Worth doing on its own merits, but it still ships
  O(n²) bytes across the boundary; a 546k-row source still delivers ~6.4M
  row-equivalents. Orthogonal to, not a substitute for, incremental delivery.
• Breaking change to replace-semantics. Linear delivery with no new API
  surface, but it silently breaks every existing plugin. Unacceptable
  ecosystem cost.
• Plugin-side workarounds — pre-aggregating sources to fit under one page.
  Legitimate and documented, and I use it where the use case allows. But it
  shouldn't be the only viable pattern the SDK supports, and it doesn't serve
  detail-grid or record-level use cases at all, which are exactly the plugins
  that need large sources.
• Host-side caching without a protocol change — the host could keep
  previously fetched rows, advance the warehouse pager's offset (the
  pager.offset field already exists in its query IR), and still deliver
  cumulative payloads to plugins. This would eliminate the network-side waste
  and is worth doing independently. Rejected as sufficient: the
  browser→plugin boundary would still re-serialize, re-clone, and re-parse
  the full accumulated set every page — O(n²) at the exact hop where the
  browser's main thread pays for it. Incremental delivery fixes both legs;
  caching alone fixes one.

Compatibility

Plugin	Host	Behavior
Existing (old SDK or old APIs)	Current (cumulative)	Unchanged — no existing method, hook, type, event, or message shape is modified
Existing (old SDK or old APIs)	Future (incremental-capable)	Unchanged — the host only sends chunks to subscriptions that advertised { mode: 'incremental' }
New (useIncrementalElementData)	Current (cumulative)	Works via degraded mode — cumulative payloads are normalized to replace-chunks at offset 0; state matches today's hooks exactly; isComplete stays false
New (useIncrementalElementData)	Future (incremental-capable)	Linear delivery — each row crosses the boundary once

Existing plugins recompile and run unchanged; the new API is purely additive.

What the host needs to do

This PR defines the plugin half of the protocol. The host-side counterpart:

1. On wb:plugin:element:subscribe:data, read the optional second argument.
   If it is { mode: 'incremental' }, mark the subscription incremental.
   (Current hosts ignore extra args, which is what makes this a safe no-op.)
2. For incremental subscriptions, post the existing
   wb:plugin:element:{configId}:data event with a chunk envelope instead of
   the cumulative array:
   { data: <rows in this page only>, offset: <absolute row offset>, isComplete: <no more rows>, totalRows?: <if known> }.
   offset must be a non-negative integer; every chunk should carry the
   subscription's full column set with all column arrays the same length. An
   empty data object at offset > 0 is a valid metadata-only chunk (e.g.
   to flip isComplete when a fetch-more finds no new rows) — the client
   treats it as such rather than as a data reset.
3. Deliver chunks in non-decreasing offset order within a load. On data
   refresh (source re-query, filter change), restart from a chunk at
   offset: 0 — the client treats an offset-0 chunk as replacing all
   accumulated state, so no separate reset message is needed.
4. Overlapping or re-sent ranges are permitted: the client trusts offset and
   overwrites on overlap, so in-order at-least-once delivery per range is
   sufficient. (Out-of-order delivery below the high-water mark is not —
   an offset-0 chunk always means restart.)
5. The delivery mode is per (plugin, element) subscription: only send chunk
   envelopes to subscriptions that advertised { mode: 'incremental' }, and
   keep sending cumulative payloads to ones that did not.
6. wb:plugin:element:fetch-more and wb:plugin:element:unsubscribe:data are
   unchanged.

Note for implementers: from the outside, the host's query IR already carries
pager.offset and pager.limit, and measured traffic shows offset pinned at
0 with a growing limit. If that observation holds internally, the counterpart
change may be as small as advancing offset by the page size and holding
limit at 25,000 — though you are in a far better position than I am to
judge that.

Testing

• Client (src/client/__tests__/initialize.test.ts): the new subscribe
  method posts the subscribe message with the capability option, dispatches
  chunk envelopes to the callback, unsubscribes cleanly, normalizes legacy
  cumulative payloads into replace-chunks at offset 0, and rejects envelopes
  with malformed offsets (negative, fractional, NaN) as legacy payloads.
• Hook (src/react/__tests__/hooks.test.tsx): chunks at offsets 0 and 2
  concatenate correctly across multiple columns; isComplete and totalRows
  are surfaced; an overlapping chunk re-sent twice applies idempotently; an
  empty terminal chunk and a chunk omitting a column preserve accumulated
  data; an offset-0 restart replaces state wholesale and re-baselines
  totalRows; a column first appearing mid-stream lands at its absolute
  offset; column ids colliding with Object.prototype members
  (constructor, toString, __proto__) are handled safely; loadMore
  wiring and unmount/falsy-configId behavior mirror the existing hook tests.
• Degraded mode was simulated end-to-end through the real client: window
  message events carrying legacy cumulative payloads (the exact shape
  current hosts send) are dispatched at an incremental subscription, and the
  hook's state matches the final cumulative payload exactly — no double
  append.
• End-to-end: I additionally verified the built ESM bundle in a real
  host-page + plugin-iframe harness (Playwright Chromium, real postMessage
  across the iframe boundary) simulating both host behaviors: degraded mode
  delivers exact data with the capability ignored; an incremental host with a
  deliberately duplicated chunk stays correct and linear; and both
  legacy-API-regression scenarios confirm the existing subscribe message and
  behavior are byte-identical to today.
• Full suite: 103 tests pass (90 pre-existing, untouched), plus yarn build,
  yarn types, and yarn lint are clean.

Happy to adjust any part of the envelope or handshake to fit host-side
constraints I can't see from the client repo — the shape of the degraded-mode
guarantee is the part I'd like to preserve. I'm also happy to drop the
version bump and CHANGELOG dating from this PR if you'd rather control
releases separately — say the word and I'll rebase them out.

---

This change was developed and reviewed with the assistance of Claude
(Anthropic's AI assistant); I directed, measured, validated, and take
responsibility for all of it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UtbehD5eW7ci5awuCz1QcM