Skip to content

Journal sync receipts in IndexedDB instead of rewriting the whole batch - #26

Merged
callumalpass merged 1 commit into
mainfrom
perf/indexeddb-journal
Oct 1, 2026
Merged

callumalpass merged 1 commit into
mainfrom
perf/indexeddb-journal

Conversation

@callumalpass

Copy link
Copy Markdown
Collaborator

Fixes PERF-19 and most of PERF-5 from the overnight perf report.

Problem. The SDK records a durable receipt after every applied action. If the state store implements appendJournal, it appends one small event; otherwise it rewrites the whole mirror state, including every prepared document in the batch. The plugin's IndexedDB store never implemented it, so a first sync's writes grew quadratically with the number of notes.

Fix.

  • IndexedDbMirrorStateStore appends events under [key, "journal", n] in the existing object store and replays them on read with the SDK's own applySyncJournalEvent. Every full write and clear drops them in the same transaction.
  • The checkpoint summary is unaffected.
  • ReceiptObservingStateStore forwards appendJournal (the SDK detects it by presence) and reports history from journal events, including for a batch resumed after a restart.
  • The SDK patch exports applySyncJournalEvent and SyncJournalEvent from the mirror entry; a matching upstream PR follows.
First sync (fake-indexeddb) Before After
250 notes 508 puts, 190 MB, 664 ms 256 puts, 1.4 MB, 80 ms
1,000 notes 2,008 puts, 3.0 GB, 11.9 s 1,006 puts, 5.7 MB, 385 ms

On real IndexedDB the "before" side is worse: every write is structured-cloned and flushed to disk.

Tests (test/indexedDbJournal.test.ts, real SDK engine and real store):

  • full-state writes stay constant as the vault grows, and bytes grow linearly;
  • a crash mid-batch resumes from state plus journal in a fresh adapter;
  • full writes and clears drop the journal atomically;
  • out-of-order and baseless journals are reported as corrupt;
  • history reports every file exactly once across a restart.

Full suite: 434 pass, plus 10 lossless round trips; lint and mobile check pass. No backwards compatibility is needed: unreleased.

The SDK records a durable receipt after every applied action. Stores that
implement appendJournal get one small event; others get a full rewrite of
the mirror state, including every prepared document in the batch. The
plugin's IndexedDB store never implemented it, so writes grew
quadratically with the number of notes: a 1,000-note first sync wrote
3.0 GB (fake-indexeddb, 11.9 s) for a 0.5 MB checkpoint.

IndexedDbMirrorStateStore now appends events under [key, "journal", n]
in the same object store, replays them on read with the SDK's own
applySyncJournalEvent, and drops them atomically on every full write and
clear. The checkpoint summary is unaffected (events never change cursor,
scope or batch presence). ReceiptObservingStateStore forwards
appendJournal (the SDK detects it by presence) and reports receipts from
journal events, loading the plan once for a batch resumed after restart.

1,000-note first sync: 3.0 GB -> 5.7 MB written, 11.9 s -> 0.39 s.
The SDK patch exports applySyncJournalEvent/SyncJournalEvent from the
mirror entry; proposed upstream.
@callumalpass
callumalpass merged commit 29e15ce into main Oct 1, 2026
1 check passed
@callumalpass callumalpass mentioned this pull request Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant